10 Jun
2013
10 Jun
'13
5 p.m.
Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
+1 for 2)
--
Thomas Mortagne
10 Jun
10 Jun
5:11 p.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Jun 10, 2013, at 5:00 PM, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote:
Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
3) is ok IMO because this plugin is for our own use case so we control our default
language and I don't think it makes sense to have no default language (even if the
page is only technical with no text we could still mark it as english by default).
Big -1 for 2). The reason I added the filtering was to remove the inconsistencies we had
everywhere. That's our only solution to make sure we don't forget to set it and to
have consistent XML files.
BTW we need to verify that the export feature sets the value correctly too.
So I'm +1 for 1) or 3).
BTW the XAR plugin should also be modified to clean up both <language> and
<defaultLanguage> and it should look at XML file names to decide if it's
cleaning a translation or not.
Thanks
-Vincent
+1 for 2)
--
Thomas Mortagne
5:37 p.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
+1 for 3), I agree with Vincent that 2) could lead to inconsistencies
and having EN as default language for technical document makes sense
to me.
Thanks,
Marius
On Mon, Jun 10, 2013 at 6:11 PM, Vincent Massol <vincent(a)massol.net> wrote:
On Jun 10, 2013, at 5:00 PM, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote:
Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
3) is ok IMO because this plugin is for our own use case so we control our default
language and I don't think it makes sense to have no default language (even if the
page is only technical with no text we could still mark it as english by default).
Big -1 for 2). The reason I added the filtering was to remove the inconsistencies we had
everywhere. That's our only solution to make sure we don't forget to set it and to
have consistent XML files.
BTW we need to verify that the export feature sets the value correctly too.
So I'm +1 for 1) or 3).
BTW the XAR plugin should also be modified to clean up both <language> and
<defaultLanguage> and it should look at XML file names to decide if it's
cleaning a translation or not.
Thanks
-Vincent
+1 for 2)
--
Thomas Mortagne
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
5:37 p.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Jun 10, 2013, at 5:11 PM, Vincent Massol <vincent(a)massol.net> wrote:
On Jun 10, 2013, at 5:00 PM, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote:
I think I have a small preference for 3) which is more explicit and less magical. If we
support the mode where no default translation is set then 3) seems best to me.
Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
3) is ok IMO because this plugin is for our own use case so we control our default
language and I don't think it makes sense to have no default language (even if the
page is only technical with no text we could still mark it as english by default).
Big -1 for 2). The reason I added the filtering was to remove the inconsistencies we had
everywhere. That's our only solution to make sure we don't forget to set it and to
have consistent XML files.
BTW we need to verify that the export feature sets the value correctly too.
So I'm +1 for 1) or 3). BTW the XAR plugin should also be modified to clean up
both <language> and <defaultLanguage> and it should look at XML file names to
decide if it's cleaning a translation or not.
We need to define the rules. For example for 3):
* Main document (e.g. WebHome.xml):
** make sure <translation> is empty
** set <defaultTranslation> to "en"
* Translation document (e.g. WebHome.fr.xml):
** make sure <translation> is set to the value from the file name (ie. we extract
"fr" from WebHome.fr.xml)
** what value do we set for <defautTranslation>? Empty or "en"?
Thanks
-Vincent
Thanks
-Vincent
> +1 for 2)
>
> --
> Thomas Mortagne
6:09 p.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Mon, Jun 10, 2013 at 5:37 PM, Vincent Massol <vincent(a)massol.net> wrote:
On Jun 10, 2013, at 5:11 PM, Vincent Massol <vincent(a)massol.net> wrote:
It's not supposed to be empty it's supposed to be 0.
On Jun 10, 2013, at 5:00 PM, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote:
I think I have a small preference for 3) which is more explicit and less magical. If we
support the mode where no default translation is set then 3) seems best to me.
Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
3) is ok IMO because this plugin is for our own use case so we control our default
language and I don't think it makes sense to have no default language (even if the
page is only technical with no text we could still mark it as english by default).
Big -1 for 2). The reason I added the filtering was to remove the inconsistencies we had
everywhere. That's our only solution to make sure we don't forget to set it and to
have consistent XML files.
BTW we need to verify that the export feature sets the value correctly too.
So I'm +1 for 1) or 3). BTW the XAR plugin should also be modified to
clean up both <language> and <defaultLanguage> and it should look at XML file
names to decide if it's cleaning a translation or not.
We need to define the rules. For example for 3):
* Main document (e.g. WebHome.xml):
** make sure <translation> is empty ** set <defaultTranslation> to "en"
* Translation document (e.g. WebHome.fr.xml):
** make sure <translation> is set to the value from the file name (ie. we extract
"fr" from WebHome.fr.xml)
This is dangerous and useless anyway IMO, someone could want to have a
document with a dot in its name for any reason and I doubt you will
even get a translation with the wrong language in its XML.
** what value do we set for <defautTranslation>?
Empty or "en"?
The default language is supposed to be the same all translations of
the document.
Thanks
-Vincent
--
Thomas Mortagne
Thanks
-Vincent
> +1 for 2)
>
> --
> Thomas Mortagne
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
6:36 p.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Jun 10, 2013, at 6:09 PM, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote:
On Mon, Jun 10, 2013 at 5:37 PM, Vincent Massol
<vincent(a)massol.net> wrote:
This is our current format for translations. For me it's never useless to ensure stuff
are in sync… The check could be the other way around: Verify that the name of the file
matches the value from the XML.
Now we could also decide to have an option to skip that check if we really want to share
this tool for outside users.
Thanks
-Vincent
On Jun 10, 2013, at 5:11 PM, Vincent Massol <vincent(a)massol.net> wrote:
It's not supposed to be empty it's supposed to be 0.
On Jun 10, 2013, at 5:00 PM, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote:
I think I have a small preference for 3) which is more explicit and less magical. If we
support the mode where no default translation is set then 3) seems best to me.
Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
3) is ok IMO because this plugin is for our own use case so we control our default
language and I don't think it makes sense to have no default language (even if the
page is only technical with no text we could still mark it as english by default).
Big -1 for 2). The reason I added the filtering was to remove the inconsistencies we had
everywhere. That's our only solution to make sure we don't forget to set it and to
have consistent XML files.
BTW we need to verify that the export feature sets the value correctly too.
So I'm +1 for 1) or 3). BTW the XAR plugin should also be modified to
clean up both <language> and <defaultLanguage> and it should look at XML file
names to decide if it's cleaning a translation or not.
We need to define the rules. For example for 3):
* Main document (e.g. WebHome.xml):
** make sure <translation> is empty ** set <defaultTranslation> to
"en"
* Translation document (e.g. WebHome.fr.xml):
** make sure <translation> is set to the value from the file name (ie. we extract
"fr" from WebHome.fr.xml)
This is dangerous and useless anyway IMO, someone could want to have a
document with a dot in its name for any reason and I doubt you will
even get a translation with the wrong language in its XML. ** what value
do we set for <defautTranslation>? Empty or "en"?
The default language is supposed to be the same all translations of
the document.
Thanks
-Vincent
--
Thomas Mortagne
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs Thanks
-Vincent
> +1 for 2)
>
> --
> Thomas Mortagne
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs 
8:25 p.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
+1 for 2)
Neither option is good in general. The main problem is that most
documents are written in the "Velocity" language, not in the
"English"
language, meaning that it only contains code (which won't be seen by the
user), and translations, which depend on a lot of factors. It's not good
to say that the default language of a dynamically translated document is
en, since a wiki configured with a different language will only display
them in that language, never in en.
There are only a few documents that contain real text (normally only the
sandbox should have real text, everything else should be localized), and
for those it's OK to specify the actual language.
Other options:
4) Detect somehow localized documents and index:
- the raw content using a non-language-specific analyzer
- the content translated into all the languages registered in the
administration, each with the proper language-specific analyzer, if they
are supported by Solr; this includes the default wiki language.
4a) localized document = the default language is empty
4b) localized document = the default language is literally "localized"
4c) add another document flag for marking localized documents
5) When the defaultLanguage is empty, render in the configured wiki
default language
I like 4) since it makes localized documents really searchable in all
the languages "supported" by that wiki instance.
4a) is a behavior change, so it might cause some trouble
4b) is the safest and requires the least amount of changes
The number of document fields is increasing, so I'm not that fond of 4c)
--
Sergiu Dumitriu
http://purl.org/net/sergiu
9:12 p.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org> wrote:
On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
I'm not sure I agree with this vision. It really depends on the use case. So far we
haven't found a perfect solution.
Some pages will have more code than content, others will have more content than code. For
the former, keys are best and for the latter translations are best.
In any case I don't understand the problem. What is the issue with saying that all our
pages are in English by default. If a wiki is configured to be in another language and
there's no translation for that language the default language (ie "English")
will be used.
What am I missing?
(I'm not commenting on anything below yet because I feel it's important to agree
on what's before first)
Thanks
-Vincent
Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
+1 for 2)
Neither option is good in general. The main problem is that most
documents are written in the "Velocity" language, not in the
"English"
language, meaning that it only contains code (which won't be seen by the
user), and translations, which depend on a lot of factors. It's not good
to say that the default language of a dynamically translated document is
en, since a wiki configured with a different language will only display
them in that language, never in en.
There are only a few documents that contain real text (normally only the
sandbox should have real text, everything else should be localized), and
for those it's OK to specify the actual language. Other options:
4) Detect somehow localized documents and index:
- the raw content using a non-language-specific analyzer
- the content translated into all the languages registered in the
administration, each with the proper language-specific analyzer, if they
are supported by Solr; this includes the default wiki language.
4a) localized document = the default language is empty
4b) localized document = the default language is literally "localized"
4c) add another document flag for marking localized documents
5) When the defaultLanguage is empty, render in the configured wiki
default language
I like 4) since it makes localized documents really searchable in all
the languages "supported" by that wiki instance.
4a) is a behavior change, so it might cause some trouble
4b) is the safest and requires the least amount of changes
The number of document fields is increasing, so I'm not that fond of 4c)
--
Sergiu Dumitriu
http://purl.org/net/sergiu
9:49 p.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On 06/10/2013 03:12 PM, Vincent Massol wrote:
On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org> wrote:
There is no actual text in the document. How can you say that the
language of
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
is English, since there's no English sentence in there? Depending on the
configuration, the same document will appear in German, Chinese, even
Klingon, without changing anything in the document, so it is definitely
not an English document.
Scenario: Set up a new XWiki instance, and change the default language
of the wiki to German. When you browse the wiki, everything is in
German. Yet all the documents say that they're in English.
Problem 1: The wiki is indexed as English text, so searching for text
that the user actually sees in the wiki won't return any results.
Problem 2: Editing such a document will automatically create a
translation, since the original document is in English, and the user
wants to edit a German document. Since the two languages are not
compatible, a translation will be created automatically. Now the code
has been forked, and automatic updates using the Distribution Wizard
will update the hidden English document, since that is the default one,
while the forked translation will stay behind.
That is why I'm saying that this kind of documents don't have a
language, and they never should have. They adapt themselves to the
user's language, so they're written in no language, yet they can match
all languages.
Of course, not all documents are like this, as I originally stated
myself, and there are valid cases where documents should have "en" as
the default translation.
On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
I'm not sure I agree with this vision. It really depends on the use case. So far we
haven't found a perfect solution.
Some pages will have more code than content, others will have more content than code. For
the former, keys are best and for the latter translations are best.
In any case I don't understand the problem. What is the issue with saying that all
our pages are in English by default. If a wiki is configured to be in another language and
there's no translation for that language the default language (ie "English")
will be used. Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
+1 for 2)
Neither option is good in general. The main problem is that most
documents are written in the "Velocity" language, not in the
"English"
language, meaning that it only contains code (which won't be seen by the
user), and translations, which depend on a lot of factors. It's not good
to say that the default language of a dynamically translated document is
en, since a wiki configured with a different language will only display
them in that language, never in en.
There are only a few documents that contain real text (normally only the
sandbox should have real text, everything else should be localized), and
for those it's OK to specify the actual language. What am I missing?
(I'm not commenting on anything below yet because I feel it's important to agree
on what's before first)
Thanks
-Vincent
> Other options:
>
> 4) Detect somehow localized documents and index:
> - the raw content using a non-language-specific analyzer
> - the content translated into all the languages registered in the
> administration, each with the proper language-specific analyzer, if they
> are supported by Solr; this includes the default wiki language.
> 4a) localized document = the default language is empty
> 4b) localized document = the default language is literally "localized"
> 4c) add another document flag for marking localized documents
>
> 5) When the defaultLanguage is empty, render in the configured wiki
> default language
>
>
> I like 4) since it makes localized documents really searchable in all
> the languages "supported" by that wiki instance.
>
> 4a) is a behavior change, so it might cause some trouble
> 4b) is the safest and requires the least amount of changes
> The number of document fields is increasing, so I'm not that fond of 4c)
--
Sergiu Dumitriu
http://purl.org/net/sergiu
11 Jun
11 Jun
8:25 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Mon, Jun 10, 2013 at 10:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org> wrote:
On 06/10/2013 03:12 PM, Vincent Massol wrote:
On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org> wrote:
On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
I'm not sure I agree with this vision. It really depends on the use case. So far we
haven't found a perfect solution.
Some pages will have more code than content, others will have more content than code. For
the former, keys are best and for the latter translations are best.
In any case I don't understand the problem. What is the issue with saying that all
our pages are in English by default. If a wiki is configured to be in another language and
there's no translation for that language the default language (ie "English")
will be used. Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
+1 for 2)
Neither option is good in general. The main problem is that most
documents are written in the "Velocity" language, not in the
"English"
language, meaning that it only contains code (which won't be seen by the
user), and translations, which depend on a lot of factors. It's not good
to say that the default language of a dynamically translated document is
en, since a wiki configured with a different language will only display
them in that language, never in en.
There are only a few documents that contain real text (normally only the
sandbox should have real text, everything else should be localized), and
for those it's OK to specify the actual language. There is no actual text in the document. How can you
say that the
language of
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
is English, since there's no English sentence in there? Depending on the
configuration, the same document will appear in German, Chinese, even
Klingon, without changing anything in the document, so it is definitely
not an English document.
It looks English to me: "if", "request", "valid",
"key", "user",
"true", "login", "services", "localization", etc.
are all English
words. There are no sentences sure (although there can be code
comments), but the code is definitely written in English. If you index
the raw content then its language should be English IMO. If you index
the rendered content then I agree the result can be different
depending on the context language, but on the other hand some of this
code might not render anything (XWiki.LiveTableResultsMacros). Anyway,
I think most of the users won't care about such documents with code,
mainly because these documents are supposed to be hidden and thus not
included in the search results by default.
Let's no forget that an important part of the code is written in
objects (JSX/SSX) and they can also contain calls to the localization
service. Should we render that content and index it in multiple
languages?
Scenario: Set up a new XWiki instance, and change the default language
of the wiki to German. When you browse the wiki, everything is in
German. Yet all the documents say that they're in English.
Problem 1: The wiki is indexed as English text, so searching for text
that the user actually sees in the wiki won't return any results.
Problem 2: Editing such a document will automatically
create a
translation, since the original document is in English, and the user
wants to edit a German document. Since the two languages are not
compatible, a translation will be created automatically. Now the code
has been forked, and automatic updates using the Distribution Wizard
will update the hidden English document, since that is the default one,
while the forked translation will stay behind.
This is indeed a problem..
Thanks,
Marius
That is why I'm saying that this kind of documents don't have a
language, and they never should have. They adapt themselves to the
user's language, so they're written in no language, yet they can match
all languages.
Of course, not all documents are like this, as I originally stated
myself, and there are valid cases where documents should have "en" as
the default translation.
What am I missing?
(I'm not commenting on anything below yet because I feel it's important to agree
on what's before first)
Thanks
-Vincent
> Other options:
>
> 4) Detect somehow localized documents and index:
> - the raw content using a non-language-specific analyzer
> - the content translated into all the languages registered in the
> administration, each with the proper language-specific analyzer, if they
> are supported by Solr; this includes the default wiki language.
> 4a) localized document = the default language is empty
> 4b) localized document = the default language is literally "localized"
> 4c) add another document flag for marking localized documents
>
> 5) When the defaultLanguage is empty, render in the configured wiki
> default language
>
>
> I like 4) since it makes localized documents really searchable in all
> the languages "supported" by that wiki instance.
>
> 4a) is a behavior change, so it might cause some trouble
> 4b) is the safest and requires the least amount of changes
> The number of document fields is increasing, so I'm not that fond of 4c)
--
Sergiu Dumitriu
http://purl.org/net/sergiu
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
10:19 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
Hi Sergiu,
On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org> wrote:
On 06/10/2013 03:12 PM, Vincent Massol wrote:
Thanks a lot for describing these 2 use cases that I definitely wouldn't have thought
about! That's very useful.
So it seems that suddenly it's becoming more complex ;)
Basically it means that if we have documents that mix content and scripting we're
going to have issues:
* Either they're marked as having no default language and the english content will be
indexed in the default language of the wiki
* Either they're marked as "en" and the user will not have the scripts in
the search results and the DW/EM will update only the default version if the user has
created a translation
I'm sure we have lots of cases like this, the easiest one being the main home page:
-------------------
It's an easy-to-edit website that will help you work better together. This Wiki is
made of //pages// sorted by //spaces//. You're currently in the **Main** space,
looking at its home page (**WebHome**).
Learn how to use XWiki with the {{velocity}}[[Getting Started
Guide>>http://enterprise.xwiki.org/xwiki/bin/view/GettingStar…]{{/velocity}}.
{{velocity}}
#if($hasEdit)You can then use the [[Sandbox space>>Sandbox.WebHome]] to try
out your wiki's features.#end
{{/velocity}}
-------------------
It has both content and script… If it's marked as "en" then if the user
searches for "hasEdit" he won't get it if his wiki is in a language other
than "en".
Unless, if there's no translation in the language of the user then we return the
default language results for that page. Would that make sense?
But there's still the issue of editing the page, which will create a translation and
then imagine that we replace the velocity script in a future version, then the user will
only get his default page updated and not his translation… However that's a general
problem I guess...
If it has no default language (as is the case now BTW) then it seems less of an issue it
seems. It just means:
* If user searches for "work" he'll get result even though he's no in an
"en" wiki. But then he's searching for an english word too ;)
* any other downside?
In view of all this, it seems that not setting any default language is a lesser evil,
doesn't it?
Thanks
-Vincent
On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org> wrote:
There is no actual text in the document. How can you say that the
language of
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
is English, since there's no English sentence in there? Depending on the
configuration, the same document will appear in German, Chinese, even
Klingon, without changing anything in the document, so it is definitely
not an English document.
Scenario: Set up a new XWiki instance, and change the default language
of the wiki to German. When you browse the wiki, everything is in
German. Yet all the documents say that they're in English.
Problem 1: The wiki is indexed as English text, so searching for text
that the user actually sees in the wiki won't return any results.
Problem 2: Editing such a document will automatically create a
translation, since the original document is in English, and the user
wants to edit a German document. Since the two languages are not
compatible, a translation will be created automatically. Now the code
has been forked, and automatic updates using the Distribution Wizard
will update the hidden English document, since that is the default one,
while the forked translation will stay behind. On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
I'm not sure I agree with this vision. It really depends on the use case. So far we
haven't found a perfect solution.
Some pages will have more code than content, others will have more content than code. For
the former, keys are best and for the latter translations are best.
In any case I don't understand the problem. What is the issue with saying that all
our pages are in English by default. If a wiki is configured to be in another language and
there's no translation for that language the default language (ie "English")
will be used. Hi devs,
Right now the XAR plugin format goal systematically empty the
<defaultLanguage> property.
This is wrong IMO since it means we have no idea what is the default
document language, it was not too visible before but it's really not
very nice for things like the localization module and especially SOLR
which store deferently the content depending on the language (stop
words, etc).
I see several possibilities:
1) We don't touch the XAR maven plugin and we state that when default
language is not set, it's en (in the importer for example or in
XWikiDocument#getDefaultLanguage)
2) We stop filtering default language in the XAR plugin and we set it
to en for all document in which it make sense
3) We force default language to "en" in the XAR plugin
WDYT ?
I don't like too much 1) since some technical document could really be
seen has having no default language, some document without any literal
content. But it's more a -0 than a -1, I understand other would want
this for simplicity.
About 3) as I said having a default language empty is a valid use case
IMO so -0 for this one to. Still a bit better than 1) since the use
case is still possible.
+1 for 2)
Neither option is good in general. The main problem is that most
documents are written in the "Velocity" language, not in the
"English"
language, meaning that it only contains code (which won't be seen by the
user), and translations, which depend on a lot of factors. It's not good
to say that the default language of a dynamically translated document is
en, since a wiki configured with a different language will only display
them in that language, never in en.
There are only a few documents that contain real text (normally only the
sandbox should have real text, everything else should be localized), and
for those it's OK to specify the actual language. That is why I'm saying that this kind of documents
don't have a
language, and they never should have. They adapt themselves to the
user's language, so they're written in no language, yet they can match
all languages.
Of course, not all documents are like this, as I originally stated
myself, and there are valid cases where documents should have "en" as
the default translation.
> What am I missing?
>
> (I'm not commenting on anything below yet because I feel it's important to
agree on what's before first)
>
> Thanks
> -Vincent
>
>> Other options:
>>
>> 4) Detect somehow localized documents and index:
>> - the raw content using a non-language-specific analyzer
>> - the content translated into all the languages registered in the
>> administration, each with the proper language-specific analyzer, if they
>> are supported by Solr; this includes the default wiki language.
>> 4a) localized document = the default language is empty
>> 4b) localized document = the default language is literally "localized"
>> 4c) add another document flag for marking localized documents
>>
>> 5) When the defaultLanguage is empty, render in the configured wiki
>> default language
>>
>>
>> I like 4) since it makes localized documents really searchable in all
>> the languages "supported" by that wiki instance.
>>
>> 4a) is a behavior change, so it might cause some trouble
>> 4b) is the safest and requires the least amount of changes
>> The number of document fields is increasing, so I'm not that fond of 4c)
10:54 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
That's not the home page, that's the Welcome page and it's not a very
good example since this page do have translations already so we
already decided what we wanted for this pages in practice: each
translation of the page copy the scripts.
On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net> wrote:
Hi Sergiu,
On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org> wrote:
--
Thomas Mortagne
On 06/10/2013 03:12 PM, Vincent Massol wrote:
Thanks a lot for describing these 2 use cases that I definitely wouldn't have thought
about! That's very useful.
So it seems that suddenly it's becoming more complex ;)
Basically it means that if we have documents that mix content and scripting we're
going to have issues:
* Either they're marked as having no default language and the english content will be
indexed in the default language of the wiki
* Either they're marked as "en" and the user will not have the scripts in
the search results and the DW/EM will update only the default version if the user has
created a translation
I'm sure we have lots of cases like this, the easiest one being the main home page:
-------------------
It's an easy-to-edit website that will help you work better together. This Wiki is
made of //pages// sorted by //spaces//. You're currently in the **Main** space,
looking at its home page (**WebHome**).
Learn how to use XWiki with the {{velocity}}[[Getting Started
Guide>>http://enterprise.xwiki.org/xwiki/bin/view/GettingStar…]{{/velocity}}.
{{velocity}}
#if($hasEdit)You can then use the [[Sandbox space>>Sandbox.WebHome]] to try
out your wiki's features.#end
{{/velocity}}
-------------------
It has both content and script… If it's marked as "en" then if the user
searches for "hasEdit" he won't get it if his wiki is in a language other
than "en".
Unless, if there's no translation in the language of the user then we return the
default language results for that page. Would that make sense?
But there's still the issue of editing the page, which will create a translation and
then imagine that we replace the velocity script in a future version, then the user will
only get his default page updated and not his translation… However that's a general
problem I guess...
If it has no default language (as is the case now BTW) then it seems less of an issue it
seems. It just means:
* If user searches for "work" he'll get result even though he's no in
an "en" wiki. But then he's searching for an english word too ;)
* any other downside?
In view of all this, it seems that not setting any default language is a lesser evil,
doesn't it?
Thanks
-Vincent
On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org> wrote:
There is no actual text in the document. How can you say that the
language of
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
is English, since there's no English sentence in there? Depending on the
configuration, the same document will appear in German, Chinese, even
Klingon, without changing anything in the document, so it is definitely
not an English document.
Scenario: Set up a new XWiki instance, and change the default language
of the wiki to German. When you browse the wiki, everything is in
German. Yet all the documents say that they're in English.
Problem 1: The wiki is indexed as English text, so searching for text
that the user actually sees in the wiki won't return any results.
Problem 2: Editing such a document will automatically create a
translation, since the original document is in English, and the user
wants to edit a German document. Since the two languages are not
compatible, a translation will be created automatically. Now the code
has been forked, and automatic updates using the Distribution Wizard
will update the hidden English document, since that is the default one,
while the forked translation will stay behind. On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
> Hi devs,
>
> Right now the XAR plugin format goal systematically empty the
> <defaultLanguage> property.
>
> This is wrong IMO since it means we have no idea what is the default
> document language, it was not too visible before but it's really not
> very nice for things like the localization module and especially SOLR
> which store deferently the content depending on the language (stop
> words, etc).
>
> I see several possibilities:
>
> 1) We don't touch the XAR maven plugin and we state that when default
> language is not set, it's en (in the importer for example or in
> XWikiDocument#getDefaultLanguage)
> 2) We stop filtering default language in the XAR plugin and we set it
> to en for all document in which it make sense
> 3) We force default language to "en" in the XAR plugin
>
> WDYT ?
>
> I don't like too much 1) since some technical document could really be
> seen has having no default language, some document without any literal
> content. But it's more a -0 than a -1, I understand other would want
> this for simplicity.
>
> About 3) as I said having a default language empty is a valid use case
> IMO so -0 for this one to. Still a bit better than 1) since the use
> case is still possible.
>
> +1 for 2)
Neither option is good in general. The main problem is that most
documents are written in the "Velocity" language, not in the
"English"
language, meaning that it only contains code (which won't be seen by the
user), and translations, which depend on a lot of factors. It's not good
to say that the default language of a dynamically translated document is
en, since a wiki configured with a different language will only display
them in that language, never in en.
There are only a few documents that contain real text (normally only the
sandbox should have real text, everything else should be localized), and
for those it's OK to specify the actual language.
I'm not sure I agree with this vision. It really depends on the use case. So far we
haven't found a perfect solution.
Some pages will have more code than content, others will have more content than code. For
the former, keys are best and for the latter translations are best.
In any case I don't understand the problem. What is the issue with saying that all
our pages are in English by default. If a wiki is configured to be in another language and
there's no translation for that language the default language (ie "English")
will be used. That is why I'm saying that this kind of
documents don't have a
language, and they never should have. They adapt themselves to the
user's language, so they're written in no language, yet they can match
all languages.
Of course, not all documents are like this, as I originally stated
myself, and there are valid cases where documents should have "en" as
the default translation.
> What am I missing?
>
> (I'm not commenting on anything below yet because I feel it's important to
agree on what's before first)
>
> Thanks
> -Vincent
>
>> Other options:
>>
>> 4) Detect somehow localized documents and index:
>> - the raw content using a non-language-specific analyzer
>> - the content translated into all the languages registered in the
>> administration, each with the proper language-specific analyzer, if they
>> are supported by Solr; this includes the default wiki language.
>> 4a) localized document = the default language is empty
>> 4b) localized document = the default language is literally "localized"
>> 4c) add another document flag for marking localized documents
>>
>> 5) When the defaultLanguage is empty, render in the configured wiki
>> default language
>>
>>
>> I like 4) since it makes localized documents really searchable in all
>> the languages "supported" by that wiki instance.
>>
>> 4a) is a behavior change, so it might cause some trouble
>> 4b) is the safest and requires the least amount of changes
>> The number of document fields is increasing, so I'm not that fond of 4c)
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs 
11:09 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
Thanks Sergui for catching up here, while I was off.
This is a situation I claim since a long time now, which is that some
document has simply no language, because all text displayed by those
document are simply produced by the localization module. Not only the
search issue are affected, but also the display of language selection in a
multi-wiki, and currently this is not as nice as it should be.
IMO, with a wiki that could produce with a single source documents,
displayed document in all available translation of the wiki (like the UI
does), we need to have the notion of a "no language" or "any
language"
document. In my own projects, I have been able to manage that properly
with the empty "default language" and empty "language" case. Since
having
both is stupid, we will surely merge those column in the future, but we
need to keep the idea of a "no language" or "any language" document,
whatever you see it, and to properly manage it, not only for our own XAR
but for user produced documents as well.
For those reasons, I am -1 for 1) and 3), and +1 for 2)
I follow Thomas, the Welcome page is not a good example, since this on has
really a translation in all languages.
I do not understand the intend to index code statement in an index, this is
the rendered document that should be indexed. And for "any language"
document, it should be indexed separately for each language enable in a
multi-language wiki.
So, to respond to Vincent, using properly the "any language" case by
clearing the default language where needed, is the right way to go, but we
also need to manage that special case properly elsewhere (indexing,
language selection on the page,...)
On Tue, Jun 11, 2013 at 10:54 AM, Thomas Mortagne <thomas.mortagne(a)xwiki.com
wrote:
> That's not the home page, that's the Welcome page and it's not a very
> good example since this page do have translations already so we
> already decided what we wanted for this pages in practice: each
> translation of the page copy the scripts.
>
> On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net>
wrote:
> > Hi Sergiu,
> >
> > On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
> >
> >> On 06/10/2013 03:12 PM, Vincent Massol wrote:
> >>>
> >>> On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
> >>>
> >>>> On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
> >>>>> Hi devs,
> >>>>>
> >>>>> Right now the XAR plugin format goal systematically empty the
> >>>>> <defaultLanguage> property.
> >>>>>
> >>>>> This is wrong IMO since it means we have no idea what is the
default
> >>>>> document language, it was not too visible before but it's
really not
> >>>>> very nice for things like the localization module and especially
SOLR
> >>>>> which store deferently the content depending on the language
(stop
> >>>>> words, etc).
> >>>>>
> >>>>> I see several possibilities:
> >>>>>
> >>>>> 1) We don't touch the XAR maven plugin and we state that
when default
> >>>>> language is not set, it's en (in the importer for example or
in
> >>>>> XWikiDocument#getDefaultLanguage)
> >>>>> 2) We stop filtering default language in the XAR plugin and we
set it
> >>>>> to en for all document in which it make sense
> >>>>> 3) We force default language to "en" in the XAR
plugin
> >>>>>
> >>>>> WDYT ?
> >>>>>
> >>>>> I don't like too much 1) since some technical document could
really
> be
> >>>>> seen has having no default language, some document without any
> literal
> >>>>> content. But it's more a -0 than a -1, I understand other
would want
> >>>>> this for simplicity.
> >>>>>
> >>>>> About 3) as I said having a default language empty is a valid
use
> case
> >>>>> IMO so -0 for this one to. Still a bit better than 1) since the
use
> >>>>> case is still possible.
> >>>>>
> >>>>> +1 for 2)
> >>>>
> >>>> Neither option is good in general. The main problem is that most
> >>>> documents are written in the "Velocity" language, not in
the "English"
> >>>> language, meaning that it only contains code (which won't be
seen by
> the
> >>>> user), and translations, which depend on a lot of factors. It's
not
> good
> >>>> to say that the default language of a dynamically translated
document
> is
> >>>> en, since a wiki configured with a different language will only
> display
> >>>> them in that language, never in en.
> >>>>
> >>>> There are only a few documents that contain real text (normally
only
> the
> >>>> sandbox should have real text, everything else should be
localized),
> and
> >>>> for those it's OK to specify the actual language.
> >>>
> >>> I'm not sure I agree with this vision. It really depends on the use
> case. So far we haven't found a perfect solution.
> >>>
> >>> Some pages will have more code than content, others will have more
> content than code. For the former, keys are best and for the latter
> translations are best.
> >>>
> >>> In any case I don't understand the problem. What is the issue with
> saying that all our pages are in English by default. If a wiki is
> configured to be in another language and there's no translation for that
> language the default language (ie "English") will be used.
> >>
> >> There is no actual text in the document. How can you say that the
> >> language of
> >>
>
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
> >> is English, since there's no English sentence in there? Depending on
the
> >> configuration, the same document will appear in German, Chinese, even
> >> Klingon, without changing anything in the document, so it is definitely
> >> not an English document.
> >>
> >> Scenario: Set up a new XWiki instance, and change the default language
> >> of the wiki to German. When you browse the wiki, everything is in
> >> German. Yet all the documents say that they're in English.
> >>
> >> Problem 1: The wiki is indexed as English text, so searching for text
> >> that the user actually sees in the wiki won't return any results.
> >>
> >> Problem 2: Editing such a document will automatically create a
> >> translation, since the original document is in English, and the user
> >> wants to edit a German document. Since the two languages are not
> >> compatible, a translation will be created automatically. Now the code
> >> has been forked, and automatic updates using the Distribution Wizard
> >> will update the hidden English document, since that is the default one,
> >> while the forked translation will stay behind.
> >
> > Thanks a lot for describing these 2 use cases that I definitely wouldn't
> have thought about! That's very useful.
> >
> > So it seems that suddenly it's becoming more complex ;)
> >
> > Basically it means that if we have documents that mix content and
> scripting we're going to have issues:
> > * Either they're marked as having no default language and the english
> content will be indexed in the default language of the wiki
> > * Either they're marked as "en" and the user will not have the
scripts
> in the search results and the DW/EM will update only the default version if
> the user has created a translation
> >
> > I'm sure we have lots of cases like this, the easiest one being the main
> home page:
> >
> > -------------------
> > It's an easy-to-edit website that will help you work better together.
> This Wiki is made of //pages// sorted by //spaces//. You're currently in
> the **Main** space, looking at its home page (**WebHome**).
> >
> > Learn how to use XWiki with the {{velocity}}[[Getting Started
> Guide>>
>
http://enterprise.xwiki.org/xwiki/bin/view/GettingStarted/WebHome?version=$…
> .
> >
> > {{velocity}}
> > #if($hasEdit)You can then use the [[Sandbox
> space>>Sandbox.WebHome]] to try out your wiki's features.#end
> > {{/velocity}}
> > -------------------
> >
> > It has both content and script… If it's marked as "en" then if the
user
> searches for "hasEdit" he won't get it if his wiki is in a language
other
> than "en".
> >
> > Unless, if there's no translation in the language of the user then we
> return the default language results for that page. Would that make sense?
> >
> > But there's still the issue of editing the page, which will create a
> translation and then imagine that we replace the velocity script in a
> future version, then the user will only get his default page updated and
> not his translation… However that's a general problem I guess...
> >
> > If it has no default language (as is the case now BTW) then it seems
> less of an issue it seems. It just means:
> > * If user searches for "work" he'll get result even though
he's no in an
> "en" wiki. But then he's searching for an english word too ;)
> > * any other downside?
> >
> > In view of all this, it seems that not setting any default language is a
> lesser evil, doesn't it?
> >
> > Thanks
> > -Vincent
> >
> >> That is why I'm saying that this kind of documents don't have a
> >> language, and they never should have. They adapt themselves to the
> >> user's language, so they're written in no language, yet they can
match
> >> all languages.
> >>
> >> Of course, not all documents are like this, as I originally stated
> >> myself, and there are valid cases where documents should have "en"
as
> >> the default translation.
> >>
> >>> What am I missing?
> >>>
> >>> (I'm not commenting on anything below yet because I feel it's
> important to agree on what's before first)
> >>>
> >>> Thanks
> >>> -Vincent
> >>>
> >>>> Other options:
> >>>>
> >>>> 4) Detect somehow localized documents and index:
> >>>> - the raw content using a non-language-specific analyzer
> >>>> - the content translated into all the languages registered in the
> >>>> administration, each with the proper language-specific analyzer, if
> they
> >>>> are supported by Solr; this includes the default wiki language.
> >>>> 4a) localized document = the default language is empty
> >>>> 4b) localized document = the default language is literally
"localized"
> >>>> 4c) add another document flag for marking localized documents
> >>>>
> >>>> 5) When the defaultLanguage is empty, render in the configured wiki
> >>>> default language
> >>>>
> >>>>
> >>>> I like 4) since it makes localized documents really searchable in
all
> >>>> the languages "supported" by that wiki instance.
> >>>>
> >>>> 4a) is a behavior change, so it might cause some trouble
> >>>> 4b) is the safest and requires the least amount of changes
> >>>> The number of document fields is increasing, so I'm not that
fond of
> 4c)
> >
> > _______________________________________________
> > devs mailing list
> > devs(a)xwiki.org
> > http://lists.xwiki.org/mailman/listinfo/devs
>
>
>
> --
> Thomas Mortagne
> _______________________________________________
> devs mailing list
> devs(a)xwiki.org
> http://lists.xwiki.org/mailman/listinfo/devs
>
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
11:21 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Jun 11, 2013, at 11:09 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
Thanks Sergui for catching up here, while I was off.
This is a situation I claim since a long time now, which is that some
document has simply no language, because all text displayed by those
document are simply produced by the localization module. Not only the
search issue are affected, but also the display of language selection in a
multi-wiki, and currently this is not as nice as it should be.
IMO, with a wiki that could produce with a single source documents,
displayed document in all available translation of the wiki (like the UI
does), we need to have the notion of a "no language" or "any
language"
document. In my own projects, I have been able to manage that properly
with the empty "default language" and empty "language" case. Since
having
both is stupid, we will surely merge those column in the future, but we
need to keep the idea of a "no language" or "any language" document,
whatever you see it, and to properly manage it, not only for our own XAR
but for user produced documents as well.
For those reasons, I am -1 for 1) and 3), and +1 for 2)
BTW I forgot to say in the my previous reply but obviously I remove my -1 for 2) since
it's much more complex than I thought.
I follow Thomas, the Welcome page is not a good
example, since this on has
really a translation in all languages.
I do not understand the intend to index code statement in an index, this is
the rendered document that should be indexed. And for "any language"
document, it should be indexed separately for each language enable in a
multi-language wiki.
ok, that's something very important to decide.
I think the 2 use cases are valide; xwiki can be used both by end users who only care
about pure content and it can also be used by developers who develop wiki pages with
scripts inside. For the former they want to see only content result and for the latter
they want to see script results. For example, as a dev I want to see where I call such
velocity macro, or where I use such rendering macro.
So I really believe that we need to index both types of content: rendered and raw.
Then we need to decide how to present that in the UI, but it could an option in the
advanced search to include raw results too for example (and remove duplications…).
So, to respond to Vincent, using properly the
"any language" case by
clearing the default language where needed, is the right way to go, but we
also need to manage that special case properly elsewhere (indexing,
language selection on the page,…)
So what's the rule for putting <defaultLanguage>en</defaultLanguage>?
Whenever there's a page having at least one word of English in the rendered result?
Thanks
-Vincent
On Tue, Jun 11, 2013 at 10:54 AM, Thomas Mortagne
<thomas.mortagne(a)xwiki.com
wrote:
> That's not the home page, that's the Welcome page and it's not a very
> good example since this page do have translations already so we
> already decided what we wanted for this pages in practice: each
> translation of the page copy the scripts.
>
> On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net>
wrote:
>> Hi Sergiu,
>>
>> On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
>>
>>> On 06/10/2013 03:12 PM, Vincent Massol wrote:
>>>>
>>>> On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
>>>>
>>>>> On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
>>>>>> Hi devs,
>>>>>>
>>>>>> Right now the XAR plugin format goal systematically empty the
>>>>>> <defaultLanguage> property.
>>>>>>
>>>>>> This is wrong IMO since it means we have no idea what is the
default
>>>>>> document language, it was not too visible before but it's
really not
>>>>>> very nice for things like the localization module and especially
SOLR
>>>>>> which store deferently the content depending on the language
(stop
>>>>>> words, etc).
>>>>>>
>>>>>> I see several possibilities:
>>>>>>
>>>>>> 1) We don't touch the XAR maven plugin and we state that when
default
>>>>>> language is not set, it's en (in the importer for example or
in
>>>>>> XWikiDocument#getDefaultLanguage)
>>>>>> 2) We stop filtering default language in the XAR plugin and we
set it
>>>>>> to en for all document in which it make sense
>>>>>> 3) We force default language to "en" in the XAR plugin
>>>>>>
>>>>>> WDYT ?
>>>>>>
>>>>>> I don't like too much 1) since some technical document could
really
> be
>>>>>> seen has having no default language, some document without any
> literal
>>>>>> content. But it's more a -0 than a -1, I understand other
would want
>>>>>> this for simplicity.
>>>>>>
>>>>>> About 3) as I said having a default language empty is a valid
use
> case
>>>>>> IMO so -0 for this one to. Still a bit better than 1) since the
use
>>>>>> case is still possible.
>>>>>>
>>>>>> +1 for 2)
>>>>>
>>>>> Neither option is good in general. The main problem is that most
>>>>> documents are written in the "Velocity" language, not in
the "English"
>>>>> language, meaning that it only contains code (which won't be seen
by
> the
>>>>> user), and translations, which depend on a lot of factors. It's
not
> good
>>>>> to say that the default language of a dynamically translated
document
> is
>>>>> en, since a wiki configured with a different language will only
> display
>>>>> them in that language, never in en.
>>>>>
>>>>> There are only a few documents that contain real text (normally only
> the
>>>>> sandbox should have real text, everything else should be localized),
> and
>>>>> for those it's OK to specify the actual language.
>>>>
>>>> I'm not sure I agree with this vision. It really depends on the use
> case. So far we haven't found a perfect solution.
>>>>
>>>> Some pages will have more code than content, others will have more
> content than code. For the former, keys are best and for the latter
> translations are best.
>>>>
>>>> In any case I don't understand the problem. What is the issue with
> saying that all our pages are in English by default. If a wiki is
> configured to be in another language and there's no translation for that
> language the default language (ie "English") will be used.
>>>
>>> There is no actual text in the document. How can you say that the
>>> language of
>>>
>
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
>>> is English, since there's no English sentence in there? Depending on the
>>> configuration, the same document will appear in German, Chinese, even
>>> Klingon, without changing anything in the document, so it is definitely
>>> not an English document.
>>>
>>> Scenario: Set up a new XWiki instance, and change the default language
>>> of the wiki to German. When you browse the wiki, everything is in
>>> German. Yet all the documents say that they're in English.
>>>
>>> Problem 1: The wiki is indexed as English text, so searching for text
>>> that the user actually sees in the wiki won't return any results.
>>>
>>> Problem 2: Editing such a document will automatically create a
>>> translation, since the original document is in English, and the user
>>> wants to edit a German document. Since the two languages are not
>>> compatible, a translation will be created automatically. Now the code
>>> has been forked, and automatic updates using the Distribution Wizard
>>> will update the hidden English document, since that is the default one,
>>> while the forked translation will stay behind.
>>
>> Thanks a lot for describing these 2 use cases that I definitely wouldn't
> have thought about! That's very useful.
>>
>> So it seems that suddenly it's becoming more complex ;)
>>
>> Basically it means that if we have documents that mix content and
> scripting we're going to have issues:
>> * Either they're marked as having no default language and the english
> content will be indexed in the default language of the wiki
>> * Either they're marked as "en" and the user will not have the
scripts
> in the search results and the DW/EM will update only the default version if
> the user has created a translation
>>
>> I'm sure we have lots of cases like this, the easiest one being the main
> home page:
>>
>> -------------------
>> It's an easy-to-edit website that will help you work better together.
> This Wiki is made of //pages// sorted by //spaces//. You're currently in
> the **Main** space, looking at its home page (**WebHome**).
>>
>> Learn how to use XWiki with the {{velocity}}[[Getting Started
> Guide>>
>
http://enterprise.xwiki.org/xwiki/bin/view/GettingStarted/WebHome?version=$…
> .
>>
>> {{velocity}}
>> #if($hasEdit)You can then use the [[Sandbox
> space>>Sandbox.WebHome]] to try out your wiki's features.#end
>> {{/velocity}}
>> -------------------
>>
>> It has both content and script… If it's marked as "en" then if the
user
> searches for "hasEdit" he won't get it if his wiki is in a language
other
> than "en".
>>
>> Unless, if there's no translation in the language of the user then we
> return the default language results for that page. Would that make sense?
>>
>> But there's still the issue of editing the page, which will create a
> translation and then imagine that we replace the velocity script in a
> future version, then the user will only get his default page updated and
> not his translation… However that's a general problem I guess...
>>
>> If it has no default language (as is the case now BTW) then it seems
> less of an issue it seems. It just means:
>> * If user searches for "work" he'll get result even though he's
no in an
> "en" wiki. But then he's searching for an english word too ;)
>> * any other downside?
>>
>> In view of all this, it seems that not setting any default language is a
> lesser evil, doesn't it?
>>
>> Thanks
>> -Vincent
>>
>>> That is why I'm saying that this kind of documents don't have a
>>> language, and they never should have. They adapt themselves to the
>>> user's language, so they're written in no language, yet they can
match
>>> all languages.
>>>
>>> Of course, not all documents are like this, as I originally stated
>>> myself, and there are valid cases where documents should have "en"
as
>>> the default translation.
>>>
>>>> What am I missing?
>>>>
>>>> (I'm not commenting on anything below yet because I feel it's
> important to agree on what's before first)
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>>> Other options:
>>>>>
>>>>> 4) Detect somehow localized documents and index:
>>>>> - the raw content using a non-language-specific analyzer
>>>>> - the content translated into all the languages registered in the
>>>>> administration, each with the proper language-specific analyzer, if
> they
>>>>> are supported by Solr; this includes the default wiki language.
>>>>> 4a) localized document = the default language is empty
>>>>> 4b) localized document = the default language is literally
"localized"
>>>>> 4c) add another document flag for marking localized documents
>>>>>
>>>>> 5) When the defaultLanguage is empty, render in the configured wiki
>>>>> default language
>>>>>
>>>>>
>>>>> I like 4) since it makes localized documents really searchable in
all
>>>>> the languages "supported" by that wiki instance.
>>>>>
>>>>> 4a) is a behavior change, so it might cause some trouble
>>>>> 4b) is the safest and requires the least amount of changes
>>>>> The number of document fields is increasing, so I'm not that fond
of
> 4c)
11:25 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Tue, Jun 11, 2013 at 11:21 AM, Vincent Massol <vincent(a)massol.net> wrote:
On Jun 11, 2013, at 11:09 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
Right now only raw content is indexed mostly because it's not easy to
index rendered content (most of it is dynamic and some are simply
dangerous or really not intended to be executed in the content of some
deamon thread, this is not a new issue). But putting aside the current
limitation I don't think indexing raw content is useless and saying
that scripting is english because the syntax looks like english and
simply nonsense...
Thanks Sergui for catching up here, while I was
off.
This is a situation I claim since a long time now, which is that some
document has simply no language, because all text displayed by those
document are simply produced by the localization module. Not only the
search issue are affected, but also the display of language selection in a
multi-wiki, and currently this is not as nice as it should be.
IMO, with a wiki that could produce with a single source documents,
displayed document in all available translation of the wiki (like the UI
does), we need to have the notion of a "no language" or "any
language"
document. In my own projects, I have been able to manage that properly
with the empty "default language" and empty "language" case. Since
having
both is stupid, we will surely merge those column in the future, but we
need to keep the idea of a "no language" or "any language" document,
whatever you see it, and to properly manage it, not only for our own XAR
but for user produced documents as well.
For those reasons, I am -1 for 1) and 3), and +1 for 2)
BTW I forgot to say in the my previous reply but obviously I remove my -1 for 2) since
it's much more complex than I thought.
I follow Thomas, the Welcome page is not a good
example, since this on has
really a translation in all languages.
I do not understand the intend to index code statement in an index, this is
the rendered document that should be indexed. And for "any language"
document, it should be indexed separately for each language enable in a
multi-language wiki.
ok, that's something very important to decide.
I think the 2 use cases are valide; xwiki can be used both by end users who only care
about pure content and it can also be used by developers who develop wiki pages with
scripts inside. For the former they want to see only content result and for the latter
they want to see script results. For example, as a dev I want to see where I call such
velocity macro, or where I use such rendering macro.
So I really believe that we need to index both types of content: rendered and raw.
Then we need to decide how to present that in the UI, but it could an option in the
advanced search to include raw results too for example (and remove duplications…).
--
Thomas Mortagne
So, to respond to Vincent, using properly the
"any language" case by
clearing the default language where needed, is the right way to go, but we
also need to manage that special case properly elsewhere (indexing,
language selection on the page,…)
So what's the rule for putting <defaultLanguage>en</defaultLanguage>?
Whenever there's a page having at least one word of English in the rendered result?
Thanks
-Vincent
On Tue, Jun 11, 2013 at 10:54 AM, Thomas Mortagne
<thomas.mortagne(a)xwiki.com
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs wrote:
> That's not the home page, that's the Welcome page and it's not a very
> good example since this page do have translations already so we
> already decided what we wanted for this pages in practice: each
> translation of the page copy the scripts.
>
> On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net>
wrote:
>> Hi Sergiu,
>>
>> On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
>>
>>> On 06/10/2013 03:12 PM, Vincent Massol wrote:
>>>>
>>>> On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
>>>>
>>>>> On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
>>>>>> Hi devs,
>>>>>>
>>>>>> Right now the XAR plugin format goal systematically empty the
>>>>>> <defaultLanguage> property.
>>>>>>
>>>>>> This is wrong IMO since it means we have no idea what is the
default
>>>>>> document language, it was not too visible before but it's
really not
>>>>>> very nice for things like the localization module and especially
SOLR
>>>>>> which store deferently the content depending on the language
(stop
>>>>>> words, etc).
>>>>>>
>>>>>> I see several possibilities:
>>>>>>
>>>>>> 1) We don't touch the XAR maven plugin and we state that when
default
>>>>>> language is not set, it's en (in the importer for example or
in
>>>>>> XWikiDocument#getDefaultLanguage)
>>>>>> 2) We stop filtering default language in the XAR plugin and we
set it
>>>>>> to en for all document in which it make sense
>>>>>> 3) We force default language to "en" in the XAR plugin
>>>>>>
>>>>>> WDYT ?
>>>>>>
>>>>>> I don't like too much 1) since some technical document could
really
> be
>>>>>> seen has having no default language, some document without any
> literal
>>>>>> content. But it's more a -0 than a -1, I understand other
would want
>>>>>> this for simplicity.
>>>>>>
>>>>>> About 3) as I said having a default language empty is a valid
use
> case
>>>>>> IMO so -0 for this one to. Still a bit better than 1) since the
use
>>>>>> case is still possible.
>>>>>>
>>>>>> +1 for 2)
>>>>>
>>>>> Neither option is good in general. The main problem is that most
>>>>> documents are written in the "Velocity" language, not in
the "English"
>>>>> language, meaning that it only contains code (which won't be seen
by
> the
>>>>> user), and translations, which depend on a lot of factors. It's
not
> good
>>>>> to say that the default language of a dynamically translated
document
> is
>>>>> en, since a wiki configured with a different language will only
> display
>>>>> them in that language, never in en.
>>>>>
>>>>> There are only a few documents that contain real text (normally only
> the
>>>>> sandbox should have real text, everything else should be localized),
> and
>>>>> for those it's OK to specify the actual language.
>>>>
>>>> I'm not sure I agree with this vision. It really depends on the use
> case. So far we haven't found a perfect solution.
>>>>
>>>> Some pages will have more code than content, others will have more
> content than code. For the former, keys are best and for the latter
> translations are best.
>>>>
>>>> In any case I don't understand the problem. What is the issue with
> saying that all our pages are in English by default. If a wiki is
> configured to be in another language and there's no translation for that
> language the default language (ie "English") will be used.
>>>
>>> There is no actual text in the document. How can you say that the
>>> language of
>>>
>
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
>>> is English, since there's no English sentence in there? Depending on the
>>> configuration, the same document will appear in German, Chinese, even
>>> Klingon, without changing anything in the document, so it is definitely
>>> not an English document.
>>>
>>> Scenario: Set up a new XWiki instance, and change the default language
>>> of the wiki to German. When you browse the wiki, everything is in
>>> German. Yet all the documents say that they're in English.
>>>
>>> Problem 1: The wiki is indexed as English text, so searching for text
>>> that the user actually sees in the wiki won't return any results.
>>>
>>> Problem 2: Editing such a document will automatically create a
>>> translation, since the original document is in English, and the user
>>> wants to edit a German document. Since the two languages are not
>>> compatible, a translation will be created automatically. Now the code
>>> has been forked, and automatic updates using the Distribution Wizard
>>> will update the hidden English document, since that is the default one,
>>> while the forked translation will stay behind.
>>
>> Thanks a lot for describing these 2 use cases that I definitely wouldn't
> have thought about! That's very useful.
>>
>> So it seems that suddenly it's becoming more complex ;)
>>
>> Basically it means that if we have documents that mix content and
> scripting we're going to have issues:
>> * Either they're marked as having no default language and the english
> content will be indexed in the default language of the wiki
>> * Either they're marked as "en" and the user will not have the
scripts
> in the search results and the DW/EM will update only the default version if
> the user has created a translation
>>
>> I'm sure we have lots of cases like this, the easiest one being the main
> home page:
>>
>> -------------------
>> It's an easy-to-edit website that will help you work better together.
> This Wiki is made of //pages// sorted by //spaces//. You're currently in
> the **Main** space, looking at its home page (**WebHome**).
>>
>> Learn how to use XWiki with the {{velocity}}[[Getting Started
> Guide>>
>
http://enterprise.xwiki.org/xwiki/bin/view/GettingStarted/WebHome?version=$…
> .
>>
>> {{velocity}}
>> #if($hasEdit)You can then use the [[Sandbox
> space>>Sandbox.WebHome]] to try out your wiki's features.#end
>> {{/velocity}}
>> -------------------
>>
>> It has both content and script… If it's marked as "en" then if the
user
> searches for "hasEdit" he won't get it if his wiki is in a language
other
> than "en".
>>
>> Unless, if there's no translation in the language of the user then we
> return the default language results for that page. Would that make sense?
>>
>> But there's still the issue of editing the page, which will create a
> translation and then imagine that we replace the velocity script in a
> future version, then the user will only get his default page updated and
> not his translation… However that's a general problem I guess...
>>
>> If it has no default language (as is the case now BTW) then it seems
> less of an issue it seems. It just means:
>> * If user searches for "work" he'll get result even though he's
no in an
> "en" wiki. But then he's searching for an english word too ;)
>> * any other downside?
>>
>> In view of all this, it seems that not setting any default language is a
> lesser evil, doesn't it?
>>
>> Thanks
>> -Vincent
>>
>>> That is why I'm saying that this kind of documents don't have a
>>> language, and they never should have. They adapt themselves to the
>>> user's language, so they're written in no language, yet they can
match
>>> all languages.
>>>
>>> Of course, not all documents are like this, as I originally stated
>>> myself, and there are valid cases where documents should have "en"
as
>>> the default translation.
>>>
>>>> What am I missing?
>>>>
>>>> (I'm not commenting on anything below yet because I feel it's
> important to agree on what's before first)
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>>> Other options:
>>>>>
>>>>> 4) Detect somehow localized documents and index:
>>>>> - the raw content using a non-language-specific analyzer
>>>>> - the content translated into all the languages registered in the
>>>>> administration, each with the proper language-specific analyzer, if
> they
>>>>> are supported by Solr; this includes the default wiki language.
>>>>> 4a) localized document = the default language is empty
>>>>> 4b) localized document = the default language is literally
"localized"
>>>>> 4c) add another document flag for marking localized documents
>>>>>
>>>>> 5) When the defaultLanguage is empty, render in the configured wiki
>>>>> default language
>>>>>
>>>>>
>>>>> I like 4) since it makes localized documents really searchable in
all
>>>>> the languages "supported" by that wiki instance.
>>>>>
>>>>> 4a) is a behavior change, so it might cause some trouble
>>>>> 4b) is the safest and requires the least amount of changes
>>>>> The number of document fields is increasing, so I'm not that fond
of
> 4c)
11:32 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Jun 11, 2013, at 11:25 AM, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote:
On Tue, Jun 11, 2013 at 11:21 AM, Vincent Massol
<vincent(a)massol.net> wrote:
Yes I agree it's hard to index rendered content perfectly...
I was just replying to Denis who was saying the opposite, i.e. that we should only index
rendered content. I see value in indexing the raw content for the reasons I pointed out
above.
I agree that ideally non tech users shouldn't see in their results some script
portions… One thing we could do could be to render without executing transformations and
only index that content. It's not perfect but it could be a first try to filter out
tech content for simple users.
Thanks
-Vincent
On Jun 11, 2013, at 11:09 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
Right now only raw content is indexed mostly because it's not easy to
index rendered content (most of it is dynamic and some are simply
dangerous or really not intended to be executed in the content of some
deamon thread, this is not a new issue). But putting aside the current
limitation I don't think indexing raw content is useless and saying
that scripting is english because the syntax looks like english and
simply nonsense… Thanks Sergui for catching up here, while I was
off.
This is a situation I claim since a long time now, which is that some
document has simply no language, because all text displayed by those
document are simply produced by the localization module. Not only the
search issue are affected, but also the display of language selection in a
multi-wiki, and currently this is not as nice as it should be.
IMO, with a wiki that could produce with a single source documents,
displayed document in all available translation of the wiki (like the UI
does), we need to have the notion of a "no language" or "any
language"
document. In my own projects, I have been able to manage that properly
with the empty "default language" and empty "language" case. Since
having
both is stupid, we will surely merge those column in the future, but we
need to keep the idea of a "no language" or "any language" document,
whatever you see it, and to properly manage it, not only for our own XAR
but for user produced documents as well.
For those reasons, I am -1 for 1) and 3), and +1 for 2)
BTW I forgot to say in the my previous reply but obviously I remove my -1 for 2) since
it's much more complex than I thought.
I follow Thomas, the Welcome page is not a good
example, since this on has
really a translation in all languages.
I do not understand the intend to index code statement in an index, this is
the rendered document that should be indexed. And for "any language"
document, it should be indexed separately for each language enable in a
multi-language wiki.
ok, that's something very important to decide.
I think the 2 use cases are valide; xwiki can be used both by end users who only care
about pure content and it can also be used by developers who develop wiki pages with
scripts inside. For the former they want to see only content result and for the latter
they want to see script results. For example, as a dev I want to see where I call such
velocity macro, or where I use such rendering macro.
So I really believe that we need to index both types of content: rendered and raw. > Then we need to decide how to present that in the
UI, but it could an option in the advanced search to include raw results too for example
(and remove duplications…).
>
>> So, to respond to Vincent, using properly the "any language" case by
>> clearing the default language where needed, is the right way to go, but we
>> also need to manage that special case properly elsewhere (indexing,
>> language selection on the page,…)
>
> So what's the rule for putting <defaultLanguage>en</defaultLanguage>?
Whenever there's a page having at least one word of English in the rendered result?
>
> Thanks
> -Vincent
>
>> On Tue, Jun 11, 2013 at 10:54 AM, Thomas Mortagne <thomas.mortagne(a)xwiki.com
>>> wrote:
>>
>>> That's not the home page, that's the Welcome page and it's not a
very
>>> good example since this page do have translations already so we
>>> already decided what we wanted for this pages in practice: each
>>> translation of the page copy the scripts.
>>>
>>> On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net>
>>> wrote:
>>>> Hi Sergiu,
>>>>
>>>> On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org>
wrote:
>>>>
>>>>> On 06/10/2013 03:12 PM, Vincent Massol wrote:
>>>>>>
>>>>>> On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu
<sergiu(a)xwiki.org> wrote:
>>>>>>
>>>>>>> On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
>>>>>>>> Hi devs,
>>>>>>>>
>>>>>>>> Right now the XAR plugin format goal systematically empty
the
>>>>>>>> <defaultLanguage> property.
>>>>>>>>
>>>>>>>> This is wrong IMO since it means we have no idea what is
the default
>>>>>>>> document language, it was not too visible before but
it's really not
>>>>>>>> very nice for things like the localization module and
especially SOLR
>>>>>>>> which store deferently the content depending on the
language (stop
>>>>>>>> words, etc).
>>>>>>>>
>>>>>>>> I see several possibilities:
>>>>>>>>
>>>>>>>> 1) We don't touch the XAR maven plugin and we state
that when default
>>>>>>>> language is not set, it's en (in the importer for
example or in
>>>>>>>> XWikiDocument#getDefaultLanguage)
>>>>>>>> 2) We stop filtering default language in the XAR plugin
and we set it
>>>>>>>> to en for all document in which it make sense
>>>>>>>> 3) We force default language to "en" in the XAR
plugin
>>>>>>>>
>>>>>>>> WDYT ?
>>>>>>>>
>>>>>>>> I don't like too much 1) since some technical
document could really
>>> be
>>>>>>>> seen has having no default language, some document
without any
>>> literal
>>>>>>>> content. But it's more a -0 than a -1, I understand
other would want
>>>>>>>> this for simplicity.
>>>>>>>>
>>>>>>>> About 3) as I said having a default language empty is a
valid use
>>> case
>>>>>>>> IMO so -0 for this one to. Still a bit better than 1)
since the use
>>>>>>>> case is still possible.
>>>>>>>>
>>>>>>>> +1 for 2)
>>>>>>>
>>>>>>> Neither option is good in general. The main problem is that
most
>>>>>>> documents are written in the "Velocity" language,
not in the "English"
>>>>>>> language, meaning that it only contains code (which won't
be seen by
>>> the
>>>>>>> user), and translations, which depend on a lot of factors.
It's not
>>> good
>>>>>>> to say that the default language of a dynamically translated
document
>>> is
>>>>>>> en, since a wiki configured with a different language will
only
>>> display
>>>>>>> them in that language, never in en.
>>>>>>>
>>>>>>> There are only a few documents that contain real text
(normally only
>>> the
>>>>>>> sandbox should have real text, everything else should be
localized),
>>> and
>>>>>>> for those it's OK to specify the actual language.
>>>>>>
>>>>>> I'm not sure I agree with this vision. It really depends on
the use
>>> case. So far we haven't found a perfect solution.
>>>>>>
>>>>>> Some pages will have more code than content, others will have
more
>>> content than code. For the former, keys are best and for the latter
>>> translations are best.
>>>>>>
>>>>>> In any case I don't understand the problem. What is the issue
with
>>> saying that all our pages are in English by default. If a wiki is
>>> configured to be in another language and there's no translation for that
>>> language the default language (ie "English") will be used.
>>>>>
>>>>> There is no actual text in the document. How can you say that the
>>>>> language of
>>>>>
>>>
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
>>>>> is English, since there's no English sentence in there? Depending
on the
>>>>> configuration, the same document will appear in German, Chinese,
even
>>>>> Klingon, without changing anything in the document, so it is
definitely
>>>>> not an English document.
>>>>>
>>>>> Scenario: Set up a new XWiki instance, and change the default
language
>>>>> of the wiki to German. When you browse the wiki, everything is in
>>>>> German. Yet all the documents say that they're in English.
>>>>>
>>>>> Problem 1: The wiki is indexed as English text, so searching for
text
>>>>> that the user actually sees in the wiki won't return any
results.
>>>>>
>>>>> Problem 2: Editing such a document will automatically create a
>>>>> translation, since the original document is in English, and the user
>>>>> wants to edit a German document. Since the two languages are not
>>>>> compatible, a translation will be created automatically. Now the
code
>>>>> has been forked, and automatic updates using the Distribution Wizard
>>>>> will update the hidden English document, since that is the default
one,
>>>>> while the forked translation will stay behind.
>>>>
>>>> Thanks a lot for describing these 2 use cases that I definitely
wouldn't
>>> have thought about! That's very useful.
>>>>
>>>> So it seems that suddenly it's becoming more complex ;)
>>>>
>>>> Basically it means that if we have documents that mix content and
>>> scripting we're going to have issues:
>>>> * Either they're marked as having no default language and the
english
>>> content will be indexed in the default language of the wiki
>>>> * Either they're marked as "en" and the user will not have
the scripts
>>> in the search results and the DW/EM will update only the default version if
>>> the user has created a translation
>>>>
>>>> I'm sure we have lots of cases like this, the easiest one being the
main
>>> home page:
>>>>
>>>> -------------------
>>>> It's an easy-to-edit website that will help you work better
together.
>>> This Wiki is made of //pages// sorted by //spaces//. You're currently in
>>> the **Main** space, looking at its home page (**WebHome**).
>>>>
>>>> Learn how to use XWiki with the {{velocity}}[[Getting Started
>>> Guide>>
>>>
http://enterprise.xwiki.org/xwiki/bin/view/GettingStarted/WebHome?version=$…
>>> .
>>>>
>>>> {{velocity}}
>>>> #if($hasEdit)You can then use the [[Sandbox
>>> space>>Sandbox.WebHome]] to try out your wiki's
features.#end
>>>> {{/velocity}}
>>>> -------------------
>>>>
>>>> It has both content and script… If it's marked as "en" then
if the user
>>> searches for "hasEdit" he won't get it if his wiki is in a
language other
>>> than "en".
>>>>
>>>> Unless, if there's no translation in the language of the user then
we
>>> return the default language results for that page. Would that make sense?
>>>>
>>>> But there's still the issue of editing the page, which will create a
>>> translation and then imagine that we replace the velocity script in a
>>> future version, then the user will only get his default page updated and
>>> not his translation… However that's a general problem I guess...
>>>>
>>>> If it has no default language (as is the case now BTW) then it seems
>>> less of an issue it seems. It just means:
>>>> * If user searches for "work" he'll get result even though
he's no in an
>>> "en" wiki. But then he's searching for an english word too ;)
>>>> * any other downside?
>>>>
>>>> In view of all this, it seems that not setting any default language is a
>>> lesser evil, doesn't it?
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>>> That is why I'm saying that this kind of documents don't have
a
>>>>> language, and they never should have. They adapt themselves to the
>>>>> user's language, so they're written in no language, yet they
can match
>>>>> all languages.
>>>>>
>>>>> Of course, not all documents are like this, as I originally stated
>>>>> myself, and there are valid cases where documents should have
"en" as
>>>>> the default translation.
>>>>>
>>>>>> What am I missing?
>>>>>>
>>>>>> (I'm not commenting on anything below yet because I feel
it's
>>> important to agree on what's before first)
>>>>>>
>>>>>> Thanks
>>>>>> -Vincent
>>>>>>
>>>>>>> Other options:
>>>>>>>
>>>>>>> 4) Detect somehow localized documents and index:
>>>>>>> - the raw content using a non-language-specific analyzer
>>>>>>> - the content translated into all the languages registered in
the
>>>>>>> administration, each with the proper language-specific
analyzer, if
>>> they
>>>>>>> are supported by Solr; this includes the default wiki
language.
>>>>>>> 4a) localized document = the default language is empty
>>>>>>> 4b) localized document = the default language is literally
"localized"
>>>>>>> 4c) add another document flag for marking localized
documents
>>>>>>>
>>>>>>> 5) When the defaultLanguage is empty, render in the
configured wiki
>>>>>>> default language
>>>>>>>
>>>>>>>
>>>>>>> I like 4) since it makes localized documents really
searchable in all
>>>>>>> the languages "supported" by that wiki instance.
>>>>>>>
>>>>>>> 4a) is a behavior change, so it might cause some trouble
>>>>>>> 4b) is the safest and requires the least amount of changes
>>>>>>> The number of document fields is increasing, so I'm not
that fond of
>>> 4c)
11:36 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Tue, Jun 11, 2013 at 11:32 AM, Vincent Massol <vincent(a)massol.net> wrote:
On Jun 11, 2013, at 11:25 AM, Thomas Mortagne <thomas.mortagne(a)xwiki.com>
wrote:
And I was just telling what end user really expect, I know that there is
issues with indexing rendered content, but there is probably something that
could be done to improve that.
On Tue, Jun 11, 2013 at 11:21 AM, Vincent Massol
<vincent(a)massol.net>
wrote:
>
> On Jun 11, 2013, at 11:09 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
>
>> Thanks Sergui for catching up here, while I was off.
>> This is a situation I claim since a long time now, which is that some
>> document has simply no language, because all text displayed by those
>> document are simply produced by the localization module. Not only the
>> search issue are affected, but also the display of language selection
in
a
>> multi-wiki, and currently this is not as
nice as it should be.
>>
>> IMO, with a wiki that could produce with a single source documents,
>> displayed document in all available translation of the wiki (like the
UI
>> does), we need to have the notion of a
"no language" or "any language"
>> document. In my own projects, I have been able to manage that properly
>> with the empty "default language" and empty "language" case.
Since
having
>> both is stupid, we will surely merge
those column in the future, but we
>> need to keep the idea of a "no language" or "any language"
document,
>> whatever you see it, and to properly manage it, not only for our own
XAR
>> but for user produced documents as well.
>>
>> For those reasons, I am -1 for 1) and 3), and +1 for 2)
>
> BTW I forgot to say in the my previous reply but obviously I remove my
-1 for
2) since it's much more complex than I thought.
>
>> I follow Thomas, the Welcome page is not a good example, since this on
has
>> really a translation in all languages.
>> I do not understand the intend to index code statement in an index,
this
is
>> the rendered document that should be
indexed. And for "any language"
>> document, it should be indexed separately for each language enable in a
>> multi-language wiki.
>
> ok, that's something very important to decide.
>
> I think the 2 use cases are valide; xwiki can be used both by end users
who
only care about pure content and it can also be used by developers who
develop wiki pages with scripts inside. For the former they want to see
only content result and for the latter they want to see script results. For
example, as a dev I want to see where I call such velocity macro, or where
I use such rendering macro.
>
> So I really believe that we need to index both types of content:
rendered and
raw.
Right now only raw content is indexed mostly because it's not easy to
index rendered content (most of it is dynamic and some are simply
dangerous or really not intended to be executed in the content of some
deamon thread, this is not a new issue). But putting aside the current
limitation I don't think indexing raw content is useless and saying
that scripting is english because the syntax looks like english and
simply nonsense…
Yes I agree it's hard to index rendered content perfectly...
I was just replying to Denis who was saying the opposite, i.e. that we
should only index rendered content. I see value in indexing the raw content
for the reasons I pointed out above.
I agree that ideally non tech users shouldn't see in their results some
script portions… One thing we could do could be to render without executing
transformations and only index that content. It's not perfect but it could
be a first try to filter out tech content for simple users.
Thanks
-Vincent
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
> Then we need to decide how to present that in
the UI, but it could an
option in the advanced search to include raw results too
for example (and
remove duplications…).
>
>> So, to respond to Vincent, using properly the "any language" case by
>> clearing the default language where needed, is the right way to go,
but
we
>> also need to manage that special case
properly elsewhere (indexing,
>> language selection on the page,…)
>
> So what's the rule for putting <defaultLanguage>en</defaultLanguage>?
Whenever there's a page having at least one word of English in the rendered
result?
>
> Thanks
> -Vincent
>
>> On Tue, Jun 11, 2013 at 10:54 AM, Thomas Mortagne <
thomas.mortagne(a)xwiki.com
>>> wrote:
>>
>>> That's not the home page, that's the Welcome page and it's not a
very
>>> good example since this page do have translations already so we
>>> already decided what we wanted for this pages in practice: each
>>> translation of the page copy the scripts.
>>>
>>> On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net>
>>> wrote:
>>>> Hi Sergiu,
>>>>
>>>> On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org>
wrote:
>>>>
>>>>> On 06/10/2013 03:12 PM, Vincent Massol wrote:
>>>>>>
>>>>>> On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu
<sergiu(a)xwiki.org>
wrote:
>>>>>>
>>>>>>> On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
>>>>>>>> Hi devs,
>>>>>>>>
>>>>>>>> Right now the XAR plugin format goal systematically empty
the
>>>>>>>> <defaultLanguage> property.
>>>>>>>>
>>>>>>>> This is wrong IMO since it means we have no idea what is
the
default
>>>>>>>> document
language, it was not too visible before but it's really
not
>>>>>>>> very nice for
things like the localization module and especially
SOLR
>>>>>>>> which store
deferently the content depending on the language
(stop
>>>>>>>> words, etc).
>>>>>>>>
>>>>>>>> I see several possibilities:
>>>>>>>>
>>>>>>>> 1) We don't touch the XAR maven plugin and we state
that when
default
>>>>>>>> language is not
set, it's en (in the importer for example or in
>>>>>>>> XWikiDocument#getDefaultLanguage)
>>>>>>>> 2) We stop filtering default language in the XAR plugin
and we
set it
>>>>>>>> to en for all
document in which it make sense
>>>>>>>> 3) We force default language to "en" in the XAR
plugin
>>>>>>>>
>>>>>>>> WDYT ?
>>>>>>>>
>>>>>>>> I don't like too much 1) since some technical
document could
really
>>> be
>>>>>>>> seen has having no default language, some document
without any
>>> literal
>>>>>>>> content. But it's more a -0 than a -1, I understand
other would
want
>>>>>>>> this for
simplicity.
>>>>>>>>
>>>>>>>> About 3) as I said having a default language empty is a
valid use
>>> case
>>>>>>>> IMO so -0 for this one to. Still a bit better than 1)
since the
use
>>>>>>>> case is still
possible.
>>>>>>>>
>>>>>>>> +1 for 2)
>>>>>>>
>>>>>>> Neither option is good in general. The main problem is that
most
>>>>>>> documents are written in the "Velocity" language,
not in the
"English"
>>>>>>> language, meaning
that it only contains code (which won't be seen
by
>>> the
>>>>>>> user), and translations, which depend on a lot of factors.
It's
not
>>> good
>>>>>>> to say that the default language of a dynamically translated
document
>>> is
>>>>>>> en, since a wiki configured with a different language will
only
>>> display
>>>>>>> them in that language, never in en.
>>>>>>>
>>>>>>> There are only a few documents that contain real text
(normally
only
>>> the
>>>>>>> sandbox should have real text, everything else should be
localized),
>>> and
>>>>>>> for those it's OK to specify the actual language.
>>>>>>
>>>>>> I'm not sure I agree with this vision. It really depends on
the use
>>> case. So far we haven't found a perfect solution.
>>>>>>
>>>>>> Some pages will have more code than content, others will have
more
>>> content than code. For the former, keys are best and for the latter
>>> translations are best.
>>>>>>
>>>>>> In any case I don't understand the problem. What is the issue
with
>>> saying that all our pages are in English by default. If a wiki is
>>> configured to be in another language and there's no translation for
that
>>> language the default language (ie
"English") will be used.
>>>>>
>>>>> There is no actual text in the document. How can you say that the
>>>>> language of
>>>>>
>>>
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
>>>>> is English, since there's
no English sentence in there? Depending
on the
>>>>> configuration, the same
document will appear in German, Chinese,
even
>>>>> Klingon, without changing
anything in the document, so it is
definitely
>>>>> not an English document.
>>>>>
>>>>> Scenario: Set up a new XWiki instance, and change the default
language
>>>>> of the wiki to German. When
you browse the wiki, everything is in
>>>>> German. Yet all the documents say that they're in English.
>>>>>
>>>>> Problem 1: The wiki is indexed as English text, so searching for
text
>>>>> that the user actually sees
in the wiki won't return any results.
>>>>>
>>>>> Problem 2: Editing such a document will automatically create a
>>>>> translation, since the original document is in English, and the user
>>>>> wants to edit a German document. Since the two languages are not
>>>>> compatible, a translation will be created automatically. Now the
code
>>>>> has been forked, and
automatic updates using the Distribution Wizard
>>>>> will update the hidden English document, since that is the default
one,
>>>>> while the forked translation
will stay behind.
>>>>
>>>> Thanks a lot for describing these 2 use cases that I definitely
wouldn't
>>> have thought about! That's very
useful.
>>>>
>>>> So it seems that suddenly it's becoming more complex ;)
>>>>
>>>> Basically it means that if we have documents that mix content and
>>> scripting we're going to have issues:
>>>> * Either they're marked as having no default language and the
english
>>> content will be indexed in the default language of the wiki
>>>> * Either they're marked as "en" and the user will not have
the
scripts
>>> in the search results and the DW/EM
will update only the default
version if
>>> the user has created a translation
>>>>
>>>> I'm sure we have lots of cases like this, the easiest one being the
main
>>> home page:
>>>>
>>>> -------------------
>>>> It's an easy-to-edit website that will help you work better
together.
>>> This Wiki is made of //pages// sorted by //spaces//. You're currently
in
>>> the **Main** space, looking at its
home page (**WebHome**).
>>>>
>>>> Learn how to use XWiki with the {{velocity}}[[Getting Started
>>> Guide>>
>>>
http://enterprise.xwiki.org/xwiki/bin/view/GettingStarted/WebHome?version=$…
>>> .
>>>>
>>>> {{velocity}}
>>>> #if($hasEdit)You can then use the [[Sandbox
>>> space>>Sandbox.WebHome]] to try out your wiki's
features.#end
>>>> {{/velocity}}
>>>> -------------------
>>>>
>>>> It has both content and script… If it's marked as "en" then
if the
user
>>> searches for "hasEdit" he
won't get it if his wiki is in a language
other
>>> than "en".
>>>>
>>>> Unless, if there's no translation in the language of the user then
we
>>> return the default language results for that page. Would that make
sense?
>>>>
>>>> But there's still the issue of editing the page, which will create a
>>> translation and then imagine that we replace the velocity script in a
>>> future version, then the user will only get his default page updated
and
>>> not his translation… However
that's a general problem I guess...
>>>>
>>>> If it has no default language (as is the case now BTW) then it seems
>>> less of an issue it seems. It just means:
>>>> * If user searches for "work" he'll get result even though
he's no
in an
>>> "en" wiki. But then
he's searching for an english word too ;)
>>>> * any other downside?
>>>>
>>>> In view of all this, it seems that not setting any default language
is a
>>> lesser evil, doesn't it?
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>>> That is why I'm saying that this kind of documents don't have
a
>>>>> language, and they never should have. They adapt themselves to the
>>>>> user's language, so they're written in no language, yet they
can
match
>>>>> all languages.
>>>>>
>>>>> Of course, not all documents are like this, as I originally stated
>>>>> myself, and there are valid cases where documents should have
"en"
as
>>>>> the default translation.
>>>>>
>>>>>> What am I missing?
>>>>>>
>>>>>> (I'm not commenting on anything below yet because I feel
it's
>>> important to agree on what's before first)
>>>>>>
>>>>>> Thanks
>>>>>> -Vincent
>>>>>>
>>>>>>> Other options:
>>>>>>>
>>>>>>> 4) Detect somehow localized documents and index:
>>>>>>> - the raw content using a non-language-specific analyzer
>>>>>>> - the content translated into all the languages registered in
the
>>>>>>> administration, each with the proper language-specific
analyzer,
if
>>> they
>>>>>>> are supported by Solr; this includes the default wiki
language.
>>>>>>> 4a) localized document = the default language is empty
>>>>>>> 4b) localized document = the default language is literally
"localized"
>>>>>>> 4c) add another
document flag for marking localized documents
>>>>>>>
>>>>>>> 5) When the defaultLanguage is empty, render in the
configured
wiki
>>>>>>> default language
>>>>>>>
>>>>>>>
>>>>>>> I like 4) since it makes localized documents really
searchable in
all
>>>>>>> the languages
"supported" by that wiki instance.
>>>>>>>
>>>>>>> 4a) is a behavior change, so it might cause some trouble
>>>>>>> 4b) is the safest and requires the least amount of changes
>>>>>>> The number of document fields is increasing, so I'm not
that fond
of
>>> 4c)
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
11:48 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Tue, Jun 11, 2013 at 11:36 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
--
Thomas Mortagne
On Tue, Jun 11, 2013 at 11:32 AM, Vincent Massol
<vincent(a)massol.net> wrote:
I did not said I'm against indexing rendered content, I'm explaining
why in many places we don't deal with rendered content just yet and
that indexing raw content is still useful.
On Jun 11, 2013, at 11:25 AM, Thomas Mortagne <thomas.mortagne(a)xwiki.com>
wrote:
And I was just telling what end user really expect, I know that there is
issues with indexing rendered content, but there is probably something that
could be done to improve that. On Tue, Jun 11, 2013 at 11:21 AM, Vincent Massol
<vincent(a)massol.net>
wrote:
>
> On Jun 11, 2013, at 11:09 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
>
>> Thanks Sergui for catching up here, while I was off.
>> This is a situation I claim since a long time now, which is that some
>> document has simply no language, because all text displayed by those
>> document are simply produced by the localization module. Not only the
>> search issue are affected, but also the display of language selection
in
a
>> multi-wiki, and currently this is not as
nice as it should be.
>>
>> IMO, with a wiki that could produce with a single source documents,
>> displayed document in all available translation of the wiki (like the
UI
>> does), we need to have the notion of a
"no language" or "any language"
>> document. In my own projects, I have been able to manage that properly
>> with the empty "default language" and empty "language" case.
Since
having
>> both is stupid, we will surely merge
those column in the future, but we
>> need to keep the idea of a "no language" or "any language"
document,
>> whatever you see it, and to properly manage it, not only for our own
XAR
>> but for user produced documents as well.
>>
>> For those reasons, I am -1 for 1) and 3), and +1 for 2)
>
> BTW I forgot to say in the my previous reply but obviously I remove my
-1 for
2) since it's much more complex than I thought.
>
>> I follow Thomas, the Welcome page is not a good example, since this on
has
>> really a translation in all languages.
>> I do not understand the intend to index code statement in an index,
this
is
>> the rendered document that should be
indexed. And for "any language"
>> document, it should be indexed separately for each language enable in a
>> multi-language wiki.
>
> ok, that's something very important to decide.
>
> I think the 2 use cases are valide; xwiki can be used both by end users
who
only care about pure content and it can also be used by developers who
develop wiki pages with scripts inside. For the former they want to see
only content result and for the latter they want to see script results. For
example, as a dev I want to see where I call such velocity macro, or where
I use such rendering macro.
>
> So I really believe that we need to index both types of content:
rendered and
raw.
Right now only raw content is indexed mostly because it's not easy to
index rendered content (most of it is dynamic and some are simply
dangerous or really not intended to be executed in the content of some
deamon thread, this is not a new issue). But putting aside the current
limitation I don't think indexing raw content is useless and saying
that scripting is english because the syntax looks like english and
simply nonsense…
Yes I agree it's hard to index rendered content perfectly...
I was just replying to Denis who was saying the opposite, i.e. that we
should only index rendered content. I see value in indexing the raw content
for the reasons I pointed out above.
I agree that ideally non tech users shouldn't see in their results some
script portions… One thing we could do could be to render without executing
transformations and only index that content. It's not perfect but it could
be a first try to filter out tech content for simple users.
Thanks
-Vincent
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs > Then we need to decide how to present that in
the UI, but it could an
option in the advanced search to include raw results too
for example (and
remove duplications…).
>
>> So, to respond to Vincent, using properly the "any language" case by
>> clearing the default language where needed, is the right way to go,
but
we
>> also need to manage that special case
properly elsewhere (indexing,
>> language selection on the page,…)
>
> So what's the rule for putting <defaultLanguage>en</defaultLanguage>?
Whenever there's a page having at least one word of English in the rendered
result?
>
> Thanks
> -Vincent
>
>> On Tue, Jun 11, 2013 at 10:54 AM, Thomas Mortagne <
thomas.mortagne(a)xwiki.com
>>> wrote:
>>
>>> That's not the home page, that's the Welcome page and it's not a
very
>>> good example since this page do have translations already so we
>>> already decided what we wanted for this pages in practice: each
>>> translation of the page copy the scripts.
>>>
>>> On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net>
>>> wrote:
>>>> Hi Sergiu,
>>>>
>>>> On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org>
wrote:
>>>>
>>>>> On 06/10/2013 03:12 PM, Vincent Massol wrote:
>>>>>>
>>>>>> On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu
<sergiu(a)xwiki.org>
wrote:
>>>>>>
>>>>>>> On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
>>>>>>>> Hi devs,
>>>>>>>>
>>>>>>>> Right now the XAR plugin format goal systematically empty
the
>>>>>>>> <defaultLanguage> property.
>>>>>>>>
>>>>>>>> This is wrong IMO since it means we have no idea what is
the
default
>>>>>>>> document
language, it was not too visible before but it's really
not
>>>>>>>> very nice for
things like the localization module and especially
SOLR
>>>>>>>> which store
deferently the content depending on the language
(stop
>>>>>>>> words, etc).
>>>>>>>>
>>>>>>>> I see several possibilities:
>>>>>>>>
>>>>>>>> 1) We don't touch the XAR maven plugin and we state
that when
default
>>>>>>>> language is not
set, it's en (in the importer for example or in
>>>>>>>> XWikiDocument#getDefaultLanguage)
>>>>>>>> 2) We stop filtering default language in the XAR plugin
and we
set it
>>>>>>>> to en for all
document in which it make sense
>>>>>>>> 3) We force default language to "en" in the XAR
plugin
>>>>>>>>
>>>>>>>> WDYT ?
>>>>>>>>
>>>>>>>> I don't like too much 1) since some technical
document could
really
>>> be
>>>>>>>> seen has having no default language, some document
without any
>>> literal
>>>>>>>> content. But it's more a -0 than a -1, I understand
other would
want
>>>>>>>> this for
simplicity.
>>>>>>>>
>>>>>>>> About 3) as I said having a default language empty is a
valid use
>>> case
>>>>>>>> IMO so -0 for this one to. Still a bit better than 1)
since the
use
>>>>>>>> case is still
possible.
>>>>>>>>
>>>>>>>> +1 for 2)
>>>>>>>
>>>>>>> Neither option is good in general. The main problem is that
most
>>>>>>> documents are written in the "Velocity" language,
not in the
"English"
>>>>>>> language, meaning
that it only contains code (which won't be seen
by
>>> the
>>>>>>> user), and translations, which depend on a lot of factors.
It's
not
>>> good
>>>>>>> to say that the default language of a dynamically translated
document
>>> is
>>>>>>> en, since a wiki configured with a different language will
only
>>> display
>>>>>>> them in that language, never in en.
>>>>>>>
>>>>>>> There are only a few documents that contain real text
(normally
only
>>> the
>>>>>>> sandbox should have real text, everything else should be
localized),
>>> and
>>>>>>> for those it's OK to specify the actual language.
>>>>>>
>>>>>> I'm not sure I agree with this vision. It really depends on
the use
>>> case. So far we haven't found a perfect solution.
>>>>>>
>>>>>> Some pages will have more code than content, others will have
more
>>> content than code. For the former, keys are best and for the latter
>>> translations are best.
>>>>>>
>>>>>> In any case I don't understand the problem. What is the issue
with
>>> saying that all our pages are in English by default. If a wiki is
>>> configured to be in another language and there's no translation for
that
>>> language the default language (ie
"English") will be used.
>>>>>
>>>>> There is no actual text in the document. How can you say that the
>>>>> language of
>>>>>
>>>
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
>>>>> is English, since there's
no English sentence in there? Depending
on the
>>>>> configuration, the same
document will appear in German, Chinese,
even
>>>>> Klingon, without changing
anything in the document, so it is
definitely
>>>>> not an English document.
>>>>>
>>>>> Scenario: Set up a new XWiki instance, and change the default
language
>>>>> of the wiki to German. When
you browse the wiki, everything is in
>>>>> German. Yet all the documents say that they're in English.
>>>>>
>>>>> Problem 1: The wiki is indexed as English text, so searching for
text
>>>>> that the user actually sees
in the wiki won't return any results.
>>>>>
>>>>> Problem 2: Editing such a document will automatically create a
>>>>> translation, since the original document is in English, and the user
>>>>> wants to edit a German document. Since the two languages are not
>>>>> compatible, a translation will be created automatically. Now the
code
>>>>> has been forked, and
automatic updates using the Distribution Wizard
>>>>> will update the hidden English document, since that is the default
one,
>>>>> while the forked translation
will stay behind.
>>>>
>>>> Thanks a lot for describing these 2 use cases that I definitely
wouldn't
>>> have thought about! That's very
useful.
>>>>
>>>> So it seems that suddenly it's becoming more complex ;)
>>>>
>>>> Basically it means that if we have documents that mix content and
>>> scripting we're going to have issues:
>>>> * Either they're marked as having no default language and the
english
>>> content will be indexed in the default language of the wiki
>>>> * Either they're marked as "en" and the user will not have
the
scripts
>>> in the search results and the DW/EM
will update only the default
version if
>>> the user has created a translation
>>>>
>>>> I'm sure we have lots of cases like this, the easiest one being the
main
>>> home page:
>>>>
>>>> -------------------
>>>> It's an easy-to-edit website that will help you work better
together.
>>> This Wiki is made of //pages// sorted by //spaces//. You're currently
in
>>> the **Main** space, looking at its
home page (**WebHome**).
>>>>
>>>> Learn how to use XWiki with the {{velocity}}[[Getting Started
>>> Guide>>
>>>
http://enterprise.xwiki.org/xwiki/bin/view/GettingStarted/WebHome?version=$…
>>> .
>>>>
>>>> {{velocity}}
>>>> #if($hasEdit)You can then use the [[Sandbox
>>> space>>Sandbox.WebHome]] to try out your wiki's
features.#end
>>>> {{/velocity}}
>>>> -------------------
>>>>
>>>> It has both content and script… If it's marked as "en" then
if the
user
>>> searches for "hasEdit" he
won't get it if his wiki is in a language
other
>>> than "en".
>>>>
>>>> Unless, if there's no translation in the language of the user then
we
>>> return the default language results for that page. Would that make
sense?
>>>>
>>>> But there's still the issue of editing the page, which will create a
>>> translation and then imagine that we replace the velocity script in a
>>> future version, then the user will only get his default page updated
and
>>> not his translation… However
that's a general problem I guess...
>>>>
>>>> If it has no default language (as is the case now BTW) then it seems
>>> less of an issue it seems. It just means:
>>>> * If user searches for "work" he'll get result even though
he's no
in an
>>> "en" wiki. But then
he's searching for an english word too ;)
>>>> * any other downside?
>>>>
>>>> In view of all this, it seems that not setting any default language
is a
>>> lesser evil, doesn't it?
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>>> That is why I'm saying that this kind of documents don't have
a
>>>>> language, and they never should have. They adapt themselves to the
>>>>> user's language, so they're written in no language, yet they
can
match
>>>>> all languages.
>>>>>
>>>>> Of course, not all documents are like this, as I originally stated
>>>>> myself, and there are valid cases where documents should have
"en"
as
>>>>> the default translation.
>>>>>
>>>>>> What am I missing?
>>>>>>
>>>>>> (I'm not commenting on anything below yet because I feel
it's
>>> important to agree on what's before first)
>>>>>>
>>>>>> Thanks
>>>>>> -Vincent
>>>>>>
>>>>>>> Other options:
>>>>>>>
>>>>>>> 4) Detect somehow localized documents and index:
>>>>>>> - the raw content using a non-language-specific analyzer
>>>>>>> - the content translated into all the languages registered in
the
>>>>>>> administration, each with the proper language-specific
analyzer,
if
>>> they
>>>>>>> are supported by Solr; this includes the default wiki
language.
>>>>>>> 4a) localized document = the default language is empty
>>>>>>> 4b) localized document = the default language is literally
"localized"
>>>>>>> 4c) add another
document flag for marking localized documents
>>>>>>>
>>>>>>> 5) When the defaultLanguage is empty, render in the
configured
wiki
>>>>>>> default language
>>>>>>>
>>>>>>>
>>>>>>> I like 4) since it makes localized documents really
searchable in
all
>>>>>>> the languages
"supported" by that wiki instance.
>>>>>>>
>>>>>>> 4a) is a behavior change, so it might cause some trouble
>>>>>>> 4b) is the safest and requires the least amount of changes
>>>>>>> The number of document fields is increasing, so I'm not
that fond
of
>>> 4c)
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
11:50 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
Hi guys,
What seems more true to reality IMHO is to consider language of some
content instead of language of a page solely... A Groovy macro (in raw)
contains english words, but it's not english language in any way, it's
groovy language, and it's not really efficient to parse it as if it was
english (though it works). Usually when you parse and index code from a
specific language, you would ignore keywords that occur very frequently in
the language because they are not significant - and it might be keywords
that would be significant in real english language (for example, you might
ignore "package" in java though you would index it in english content).
What's nice with syntax 2.x is that code content will always be inside a
macro. So you could imagine having a new parameter "localized" common to
all macros (ie, could be true / false, depending if the macro manages its
translations by itself, or if it should be translated along with the rest
of the document content). Content from a "localized" macro would then not
be forked when editing new translations for the page, considering that this
macro content should be the same for all document. Built-in macros could
have appropriate default value for this "localized" field (ie {{warning}}
would not be localized by default, {{groovy}} would be, etc...). But maybe
that's way too much to only improve translations of mixed-content documents
(with code and translatable content) that is not a very good practice
anyway ... (but much easier sometimes).
BR,
Jeremie
2013/6/11 Vincent Massol <vincent(a)massol.net>
On Jun 11, 2013, at 11:09 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
wrote:
Thanks Sergui for catching up here, while I was
off.
This is a situation I claim since a long time now, which is that some
document has simply no language, because all text displayed by those
document are simply produced by the localization module. Not only the
search issue are affected, but also the display of language selection in
a
multi-wiki, and currently this is not as nice as
it should be.
IMO, with a wiki that could produce with a single source documents,
displayed document in all available translation of the wiki (like the UI
does), we need to have the notion of a "no language" or "any
language"
document. In my own projects, I have been able to manage that properly
with the empty "default language" and empty "language" case. Since
having
both is stupid, we will surely merge those column in the future, but we
need to keep the idea of a "no language" or "any language" document,
whatever you see it, and to properly manage it, not only for our own XAR
but for user produced documents as well.
For those reasons, I am -1 for 1) and 3), and +1 for 2)
BTW I forgot to say in the my previous reply but obviously I remove my -1
for 2) since it's much more complex than I thought.
I follow Thomas, the Welcome page is not a good
example, since this on
has
really a translation in all languages.
I do not understand the intend to index code statement in an index, this
is
the rendered document that should be indexed. And
for "any language"
document, it should be indexed separately for each language enable in a
multi-language wiki.
ok, that's something very important to decide.
I think the 2 use cases are valide; xwiki can be used both by end users
who only care about pure content and it can also be used by developers who
develop wiki pages with scripts inside. For the former they want to see
only content result and for the latter they want to see script results. For
example, as a dev I want to see where I call such velocity macro, or where
I use such rendering macro.
So I really believe that we need to index both types of content: rendered
and raw.
Then we need to decide how to present that in the UI, but it could an
option in the advanced search to include raw results too for example (and
remove duplications…).
So, to respond to Vincent, using properly the
"any language" case by
clearing the default language where needed, is the right way to go, but
we
also need to manage that special case properly
elsewhere (indexing,
language selection on the page,…)
So what's the rule for putting <defaultLanguage>en</defaultLanguage>?
Whenever there's a page having at least one word of English in the rendered
result?
Thanks
-Vincent
On Tue, Jun 11, 2013 at 10:54 AM, Thomas Mortagne
<
thomas.mortagne(a)xwiki.com
wrote:
> That's not the home page, that's the Welcome page and it's not a very
> good example since this page do have translations already so we
> already decided what we wanted for this pages in practice: each
> translation of the page copy the scripts.
>
> On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net>
wrote:
>> Hi Sergiu,
>>
>> On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
>>
>>> On 06/10/2013 03:12 PM, Vincent Massol wrote:
>>>>
>>>> On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org>
>>>>
>>>>> On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
>>>>>> Hi devs,
>>>>>>
>>>>>> Right now the XAR plugin format goal systematically empty the
>>>>>> <defaultLanguage> property.
>>>>>>
>>>>>> This is wrong IMO since it means we have no idea what is the
default
>>>>>> document language, it was
not too visible before but it's really
not
>>>>>> very nice for things like
the localization module and especially
SOLR
>>>>>> which store deferently
the content depending on the language (stop
>>>>>> words, etc).
>>>>>>
>>>>>> I see several possibilities:
>>>>>>
>>>>>> 1) We don't touch the XAR maven plugin and we state that when
default
>>>>>> language is not set,
it's en (in the importer for example or in
>>>>>> XWikiDocument#getDefaultLanguage)
>>>>>> 2) We stop filtering default language in the XAR plugin and we
set
it
>>>>>> to en for all document in
which it make sense
>>>>>> 3) We force default language to "en" in the XAR plugin
>>>>>>
>>>>>> WDYT ?
>>>>>>
>>>>>> I don't like too much 1) since some technical document could
really
> be
>>>>>> seen has having no default language, some document without any
> literal
>>>>>> content. But it's more a -0 than a -1, I understand other
would
want
>>>>>> this for simplicity.
>>>>>>
>>>>>> About 3) as I said having a default language empty is a valid
use
> case
>>>>>> IMO so -0 for this one to. Still a bit better than 1) since the
use
>>>>>> case is still possible.
>>>>>>
>>>>>> +1 for 2)
>>>>>
>>>>> Neither option is good in general. The main problem is that most
>>>>> documents are written in the "Velocity" language, not in
the
"English"
>>>>> language, meaning that it
only contains code (which won't be seen by
> the
>>>>> user), and translations, which depend on a lot of factors. It's
not
> good
>>>>> to say that the default language of a dynamically translated
document
> is
>>>>> en, since a wiki configured with a different language will only
> display
>>>>> them in that language, never in en.
>>>>>
>>>>> There are only a few documents that contain real text (normally only
> the
>>>>> sandbox should have real text, everything else should be localized),
> and
>>>>> for those it's OK to specify the actual language.
>>>>
>>>> I'm not sure I agree with this vision. It really depends on the use
> case. So far we haven't found a perfect solution.
>>>>
>>>> Some pages will have more code than content, others will have more
> content than code. For the former, keys are best and for the latter
> translations are best.
>>>>
>>>> In any case I don't understand the problem. What is the issue with
> saying that all our pages are in English by default. If a wiki is
> configured to be in another language and there's no translation for that
> language the default language (ie "English") will be used.
>>>
>>> There is no actual text in the document. How can you say that the
>>> language of
>>>
>
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
>>> is English, since there's no
English sentence in there? Depending on
the
>>> configuration, the same document will
appear in German, Chinese, even
>>> Klingon, without changing anything in the document, so it is
definitely
>>> not an English document.
>>>
>>> Scenario: Set up a new XWiki instance, and change the default language
>>> of the wiki to German. When you browse the wiki, everything is in
>>> German. Yet all the documents say that they're in English.
>>>
>>> Problem 1: The wiki is indexed as English text, so searching for text
>>> that the user actually sees in the wiki won't return any results.
>>>
>>> Problem 2: Editing such a document will automatically create a
>>> translation, since the original document is in English, and the user
>>> wants to edit a German document. Since the two languages are not
>>> compatible, a translation will be created automatically. Now the code
>>> has been forked, and automatic updates using the Distribution Wizard
>>> will update the hidden English document, since that is the default
one,
>>> while the forked translation will
stay behind.
>>
>> Thanks a lot for describing these 2 use cases that I definitely
wouldn't
> have thought about! That's very useful.
>>
>> So it seems that suddenly it's becoming more complex ;)
>>
>> Basically it means that if we have documents that mix content and
> scripting we're going to have issues:
>> * Either they're marked as having no default language and the english
> content will be indexed in the default language of the wiki
>> * Either they're marked as "en" and the user will not have the
scripts
> in the search results and the DW/EM will update only the default
version if
> the user has created a translation
>>
>> I'm sure we have lots of cases like this, the easiest one being the
main
> home page:
>>
>> -------------------
>> It's an easy-to-edit website that will help you work better together.
> This Wiki is made of //pages// sorted by //spaces//. You're currently in
> the **Main** space, looking at its home page (**WebHome**).
>>
>> Learn how to use XWiki with the {{velocity}}[[Getting Started
> Guide>>
>
http://enterprise.xwiki.org/xwiki/bin/view/GettingStarted/WebHome?version=$…
> .
>>
>> {{velocity}}
>> #if($hasEdit)You can then use the [[Sandbox
> space>>Sandbox.WebHome]] to try out your wiki's features.#end
>> {{/velocity}}
>> -------------------
>>
>> It has both content and script… If it's marked as "en" then if the
user
> searches for "hasEdit" he won't get it if his wiki is in a language
other
> than "en".
>>
>> Unless, if there's no translation in the language of the user then we
> return the default language results for that page. Would that make
sense?
>>
>> But there's still the issue of editing the page, which will create a
> translation and then imagine that we replace the velocity script in a
> future version, then the user will only get his default page updated and
> not his translation… However that's a general problem I guess...
>>
>> If it has no default language (as is the case now BTW) then it seems
> less of an issue it seems. It just means:
>> * If user searches for "work" he'll get result even though he's
no in
an
> "en" wiki. But then he's
searching for an english word too ;)
>> * any other downside?
>>
>> In view of all this, it seems that not setting any default language is
a
> lesser evil, doesn't it?
>>
>> Thanks
>> -Vincent
>>
>>> That is why I'm saying that this kind of documents don't have a
>>> language, and they never should have. They adapt themselves to the
>>> user's language, so they're written in no language, yet they can
match
>>> all languages.
>>>
>>> Of course, not all documents are like this, as I originally stated
>>> myself, and there are valid cases where documents should have "en"
as
>>> the default translation.
>>>
>>>> What am I missing?
>>>>
>>>> (I'm not commenting on anything below yet because I feel it's
> important to agree on what's before first)
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>>> Other options:
>>>>>
>>>>> 4) Detect somehow localized documents and index:
>>>>> - the raw content using a non-language-specific analyzer
>>>>> - the content translated into all the languages registered in the
>>>>> administration, each with the proper language-specific analyzer, if
> they
>>>>> are supported by Solr; this includes the default wiki language.
>>>>> 4a) localized document = the default language is empty
>>>>> 4b) localized document = the default language is literally
"localized"
>>>>> 4c) add another document flag
for marking localized documents
>>>>>
>>>>> 5) When the defaultLanguage is empty, render in the configured wiki
>>>>> default language
>>>>>
>>>>>
>>>>> I like 4) since it makes localized documents really searchable in
all
>>>>> the languages
"supported" by that wiki instance.
>>>>>
>>>>> 4a) is a behavior change, so it might cause some trouble
>>>>> 4b) is the safest and requires the least amount of changes
>>>>> The number of document fields is increasing, so I'm not that fond
of
> 4c)
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
11:21 a.m.
New subject: [xwiki-devs] What default language for our documents in releases XAR ?
On Tue, Jun 11, 2013 at 11:09 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
Thanks Sergui for catching up here, while I was off.
This is a situation I claim since a long time now, which is that some
document has simply no language, because all text displayed by those
document are simply produced by the localization module. Not only the
search issue are affected, but also the display of language selection in a
multi-wiki, and currently this is not as nice as it should be.
IMO, with a wiki that could produce with a single source documents,
displayed document in all available translation of the wiki (like the UI
does), we need to have the notion of a "no language" or "any
language"
document. In my own projects, I have been able to manage that properly
with the empty "default language" and empty "language" case. Since
having
both is stupid, we will surely merge those column in the future, but we
need to keep the idea of a "no language" or "any language" document,
whatever you see it, and to properly manage it, not only for our own XAR
but for user produced documents as well.
For those reasons, I am -1 for 1) and 3), and +1 for 2)
This is exactly my position and why I proposed 2).
I follow Thomas, the Welcome page is not a good example, since this on has
really a translation in all languages.
I do not understand the intend to index code statement in an index, this is
the rendered document that should be indexed. And for "any language"
document, it should be indexed separately for each language enable in a
multi-language wiki.
So, to respond to Vincent, using properly the "any language" case by
clearing the default language where needed, is the right way to go, but we
also need to manage that special case properly elsewhere (indexing,
language selection on the page,...)
On Tue, Jun 11, 2013 at 10:54 AM, Thomas Mortagne <thomas.mortagne(a)xwiki.com
--
Thomas Mortagne
wrote:
> That's not the home page, that's the Welcome page and it's not a very
> good example since this page do have translations already so we
> already decided what we wanted for this pages in practice: each
> translation of the page copy the scripts.
>
> On Tue, Jun 11, 2013 at 10:19 AM, Vincent Massol <vincent(a)massol.net>
wrote:
> > Hi Sergiu,
> >
> > On Jun 10, 2013, at 9:49 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
> >
> >> On 06/10/2013 03:12 PM, Vincent Massol wrote:
> >>>
> >>> On Jun 10, 2013, at 8:25 PM, Sergiu Dumitriu <sergiu(a)xwiki.org wrote:
> >>>
> >>>> On 06/10/2013 11:00 AM, Thomas Mortagne wrote:
> >>>>> Hi devs,
> >>>>>
> >>>>> Right now the XAR plugin format goal systematically empty the
> >>>>> <defaultLanguage> property.
> >>>>>
> >>>>> This is wrong IMO since it means we have no idea what is the
default
> >>>>> document language, it was not too visible before but it's
really not
> >>>>> very nice for things like the localization module and especially
SOLR
> >>>>> which store deferently the content depending on the language
(stop
> >>>>> words, etc).
> >>>>>
> >>>>> I see several possibilities:
> >>>>>
> >>>>> 1) We don't touch the XAR maven plugin and we state that
when default
> >>>>> language is not set, it's en (in the importer for example or
in
> >>>>> XWikiDocument#getDefaultLanguage)
> >>>>> 2) We stop filtering default language in the XAR plugin and we
set it
> >>>>> to en for all document in which it make sense
> >>>>> 3) We force default language to "en" in the XAR
plugin
> >>>>>
> >>>>> WDYT ?
> >>>>>
> >>>>> I don't like too much 1) since some technical document could
really
> be
> >>>>> seen has having no default language, some document without any
> literal
> >>>>> content. But it's more a -0 than a -1, I understand other
would want
> >>>>> this for simplicity.
> >>>>>
> >>>>> About 3) as I said having a default language empty is a valid
use
> case
> >>>>> IMO so -0 for this one to. Still a bit better than 1) since the
use
> >>>>> case is still possible.
> >>>>>
> >>>>> +1 for 2)
> >>>>
> >>>> Neither option is good in general. The main problem is that most
> >>>> documents are written in the "Velocity" language, not in
the "English"
> >>>> language, meaning that it only contains code (which won't be
seen by
> the
> >>>> user), and translations, which depend on a lot of factors. It's
not
> good
> >>>> to say that the default language of a dynamically translated
document
> is
> >>>> en, since a wiki configured with a different language will only
> display
> >>>> them in that language, never in en.
> >>>>
> >>>> There are only a few documents that contain real text (normally
only
> the
> >>>> sandbox should have real text, everything else should be
localized),
> and
> >>>> for those it's OK to specify the actual language.
> >>>
> >>> I'm not sure I agree with this vision. It really depends on the use
> case. So far we haven't found a perfect solution.
> >>>
> >>> Some pages will have more code than content, others will have more
> content than code. For the former, keys are best and for the latter
> translations are best.
> >>>
> >>> In any case I don't understand the problem. What is the issue with
> saying that all our pages are in English by default. If a wiki is
> configured to be in another language and there's no translation for that
> language the default language (ie "English") will be used.
> >>
> >> There is no actual text in the document. How can you say that the
> >> language of
> >>
>
https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwi…
> >> is English, since there's no English sentence in there? Depending on
the
> >> configuration, the same document will appear in German, Chinese, even
> >> Klingon, without changing anything in the document, so it is definitely
> >> not an English document.
> >>
> >> Scenario: Set up a new XWiki instance, and change the default language
> >> of the wiki to German. When you browse the wiki, everything is in
> >> German. Yet all the documents say that they're in English.
> >>
> >> Problem 1: The wiki is indexed as English text, so searching for text
> >> that the user actually sees in the wiki won't return any results.
> >>
> >> Problem 2: Editing such a document will automatically create a
> >> translation, since the original document is in English, and the user
> >> wants to edit a German document. Since the two languages are not
> >> compatible, a translation will be created automatically. Now the code
> >> has been forked, and automatic updates using the Distribution Wizard
> >> will update the hidden English document, since that is the default one,
> >> while the forked translation will stay behind.
> >
> > Thanks a lot for describing these 2 use cases that I definitely wouldn't
> have thought about! That's very useful.
> >
> > So it seems that suddenly it's becoming more complex ;)
> >
> > Basically it means that if we have documents that mix content and
> scripting we're going to have issues:
> > * Either they're marked as having no default language and the english
> content will be indexed in the default language of the wiki
> > * Either they're marked as "en" and the user will not have the
scripts
> in the search results and the DW/EM will update only the default version if
> the user has created a translation
> >
> > I'm sure we have lots of cases like this, the easiest one being the main
> home page:
> >
> > -------------------
> > It's an easy-to-edit website that will help you work better together.
> This Wiki is made of //pages// sorted by //spaces//. You're currently in
> the **Main** space, looking at its home page (**WebHome**).
> >
> > Learn how to use XWiki with the {{velocity}}[[Getting Started
> Guide>>
>
http://enterprise.xwiki.org/xwiki/bin/view/GettingStarted/WebHome?version=$…
> .
> >
> > {{velocity}}
> > #if($hasEdit)You can then use the [[Sandbox
> space>>Sandbox.WebHome]] to try out your wiki's features.#end
> > {{/velocity}}
> > -------------------
> >
> > It has both content and script… If it's marked as "en" then if the
user
> searches for "hasEdit" he won't get it if his wiki is in a language
other
> than "en".
> >
> > Unless, if there's no translation in the language of the user then we
> return the default language results for that page. Would that make sense?
> >
> > But there's still the issue of editing the page, which will create a
> translation and then imagine that we replace the velocity script in a
> future version, then the user will only get his default page updated and
> not his translation… However that's a general problem I guess...
> >
> > If it has no default language (as is the case now BTW) then it seems
> less of an issue it seems. It just means:
> > * If user searches for "work" he'll get result even though
he's no in an
> "en" wiki. But then he's searching for an english word too ;)
> > * any other downside?
> >
> > In view of all this, it seems that not setting any default language is a
> lesser evil, doesn't it?
> >
> > Thanks
> > -Vincent
> >
> >> That is why I'm saying that this kind of documents don't have a
> >> language, and they never should have. They adapt themselves to the
> >> user's language, so they're written in no language, yet they can
match
> >> all languages.
> >>
> >> Of course, not all documents are like this, as I originally stated
> >> myself, and there are valid cases where documents should have "en"
as
> >> the default translation.
> >>
> >>> What am I missing?
> >>>
> >>> (I'm not commenting on anything below yet because I feel it's
> important to agree on what's before first)
> >>>
> >>> Thanks
> >>> -Vincent
> >>>
> >>>> Other options:
> >>>>
> >>>> 4) Detect somehow localized documents and index:
> >>>> - the raw content using a non-language-specific analyzer
> >>>> - the content translated into all the languages registered in the
> >>>> administration, each with the proper language-specific analyzer, if
> they
> >>>> are supported by Solr; this includes the default wiki language.
> >>>> 4a) localized document = the default language is empty
> >>>> 4b) localized document = the default language is literally
"localized"
> >>>> 4c) add another document flag for marking localized documents
> >>>>
> >>>> 5) When the defaultLanguage is empty, render in the configured wiki
> >>>> default language
> >>>>
> >>>>
> >>>> I like 4) since it makes localized documents really searchable in
all
> >>>> the languages "supported" by that wiki instance.
> >>>>
> >>>> 4a) is a behavior change, so it might cause some trouble
> >>>> 4b) is the safest and requires the least amount of changes
> >>>> The number of document fields is increasing, so I'm not that
fond of
> 4c)
> >
> > _______________________________________________
> > devs mailing list
> > devs(a)xwiki.org
> > http://lists.xwiki.org/mailman/listinfo/devs
>
>
>
> --
> Thomas Mortagne
> _______________________________________________
> devs mailing list
> devs(a)xwiki.org
> http://lists.xwiki.org/mailman/listinfo/devs
>
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
4211
days inactive
4212
days old
19 comments
6 participants
participants (6)
-
Denis Gervalle -
Jeremie BOUSQUET -
Marius Dumitru Florea -
Sergiu Dumitriu -
Thomas Mortagne -
Vincent Massol