19 Nov
2012
19 Nov
'12
2:12 p.m.
Hi devs,
This issue has already been previously [1] discussed during the GSoC
project, but I am not particularly happy with the chosen approach.
When handling multiple languages, there are generally[2][3] 3 different
approaches:
1) Indexing the content in a single field (like title, doccontent, etc.)
- This has the advantage that queries are clear and fast
- The disadvantage is that you can not run very well tuned analyzers on the
fields, having to resort to (at best) basic tokenization and lowercasing.
2) Indexing the content in multiple fields, one field for each language
(like title_en, title_fr, doccontent_en, doccontent_fr, etc.)
- This has the advantage that you can easily specify (as dynamic fields)
that *_en fields are of type text_en (and analyzed by an english-centered
chain of analyzers); *_fr of type text_fr (focused on french, etc.), thus
making the results much better.
- The disadvantage is that querying such a schema is a pain. If you want
all the results in all languages, you end up with a big and expensive
query. If you want just some language, you have to read the right fields
(ex title_en) instead of just getting a clear field name (title).
-- Also, the schema.xml definition is a static one in this concern,
requiring you to know beforehand which languages you want to support (for
example when defining the default fields to search for). Adding a new
language requires you to start editing the xml files by hand.
3) Indexing the content in different Solr cores (indexes), one for each
language. Each core requires it's on directory and configuration files.
- The advantage is that queries are clean to write (like option 1) and that
you have a nice separation
- The disadvantage is that it's difficult to get it right (administrative
issues) and then you also have the (considerable) problem of having to fix
the relevancy score of a query result that has entries from different
cores; each core has it's own relevancy computed and does not consider the
others.
- To make it even worst, it seems that you can not [5] also push to a
remote Solr instance the configuration files when creating a new core
programatically. However, if we are running an embedded Solr instance, we
could provide a way to generate the config files and write them to the data
directory.
Currently I have implemented option 1) in our existing Solr integration,
which is also more or less compatible with our existing Lucene queries, but
I would like to find a better solution that actually analyses the content.
During GSoC, option 2) was preferred but the implementation did not
consider practical reasons like the ones described above (query complexity,
user configuration, etc.)
On a related note, I have also watched an interesting presentation [3]
about how Drupal handles its Solr integration and, particularly, a plugin
[4] that handles the multilingual aspect.
The idea seen there is that you have this UI that helps you generate
configuration files, depending you your needs. For instance, you (admin)
check that you need search for language English, French and German and the
ui/extension gives you a zip with the configuration you need to use in your
(remote or embedded) solr instance. The configuration for each language
comes preset with the analyzers you should use for it and the additional
resources (stopwords.txt, synonims.txt, etc.).
This approach helps with avoiding the need for admins to be forced to edit
xml files and could also still be useful for other cases, not only option
2).
All these problems basically come from the fact that there is no way to
specify in the schema.xml that, based on the value of a field (like the
field "lang" that stores the document language), you want to run this or
that group of analyzers.
Perhaps a solution would be a custom kind of "AggregatorAnalyzer" that
would call other analyzers at runtime, based on the value of the lang
field. However, this solution could only be applied at index time, when you
have the lang information (in the solrDocument to be indexed), but when you
perform the query, you can not analyze the query text since you do not know
the language of the field you're querying (it was determined at runtime -
at index time) and thus do not know what operations to apply to the query
(to reduce it to the same form as the indexed values).
I have also read another interesting analysis [6] on this problem that
elaborates on the complexities and limitations of each options. (Ignore the
Rosette stuff mentioned there)
I have been thinking about this for some time now, but the solution is
probably somewhere in between, finding an option that is acceptable while
not restrictive. I will probably also send a mail on the Solr list to get
some more input from there, but I get the feeling that whatever solution we
choose, it will most likely require the users to at least copy (or even
edit) some files into some directories (configurations and/or jars), since
it does not seem to be easy/possible to do everything on-the-fly,
programatically.
Any input on this would be highly appreciated, specially if others have
more experience with Solr setups.
Thanks,
Eduard
----------
[1] http://markmail.org/message/kaxaka7lsbgo57ms
[2]
http://lucidworks.lucidimagination.com/display/lweug/Multilingual+Indexing+…
[3]
http://drupalcity.de/session/language-specific-and-multilingual-full-text-s…
[4] http://drupal.org/project/apachesolr_multilingual
[5]
http://stackoverflow.com/questions/4064880/create-new-core-directories-in-s…
[6]
http://info.basistech.com/blog/bid/171842/Indexing-Strategies-for-Multiling…
21 Nov
21 Nov
2:04 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Hello Eduard,
it's nice of you to see you take this further.
This issue has already been previously [1] discussed
during the GSoC
project, but I am not particularly happy with the chosen approach.
When handling multiple languages, there are generally[2][3] 3 different
approaches:
1) Indexing the content in a single field (like title, doccontent, etc.)
- This has the advantage that queries are clear and fast
- The disadvantage is that you can not run very well tuned analyzers on the
fields, having to resort to (at best) basic tokenization and lowercasing.
2) Indexing the content in multiple fields, one field for each language
(like title_en, title_fr, doccontent_en, doccontent_fr, etc.)
- This has the advantage that you can easily specify (as dynamic fields)
that *_en fields are of type text_en (and analyzed by an english-centered
chain of analyzers); *_fr of type text_fr (focused on french, etc.), thus
making the results much better.
I would add one more field here: title_ws and text_ws where the full text is analyzed just
as words (using the whitespace-tokenizer?).
A match there would even be preferred to a match in the below text-fields.
(maybe that would be called title and text?)
- The disadvantage is that querying such a schema is a
pain. If you want
all the results in all languages, you end up with a big and expensive
query.
Why is this an issue?
Dismax does it for you for free (thanks to the "form" parameter that gives
weight to each of the fields).
This is an issue only if you start to have more than 100 languages or so...
Lucene, the underlying engine of solr, handles thousands of clauses in a query without an
issue (this is how prefix-queries are handled... they are expanded to a query for any of
the term that matches the prefix, a setting deep somewhere, which is about 2000 avoids
this to explode).
If you want just some language, you have to read the
right fields
(ex title_en) instead of just getting a clear field name (title).
You have to be careful, this is really only if you want to be specific. In this case, it
is likely that you also do not want so much stemming.
My experience, which was before dismax on curriki.org, has made it so that any query that
is a bit specific is likely to not desire stemming.
-- Also, the schema.xml definition is a static one in
this concern,
requiring you to know beforehand which languages you want to support (for
example when defining the default fields to search for). Adding a new
language requires you to start editing the xml files by hand.
True but the available languages are almost all hand-coded.
You could generate the schema.xml based on the available languages if not hand-generated?
There's one catch with this approach which is new to me but seems to be quite
important to implement this approach: the idf should be modified, the Similarity class
should be, so that the total number of documents is the total number of documents having
that language.
See:
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
The solution sketched there sounds easy but I have not tried it.
3) Indexing the content in different Solr cores
(indexes), one for each
language. Each core requires it's on directory and configuration files.
- The advantage is that queries are clean to write (like option 1) and that
you have a nice separation
- The disadvantage is that it's difficult to get it right (administrative
issues) and then you also have the (considerable) problem of having to fix
the relevancy score of a query result that has entries from different
cores; each core has it's own relevancy computed and does not consider the
others.
- To make it even worst, it seems that you can not [5] also push to a
remote Solr instance the configuration files when creating a new core
programatically. However, if we are running an embedded Solr instance, we
could provide a way to generate the config files and write them to the data
directory.
Post-processing results is very very very dangerous as performance is at risk (e.g. if a
core does not answer)... I would tend to avoid that as much as possible.
Currently I have implemented option 1) in our existing
Solr integration,
which is also more or less compatible with our existing Lucene queries, but
I would like to find a better solution that actually analyses the content.
During GSoC, option 2) was preferred but the implementation did not
consider practical reasons like the ones described above (query complexity,
user configuration, etc.)
True, Savitha surfed the possibility of having different solr documents per language.
I still could not be sure that this was not showing the document match single in one
language.
However, indicating which language it is matched into is probably useful...
Funnily, cross-language-retrieval is a mature research field but retrieval for
multilanguage user is not so!
On a related note, I have also watched an interesting
presentation [3]
about how Drupal handles its Solr integration and, particularly, a plugin
[4] that handles the multilingual aspect.
The idea seen there is that you have this UI that helps you generate
configuration files, depending you your needs. For instance, you (admin)
check that you need search for language English, French and German and the
ui/extension gives you a zip with the configuration you need to use in your
(remote or embedded) solr instance. The configuration for each language
comes preset with the analyzers you should use for it and the additional
resources (stopwords.txt, synonims.txt, etc.).
This approach helps with avoiding the need for admins to be forced to edit
xml files and could also still be useful for other cases, not only option
2).
Generating sounds like an easy approach to me.
All these problems basically come from the fact that
there is no way to
specify in the schema.xml that, based on the value of a field (like the
field "lang" that stores the document language), you want to run this or
that group of analyzers.
Well, this is possible with ThreadLocal but is not necessarily a good idea.
Also, it is very common that users formulate queries without formulating their language
and thus you need to "or" the user's queries through multiple languages
(e.g. given by the browser).
Perhaps a solution would be a custom kind of
"AggregatorAnalyzer" that
would call other analyzers at runtime, based on the value of the lang
field. However, this solution could only be applied at index time, when you
have the lang information (in the solrDocument to be indexed), but when you
perform the query, you can not analyze the query text since you do not know
the language of the field you're querying (it was determined at runtime -
at index time) and thus do not know what operations to apply to the query
(to reduce it to the same form as the indexed values).
How would that look at query time?
I have also read another interesting analysis [6] on
this problem that
elaborates on the complexities and limitations of each options. (Ignore the
Rosette stuff mentioned there)
I have been thinking about this for some time now, but the solution is
probably somewhere in between, finding an option that is acceptable while
not restrictive. I will probably also send a mail on the Solr list to get
some more input from there, but I get the feeling that whatever solution we
choose, it will most likely require the users to at least copy (or even
edit) some files into some directories (configurations and/or jars), since
it does not seem to be easy/possible to do everything on-the-fly,
programatically.
The only hard step is when changing the supported languages, I think.
In this case, when automatically generating the index, you need to warn the user.
The admin UI should have a checkbox "use generated schema" or a textarea for the
schema.
Those that want particular fields and tunings need to write their own schema.
The same UI could also include whether to include a phonetic track or not (then require
reindexing).
hope it helps.
paul
9:49 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Hi Paul,
I was counting on your feedback :)
On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <paul(a)hoplahup.net> wrote:
Can you please elaborate on this? I`m not sure I understand the problem.
Basically I would have to output a zip with schema.xml, solrconfig.xml and
then all the resources specific to all the selected languages (stopwords,
synonims, etc) for the languages that we can provide out of the box. For
other languages, the admin would have to get dirty with the xmls.
Already doing that.
That's what I was saying, that at query time, the searched term will not
get analyzed by the right chain. When you search for a single language, you
could add that language as a query filter and then you could apply the
right chain, but when searching in 2 or more (or no, meaning all) languages
you are stuck.
Please see above regarding configuration generation. Basically, since we
are going to support both embedded and remote solr instances, we could
support things like editing the schema from XWiki only for the embedded
instance, but not for the remote one. We might end up having separate UIs
for each case, since we might want to exploit the flexibility of the
embedded one as much as possible.
Hello Eduard,
it's nice of you to see you take this further.
Sure, Solr is great when you want to do simple queries like "XWiki Open
Source", however, since in XWiki we also expose the Solr/Lucene query APIs
to the platform, there will be (as as it is currently with Lucene) a lot of
extensions wanting to do search using this API. These extensions (like the
search suggest for example, rest search, etc) want to do something like
"title:'Open Source' AND type:document AND doccontent:XWiki". Because
option 2) is so messy in it's fields, it would mean that the extension
would have to come up with a query like "title_en:'Open Source' AND
type:document AND doccontent_en:XWiki" (assuming that it is only limited to
the current -- english or whatever -- language; what happens if it wants to
do that no matter what language? It will have to specify each combination
possible because we can't use generic field names).
Solr's approach works for using it in your web application's search input,
in a specific usecase, where you have precisely specified the default
search fields and their boosts inside your schema.xml. However, as a search
API, using option 2) you are making the life of anyone else wanting to use
the Solr search API really hard. Also, your search application will work
nicely when the user enters a simple query in the input field, but an
advanced user will suffer the same fate when trying to write an advanced
query, thus not relying on the default query (computed by solr based on
schema.xml).
Also, based on your note above regarding improvements like title_ws and
such, again, all of these are very well suited for the search application
use case, together with the default query that you configure in schema.xml,
making the search results perform really well. However, what does all these
fields mean to another extension wanting to do search? Will it have to
handle all these implementation details to query for title, content and
such? I`m not sure how well this would work in practice.
Unrealistic idea(?): perhaps we should come up with an abstract search
language (Solr/Lucene clone) that parses the searched fields andhides the
complexities of all the indexed fields, allowing to write simple queries
like "title:XWiki", while this gets translated to "title_en:XWiki OR
title_fr:XWiki OR title_de:XWiki..." :)
Am I approaching this wrong by trying to have both a tweakable/tweaked
search application AND a search API? Are the two not compatible? Do we have
to sacrifice search result performance (no language-specific stuff) to be
able to have a usable API?
This issue has already been previously [1]
discussed during the GSoC
project, but I am not particularly happy with the chosen approach.
When handling multiple languages, there are generally[2][3] 3 different
approaches:
1) Indexing the content in a single field (like title, doccontent, etc.)
- This has the advantage that queries are clear and fast
- The disadvantage is that you can not run very well tuned analyzers on
the
fields, having to resort to (at best) basic
tokenization and lowercasing.
2) Indexing the content in multiple fields, one field for each language
(like title_en, title_fr, doccontent_en, doccontent_fr, etc.)
- This has the advantage that you can easily specify (as dynamic fields)
that *_en fields are of type text_en (and analyzed by an english-centered
chain of analyzers); *_fr of type text_fr (focused on french, etc.), thus
making the results much better.
I would add one more field here: title_ws and text_ws where the full text
is analyzed just as words (using the whitespace-tokenizer?).
A match there would even be preferred to a match in the below text-fields.
(maybe that would be called title and text?)
- The disadvantage is that querying such a schema
is a pain. If you want
all the results in all languages, you end up with a big and expensive
query.
Why is this an issue?
Dismax does it for you for free (thanks to the "form" parameter that gives
weight to each of the fields).
This is an issue only if you start to have more than 100 languages or so...
Lucene, the underlying engine of solr, handles thousands of clauses in a
query without an issue (this is how prefix-queries are handled... they are
expanded to a query for any of the term that matches the prefix, a setting
deep somewhere, which is about 2000 avoids this to explode).
If you want
just some language, you have to read the right fields
(ex title_en) instead of just getting a clear field name (title).
You have to be careful, this is really only if you want to be specific. In
this case, it is likely that you also do not want so much stemming.
My experience, which was before dismax on curriki.org, has made it so
that any query that is a bit specific is likely to not desire stemming.
-- Also, the schema.xml definition is a static
one in this concern,
requiring you to know beforehand which languages you want to support (for
example when defining the default fields to search for). Adding a new
language requires you to start editing the xml files by hand.
True but the available languages are almost all hand-coded.
You could generate the schema.xml based on the available languages if not
hand-generated?
There's one catch with this approach which is new to me but seems to be
quite important to implement this approach: the idf should be modified, the
Similarity class should be, so that the total number of documents is the
total number of documents having that language.
See:
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
The solution sketched there sounds easy but I have not tried it.
Not really related, but this reminds me about the post processing that I do
for checking view rights over the returned result, but that's another
discussion that we will probably need to have :)
3) Indexing the content in different Solr cores
(indexes), one for each
language. Each core requires it's on directory and configuration files.
- The advantage is that queries are clean to write (like option 1) and
that
you have a nice separation
- The disadvantage is that it's difficult to get it right (administrative
issues) and then you also have the (considerable) problem of having to
fix
the relevancy score of a query result that has
entries from different
cores; each core has it's own relevancy computed and does not consider
the
others.
- To make it even worst, it seems that you can not [5] also push to a
remote Solr instance the configuration files when creating a new core
programatically. However, if we are running an embedded Solr instance, we
could provide a way to generate the config files and write them to the
data
directory.
Post-processing results is very very very dangerous as performance is at
risk (e.g. if a core does not answer)... I would tend to avoid that as much
as possible.
Currently I have implemented option 1) in our
existing Solr integration,
which is also more or less compatible with our existing Lucene queries,
but
I would like to find a better solution that
actually analyses the
content.
During GSoC, option 2) was preferred but the implementation did not
consider practical reasons like the ones described above (query
complexity,
user configuration, etc.)
True, Savitha surfed the possibility of having different solr documents
per language.
I still could not be sure that this was not showing the document match
single in one language.
However, indicating which language it is matched into is probably useful...
Funnily, cross-language-retrieval is a mature research
field but retrieval
for multilanguage user is not so!
Yes, however I don`t like the fact that we can not do everything from the
webapp and the admin needs to access the filesystem to install the given
configuration on the embedded/remote solr directory. Lucene does not have
this problem now. It just works with XWiki and everything is done from
XWiki UI. I feel that losing this commodity will not be very well received
by users that now have some new install steps to get XWiki running.
Well, of course, for the embedded solr version, we could handle it like we
do now and push the files directly from the webapp to the filesystem. Since
embedded will be default, it should be OK and avoid the extra install step.
Users with a remote solr machine should have the option to get the zip
instead.
Not sure if we can apply the new configuration without a restart, but I`ll
have to look more into it. I know the multi-core architecture supports
something like this but will have to see the details.
On a related note, I have also watched an
interesting presentation [3]
about how Drupal handles its Solr integration and, particularly, a plugin
[4] that handles the multilingual aspect.
The idea seen there is that you have this UI that helps you generate
configuration files, depending you your needs. For instance, you (admin)
check that you need search for language English, French and German and
the
ui/extension gives you a zip with the
configuration you need to use in
your
(remote or embedded) solr instance. The
configuration for each language
comes preset with the analyzers you should use for it and the additional
resources (stopwords.txt, synonims.txt, etc.).
This approach helps with avoiding the need for admins to be forced to
edit
xml files and could also still be useful for
other cases, not only option
2).
Generating sounds like an easy approach to me.
All these problems basically come from the fact
that there is no way to
specify in the schema.xml that, based on the value of a field (like the
field "lang" that stores the document language), you want to run this or
that group of analyzers.
Well, this is possible with ThreadLocal but is not necessarily a good idea.
Also, it is very common that users formulate queries without formulating
their language and thus you need to "or" the user's queries through
multiple languages (e.g. given by the browser).
Perhaps a solution would be a custom kind of
"AggregatorAnalyzer" that
would call other analyzers at runtime, based on the value of the lang
field. However, this solution could only be applied at index time, when
you
have the lang information (in the solrDocument to
be indexed), but when
you
perform the query, you can not analyze the query
text since you do not
know
the language of the field you're querying (it
was determined at runtime -
at index time) and thus do not know what operations to apply to the query
(to reduce it to the same form as the indexed values).
How would that look at query time?
I have also read another interesting analysis [6]
on this problem that
elaborates on the complexities and limitations of each options. (Ignore
the
Rosette stuff mentioned there)
I have been thinking about this for some time now, but the solution is
probably somewhere in between, finding an option that is acceptable while
not restrictive. I will probably also send a mail on the Solr list to get
some more input from there, but I get the feeling that whatever solution
we
choose, it will most likely require the users to
at least copy (or even
edit) some files into some directories (configurations and/or jars),
since
it does not seem to be easy/possible to do
everything on-the-fly,
programatically.
The only hard step is when changing the supported languages, I think.
In this case, when automatically generating the index, you need to warn
the user.
The admin UI should have a checkbox "use generated schema" or a textarea
for the schema.
Those that want particular fields and tunings need to write their own
schema.
The same UI could also include whether to include a phonetic track or not
(then require reindexing).
hope it helps.
Yes, very helpful so far. I`m counting on your expertise with Lucene/Solr
on the details. My current approach is a practical one without previous
experience on the topic, so I`m still doing mostly guesswork in some areas.
Thanks,
Eduard
paul
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
26 Nov
26 Nov
7:25 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Hi devs,
Any other input on this matter?
To summarize a bit, if we go with the multiple fields for each language, we
end up with an index like:
English version:
id: xwiki:Main.SomeDocument_en
language: en
space: Main
title_en: XWiki document
doccontent_en: This is some content
French version:
id: xwiki:Main.SomeDocument_fr
language: fr
space: Main
title_fr: XWiki document
doccontent_fr: This is some content
The Solr configuration is generated by some XWiki UI that returns a zip
that the admin has to unpack in his (remote) Solr instance. This could be
automated for the embedded instance. This operation is to be performed each
time an admin changes the indexed languages (rarely or even only once).
Querying such a schema is a bit tricky when you are interested in more than
one language, because you have to add all the clauses (title_en, title_fr,
etc.) specific to the languages you are interested in.
Some extra fields might also be added like title_ws (for whitespace
tokenization only) that have various approaches to the indexing operation,
with the aim of improving the relevancy.
One solution to simplify the query for API clients would be to use fields
like "title" and "doccontent" and to put as values very lightly (or
not at
all) analyzed content, as Paul suggested. This would allow applications to
write simple (and backwards compatible maybe) queries that will still work,
but will not catch some of the nuances of specific languages. As far as
I`ve seen until now, applications are not very interested in nuances, but
rather in filtering the results, a task for which this solution might be
well suited. Of course, nothing stops applications from using the *new* and
more expressive fields that are properly analized.
Thus, the search application will be the major beneficiary of these
analyzed fields (title_en, title_fr, etc.), while still allowing
applications to get their job done (trough generic, but less/not analized
fields like "title", "doccontent", etc.).
WDYT?
Thanks,
Eduard
On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru <enygma2002(a)gmail.com>wrote;wrote:
Hi Paul,
I was counting on your feedback :)
On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <paul(a)hoplahup.net> wrote:
Can you please elaborate on this? I`m not sure I understand the problem.
Basically I would have to output a zip with schema.xml, solrconfig.xml and
then all the resources specific to all the selected languages (stopwords,
synonims, etc) for the languages that we can provide out of the box. For
other languages, the admin would have to get dirty with the xmls.
Already doing that.
That's what I was saying, that at query time, the searched term will not
get analyzed by the right chain. When you search for a single language, you
could add that language as a query filter and then you could apply the
right chain, but when searching in 2 or more (or no, meaning all) languages
you are stuck.
Please see above regarding configuration generation. Basically, since we
are going to support both embedded and remote solr instances, we could
support things like editing the schema from XWiki only for the embedded
instance, but not for the remote one. We might end up having separate UIs
for each case, since we might want to exploit the flexibility of the
embedded one as much as possible.
Hello Eduard,
it's nice of you to see you take this further.
Sure, Solr is great when you want to do simple queries like "XWiki Open
Source", however, since in XWiki we also expose the Solr/Lucene query APIs
to the platform, there will be (as as it is currently with Lucene) a lot of
extensions wanting to do search using this API. These extensions (like the
search suggest for example, rest search, etc) want to do something like
"title:'Open Source' AND type:document AND doccontent:XWiki". Because
option 2) is so messy in it's fields, it would mean that the extension
would have to come up with a query like "title_en:'Open Source' AND
type:document AND doccontent_en:XWiki" (assuming that it is only limited to
the current -- english or whatever -- language; what happens if it wants to
do that no matter what language? It will have to specify each combination
possible because we can't use generic field names).
Solr's approach works for using it in your web application's search input,
in a specific usecase, where you have precisely specified the default
search fields and their boosts inside your schema.xml. However, as a search
API, using option 2) you are making the life of anyone else wanting to use
the Solr search API really hard. Also, your search application will work
nicely when the user enters a simple query in the input field, but an
advanced user will suffer the same fate when trying to write an advanced
query, thus not relying on the default query (computed by solr based on
schema.xml).
Also, based on your note above regarding improvements like title_ws and
such, again, all of these are very well suited for the search application
use case, together with the default query that you configure in schema.xml,
making the search results perform really well. However, what does all these
fields mean to another extension wanting to do search? Will it have to
handle all these implementation details to query for title, content and
such? I`m not sure how well this would work in practice.
Unrealistic idea(?): perhaps we should come up with an abstract search
language (Solr/Lucene clone) that parses the searched fields andhides the
complexities of all the indexed fields, allowing to write simple queries
like "title:XWiki", while this gets translated to "title_en:XWiki OR
title_fr:XWiki OR title_de:XWiki..." :)
Am I approaching this wrong by trying to have both a tweakable/tweaked
search application AND a search API? Are the two not compatible? Do we have
to sacrifice search result performance (no language-specific stuff) to be
able to have a usable API?
This issue has already been previously [1]
discussed during the GSoC
project, but I am not particularly happy with the chosen approach.
When handling multiple languages, there are generally[2][3] 3 different
approaches:
1) Indexing the content in a single field (like title, doccontent, etc.)
- This has the advantage that queries are clear and fast
- The disadvantage is that you can not run very well tuned analyzers on
the
fields, having to resort to (at best) basic
tokenization and
lowercasing.
2) Indexing the content in multiple fields, one field for each language
(like title_en, title_fr, doccontent_en, doccontent_fr, etc.)
- This has the advantage that you can easily specify (as dynamic fields)
that *_en fields are of type text_en (and analyzed by an
english-centered
chain of analyzers); *_fr of type text_fr
(focused on french, etc.),
thus
making the results much better.
I would add one more field here: title_ws and text_ws where the full text
is analyzed just as words (using the whitespace-tokenizer?).
A match there would even be preferred to a match in the below text-fields.
(maybe that would be called title and text?)
- The disadvantage is that querying such a schema
is a pain. If you want
all the results in all languages, you end up with a big and expensive
query.
Why is this an issue?
Dismax does it for you for free (thanks to the "form" parameter that
gives weight to each of the fields).
This is an issue only if you start to have more than 100 languages or
so...
Lucene, the underlying engine of solr, handles thousands of clauses in a
query without an issue (this is how prefix-queries are handled... they are
expanded to a query for any of the term that matches the prefix, a setting
deep somewhere, which is about 2000 avoids this to explode).
If you
want just some language, you have to read the right fields
(ex title_en) instead of just getting a clear field name (title).
You have to be careful, this is really only if you want to be specific.
In this case, it is likely that you also do not want so much stemming.
My experience, which was before dismax on curriki.org, has made it so
that any query that is a bit specific is likely to not desire stemming.
-- Also, the schema.xml definition is a static
one in this concern,
requiring you to know beforehand which languages you want to support
(for
example when defining the default fields to
search for). Adding a new
language requires you to start editing the xml files by hand.
True but the available languages are almost all hand-coded.
You could generate the schema.xml based on the available languages if not
hand-generated?
There's one catch with this approach which is new to me but seems to be
quite important to implement this approach: the idf should be modified, the
Similarity class should be, so that the total number of documents is the
total number of documents having that language.
See:
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
The solution sketched there sounds easy but I have not tried it.
Not really related, but this reminds me about the post processing that I
do for checking view rights over the returned result, but that's another
discussion that we will probably need to have :)
3) Indexing the content in different Solr cores
(indexes), one for each
language. Each core requires it's on directory and configuration files.
- The advantage is that queries are clean to write (like option 1) and
that
you have a nice separation
- The disadvantage is that it's difficult to get it right
(administrative
issues) and then you also have the (considerable)
problem of having to
fix
the relevancy score of a query result that has
entries from different
cores; each core has it's own relevancy computed and does not consider
the
others.
- To make it even worst, it seems that you can not [5] also push to a
remote Solr instance the configuration files when creating a new core
programatically. However, if we are running an embedded Solr instance,
we
could provide a way to generate the config files
and write them to the
data
directory.
Post-processing results is very very very dangerous as performance is at
risk (e.g. if a core does not answer)... I would tend to avoid that as much
as possible.
Currently I have implemented option 1) in our
existing Solr integration,
which is also more or less compatible with our existing Lucene queries,
but
I would like to find a better solution that
actually analyses the
content.
During GSoC, option 2) was preferred but the implementation did not
consider practical reasons like the ones described above (query
complexity,
user configuration, etc.)
True, Savitha surfed the possibility of having different solr documents
per language.
I still could not be sure that this was not showing the document match
single in one language.
However, indicating which language it is matched into is probably
useful...
Funnily, cross-language-retrieval is a mature
research field but
retrieval for multilanguage user is not so!
Yes, however I don`t like the fact that we can not do everything from the
webapp and the admin needs to access the filesystem to install the given
configuration on the embedded/remote solr directory. Lucene does not have
this problem now. It just works with XWiki and everything is done from
XWiki UI. I feel that losing this commodity will not be very well received
by users that now have some new install steps to get XWiki running.
Well, of course, for the embedded solr version, we could handle it like we
do now and push the files directly from the webapp to the filesystem. Since
embedded will be default, it should be OK and avoid the extra install step.
Users with a remote solr machine should have the option to get the zip
instead.
Not sure if we can apply the new configuration without a restart, but I`ll
have to look more into it. I know the multi-core architecture supports
something like this but will have to see the details.
On a related note, I have also watched an
interesting presentation [3]
about how Drupal handles its Solr integration and, particularly, a
plugin
[4] that handles the multilingual aspect.
The idea seen there is that you have this UI that helps you generate
configuration files, depending you your needs. For instance, you (admin)
check that you need search for language English, French and German and
the
ui/extension gives you a zip with the
configuration you need to use in
your
(remote or embedded) solr instance. The
configuration for each language
comes preset with the analyzers you should use for it and the additional
resources (stopwords.txt, synonims.txt, etc.).
This approach helps with avoiding the need for admins to be forced to
edit
xml files and could also still be useful for
other cases, not only
option
2).
Generating sounds like an easy approach to me.
All these problems basically come from the fact
that there is no way to
specify in the schema.xml that, based on the value of a field (like the
field "lang" that stores the document language), you want to run this or
that group of analyzers.
Well, this is possible with ThreadLocal but is not necessarily a good
idea.
Also, it is very common that users formulate queries without formulating
their language and thus you need to "or" the user's queries through
multiple languages (e.g. given by the browser).
Perhaps a solution would be a custom kind of
"AggregatorAnalyzer" that
would call other analyzers at runtime, based on the value of the lang
field. However, this solution could only be applied at index time, when
you
have the lang information (in the solrDocument to
be indexed), but when
you
perform the query, you can not analyze the query
text since you do not
know
the language of the field you're querying (it
was determined at runtime
-
at index time) and thus do not know what
operations to apply to the
query
(to reduce it to the same form as the indexed
values).
How would that look at query time?
I have also read another interesting analysis [6]
on this problem that
elaborates on the complexities and limitations of each options. (Ignore
the
Rosette stuff mentioned there)
I have been thinking about this for some time now, but the solution is
probably somewhere in between, finding an option that is acceptable
while
not restrictive. I will probably also send a mail
on the Solr list to
get
some more input from there, but I get the feeling
that whatever
solution we
choose, it will most likely require the users to
at least copy (or even
edit) some files into some directories (configurations and/or jars),
since
it does not seem to be easy/possible to do
everything on-the-fly,
programatically.
The only hard step is when changing the supported languages, I think.
In this case, when automatically generating the index, you need to warn
the user.
The admin UI should have a checkbox "use generated schema" or a textarea
for the schema.
Those that want particular fields and tunings need to write their own
schema.
The same UI could also include whether to include a phonetic track or not
(then require reindexing).
hope it helps.
Yes, very helpful so far. I`m counting on your expertise with Lucene/Solr
on the details. My current approach is a practical one without previous
experience on the topic, so I`m still doing mostly guesswork in some areas.
Thanks,
Eduard
paul
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
27 Nov
27 Nov
4:27 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Hi Edy,
On 11/26/2012 07:25 PM, Eduard Moraru wrote:
Hi devs,
Any other input on this matter?
To summarize a bit, if we go with the multiple fields for each language, we
end up with an index like:
English version:
id: xwiki:Main.SomeDocument_en
language: en
space: Main
title_en: XWiki document
doccontent_en: This is some content
French version:
id: xwiki:Main.SomeDocument_fr
language: fr
space: Main
title_fr: XWiki document
doccontent_fr: This is some content
The Solr configuration is generated by some XWiki UI that returns a zip
that the admin has to unpack in his (remote) Solr instance. This could be
automated for the embedded instance.
IMHO this is a "must", not a "could" for embedded instances.
This operation is to be performed each
time an admin changes the indexed languages (rarely or even only once).
There could be a reminder in the admin language UI.
Querying such a schema is a bit tricky when you are interested in more than
one language, because you have to add all the clauses (title_en, title_fr,
etc.) specific to the languages you are interested in.
But that's an exotic use case already I think. The common case is to
query for the context language only.
Some extra fields might also be added like title_ws (for whitespace
tokenization only) that have various approaches to the indexing operation,
with the aim of improving the relevancy.
One solution to simplify the query for API clients would be to use fields
like "title" and "doccontent" and to put as values very lightly (or
not at
all) analyzed content, as Paul suggested. This would allow applications to
write simple (and backwards compatible maybe) queries that will still work,
but will not catch some of the nuances of specific languages. As far as
I`ve seen until now, applications are not very interested in nuances, but
rather in filtering the results, a task for which this solution might be
well suited. Of course, nothing stops applications from using the *new* and
more expressive fields that are properly analized.
Thus, the search application will be the major beneficiary of these
analyzed fields (title_en, title_fr, etc.), while still allowing
applications to get their job done (trough generic, but less/not analized
fields like "title", "doccontent", etc.).
I think for applications this complexity/implementations details would
benefit being hidden behind a "query builder" interface of some sort, WDYT ?
Jerome
WDYT?
Thanks,
Eduard
On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru <enygma2002(a)gmail.com>wrote;wrote:
Hi Paul,
I was counting on your feedback :)
On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <paul(a)hoplahup.net> wrote:
Can you please elaborate on this? I`m not sure I understand the problem.
Basically I would have to output a zip with schema.xml, solrconfig.xml and
then all the resources specific to all the selected languages (stopwords,
synonims, etc) for the languages that we can provide out of the box. For
other languages, the admin would have to get dirty with the xmls.
Already doing that.
That's what I was saying, that at query time, the searched term will not
get analyzed by the right chain. When you search for a single language, you
could add that language as a query filter and then you could apply the
right chain, but when searching in 2 or more (or no, meaning all) languages
you are stuck.
Please see above regarding configuration generation. Basically, since we
are going to support both embedded and remote solr instances, we could
support things like editing the schema from XWiki only for the embedded
instance, but not for the remote one. We might end up having separate UIs
for each case, since we might want to exploit the flexibility of the
embedded one as much as possible.
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs Hello Eduard,
it's nice of you to see you take this further.
Sure, Solr is great when you want to do simple queries like "XWiki Open
Source", however, since in XWiki we also expose the Solr/Lucene query APIs
to the platform, there will be (as as it is currently with Lucene) a lot of
extensions wanting to do search using this API. These extensions (like the
search suggest for example, rest search, etc) want to do something like
"title:'Open Source' AND type:document AND doccontent:XWiki". Because
option 2) is so messy in it's fields, it would mean that the extension
would have to come up with a query like "title_en:'Open Source' AND
type:document AND doccontent_en:XWiki" (assuming that it is only limited to
the current -- english or whatever -- language; what happens if it wants to
do that no matter what language? It will have to specify each combination
possible because we can't use generic field names).
Solr's approach works for using it in your web application's search input,
in a specific usecase, where you have precisely specified the default
search fields and their boosts inside your schema.xml. However, as a search
API, using option 2) you are making the life of anyone else wanting to use
the Solr search API really hard. Also, your search application will work
nicely when the user enters a simple query in the input field, but an
advanced user will suffer the same fate when trying to write an advanced
query, thus not relying on the default query (computed by solr based on
schema.xml).
Also, based on your note above regarding improvements like title_ws and
such, again, all of these are very well suited for the search application
use case, together with the default query that you configure in schema.xml,
making the search results perform really well. However, what does all these
fields mean to another extension wanting to do search? Will it have to
handle all these implementation details to query for title, content and
such? I`m not sure how well this would work in practice.
Unrealistic idea(?): perhaps we should come up with an abstract search
language (Solr/Lucene clone) that parses the searched fields andhides the
complexities of all the indexed fields, allowing to write simple queries
like "title:XWiki", while this gets translated to "title_en:XWiki OR
title_fr:XWiki OR title_de:XWiki..." :)
Am I approaching this wrong by trying to have both a tweakable/tweaked
search application AND a search API? Are the two not compatible? Do we have
to sacrifice search result performance (no language-specific stuff) to be
able to have a usable API?
This issue has already been previously [1]
discussed during the GSoC
project, but I am not particularly happy with the chosen approach.
When handling multiple languages, there are generally[2][3] 3 different
approaches:
1) Indexing the content in a single field (like title, doccontent, etc.)
- This has the advantage that queries are clear and fast
- The disadvantage is that you can not run very well tuned analyzers on
the
fields, having to resort to (at best) basic
tokenization and
lowercasing.
2) Indexing the content in multiple fields, one
field for each language
(like title_en, title_fr, doccontent_en, doccontent_fr, etc.)
- This has the advantage that you can easily specify (as dynamic fields)
that *_en fields are of type text_en (and analyzed by an
english-centered
chain of analyzers); *_fr of type text_fr
(focused on french, etc.),
thus
making the results much better.
I would
add one more field here: title_ws and text_ws where the full text
is analyzed just as words (using the whitespace-tokenizer?).
A match there would even be preferred to a match in the below text-fields.
(maybe that would be called title and text?)
- The disadvantage is that querying such a schema
is a pain. If you want
all the results in all languages, you end up with a big and expensive
query.
Why is this an issue?
Dismax does it for you for free (thanks to the "form" parameter that
gives weight to each of the fields).
This is an issue only if you start to have more than 100 languages or
so...
Lucene, the underlying engine of solr, handles thousands of clauses in a
query without an issue (this is how prefix-queries are handled... they are
expanded to a query for any of the term that matches the prefix, a setting
deep somewhere, which is about 2000 avoids this to explode).
If you
want just some language, you have to read the right fields
(ex title_en) instead of just getting a clear field name (title).
You have to be
careful, this is really only if you want to be specific.
In this case, it is likely that you also do not want so much stemming.
My experience, which was before dismax on curriki.org, has made it so
that any query that is a bit specific is likely to not desire stemming.
-- Also,
the schema.xml definition is a static one in this concern,
requiring you to know beforehand which languages you want to support
(for
example when defining the default fields to
search for). Adding a new
language requires you to start editing the xml files by hand.
True but the
available languages are almost all hand-coded.
You could generate the schema.xml based on the available languages if not
hand-generated?
There's one catch with this approach which is
new to me but seems to be
quite important to implement this approach: the idf should be modified, the
Similarity class should be, so that the total number of documents is the
total number of documents having that language.
See:
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
The solution sketched there sounds easy but I have not tried it.
Not really related, but this reminds me about the post processing that I
do for checking view rights over the returned result, but that's another
discussion that we will probably need to have :)
3) Indexing the content in different Solr cores
(indexes), one for each
language. Each core requires it's on directory and configuration files.
- The advantage is that queries are clean to write (like option 1) and
that
you have a nice separation
- The disadvantage is that it's difficult to get it right
(administrative
issues) and then you also have the (considerable)
problem of having to
fix
the relevancy score of a query result that has
entries from different
cores; each core has it's own relevancy computed and does not consider
the
others.
- To make it even worst, it seems that you can not [5] also push to a
remote Solr instance the configuration files when creating a new core
programatically. However, if we are running an embedded Solr instance,
we
could provide a way to generate the config files
and write them to the
data
directory.
Post-processing results is very
very very dangerous as performance is at
risk (e.g. if a core does not answer)... I would tend to avoid that as much
as possible.
Currently
I have implemented option 1) in our existing Solr integration,
which is also more or less compatible with our existing Lucene queries,
but
I would like to find a better solution that
actually analyses the
content.
During GSoC, option 2) was preferred but the
implementation did not
consider practical reasons like the ones described above (query
complexity,
user configuration, etc.)
True, Savitha
surfed the possibility of having different solr documents
per language.
I still could not be sure that this was not showing the document match
single in one language.
However, indicating which language it is matched into is probably
useful...
Funnily, cross-language-retrieval is a mature
research field but
retrieval for multilanguage user is not so!
Yes, however I don`t like the fact that we can not do everything from the
webapp and the admin needs to access the filesystem to install the given
configuration on the embedded/remote solr directory. Lucene does not have
this problem now. It just works with XWiki and everything is done from
XWiki UI. I feel that losing this commodity will not be very well received
by users that now have some new install steps to get XWiki running.
Well, of course, for the embedded solr version, we could handle it like we
do now and push the files directly from the webapp to the filesystem. Since
embedded will be default, it should be OK and avoid the extra install step.
Users with a remote solr machine should have the option to get the zip
instead.
Not sure if we can apply the new configuration without a restart, but I`ll
have to look more into it. I know the multi-core architecture supports
something like this but will have to see the details.
On a related note, I have also watched an
interesting presentation [3]
about how Drupal handles its Solr integration and, particularly, a
plugin
[4] that handles the multilingual aspect.
The idea seen there is that you have this UI that helps you generate
configuration files, depending you your needs. For instance, you (admin)
check that you need search for language English, French and German and
the
ui/extension gives you a zip with the
configuration you need to use in
your
(remote or embedded) solr instance. The
configuration for each language
comes preset with the analyzers you should use for it and the additional
resources (stopwords.txt, synonims.txt, etc.).
This approach helps with avoiding the need for admins to be forced to
edit
xml files and could also still be useful for
other cases, not only
option
2).
Generating sounds like an easy
approach to me.
All these
problems basically come from the fact that there is no way to
specify in the schema.xml that, based on the value of a field (like the
field "lang" that stores the document language), you want to run this or
that group of analyzers.
Well, this is possible with ThreadLocal but is not
necessarily a good
idea.
Also, it is very common that users formulate queries without formulating
their language and thus you need to "or" the user's queries through
multiple languages (e.g. given by the browser).
Perhaps a solution would be a custom kind of
"AggregatorAnalyzer" that
would call other analyzers at runtime, based on the value of the lang
field. However, this solution could only be applied at index time, when
you
have the lang information (in the solrDocument to
be indexed), but when
you
perform the query, you can not analyze the query
text since you do not
know
the language of the field you're querying (it
was determined at runtime
-
at index time) and thus do not know what
operations to apply to the
query
(to reduce it to the same form as the indexed
values).
How would that look at query time?
I have
also read another interesting analysis [6] on this problem that
elaborates on the complexities and limitations of each options. (Ignore
the
Rosette stuff mentioned there)
I have been thinking about this for some time now, but the solution is
probably somewhere in between, finding an option that is acceptable
while
not restrictive. I will probably also send a mail
on the Solr list to
get
some more input from there, but I get the feeling
that whatever
solution we
choose, it will most likely require the users to
at least copy (or even
edit) some files into some directories (configurations and/or jars),
since
it does not seem to be easy/possible to do
everything on-the-fly,
programatically.
The only hard step is when changing the supported languages, I
think.
In this case, when automatically generating the index, you need to warn
the user.
The admin UI should have a checkbox "use generated schema" or a textarea
for the schema.
Those that want particular fields and tunings
need to write their own
schema.
The same UI could also include whether to include a phonetic track or not
(then require reindexing).
hope it helps.
Yes, very helpful so far. I`m counting on your expertise with Lucene/Solr
on the details. My current approach is a practical one without previous
experience on the topic, so I`m still doing mostly guesswork in some areas.
Thanks,
Eduard
paul
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
28 Nov
28 Nov
4:52 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Hi Jerome,
Thanks for the reply. Please read below...
On Tue, Nov 27, 2012 at 5:27 PM, Jerome Velociter <jerome(a)velociter.fr>wrote;wrote:
I also thought about something in that direction. The current query logic
uses XWiki`s query API, so the query processing part would have to be done
inside the query.execute() method. Something like expanding multilingual
fields:
"title:something AND doccontent:somethingElse AND lang:en" becomes
"(doccontent_en:something OR doccontent_fr:something OR ...) AND
(title_en:something OR title_fr:something OR ...) AND lang:en" (based on
the currently supported -- or just indexed -- languages)
Maybe we could also allow a parameter that disables this default action for
queries that want to be explicit and don`t want this field expansion.
I wonder if such a simple solution would fix our query issues or if it will
create new ones that we don`t yet know about :)
Thanks,
Eduard
Hi Edy,
On 11/26/2012 07:25 PM, Eduard Moraru wrote:
Even so, the developer would still have to do something like
"title_${xcontext.language}:something AND
doccontent_${xcontext.language}:somethingElse ..." which is not really nice.
Hi devs,
Any other input on this matter?
To summarize a bit, if we go with the multiple fields for each language,
we
end up with an index like:
English version:
id: xwiki:Main.SomeDocument_en
language: en
space: Main
title_en: XWiki document
doccontent_en: This is some content
French version:
id: xwiki:Main.SomeDocument_fr
language: fr
space: Main
title_fr: XWiki document
doccontent_fr: This is some content
The Solr configuration is generated by some XWiki UI that returns a zip
that the admin has to unpack in his (remote) Solr instance. This could be
automated for the embedded instance.
IMHO this is a "must", not a "could" for embedded instances.
This operation is to be performed each
time an admin changes the indexed languages
(rarely or even only once).
There could be a reminder in the admin language UI.
Querying such a schema is a bit tricky when you
are interested in more
than
one language, because you have to add all the clauses (title_en, title_fr,
etc.) specific to the languages you are interested in.
But that's an exotic use case already I think. The common case is to query
for the context language only.
Some extra fields might also be added like
title_ws (for whitespace
tokenization only) that have various approaches to the indexing operation,
with the aim of improving the relevancy.
One solution to simplify the query for API clients would be to use fields
like "title" and "doccontent" and to put as values very lightly (or
not at
all) analyzed content, as Paul suggested. This would allow applications to
write simple (and backwards compatible maybe) queries that will still
work,
but will not catch some of the nuances of specific languages. As far as
I`ve seen until now, applications are not very interested in nuances, but
rather in filtering the results, a task for which this solution might be
well suited. Of course, nothing stops applications from using the *new*
and
more expressive fields that are properly analized.
Thus, the search application will be the major beneficiary of these
analyzed fields (title_en, title_fr, etc.), while still allowing
applications to get their job done (trough generic, but less/not analized
fields like "title", "doccontent", etc.).
I think for applications this complexity/implementations details would
benefit being hidden behind a "query builder" interface of some sort, WDYT ?
Jerome
WDYT?
Thanks,
Eduard
On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru <enygma2002(a)gmail.com
______________________________**_________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/**mailman/listinfo/devs<http://lists.xwiki.org/ma…
wrote:
Hi Paul,
I was counting on your feedback :)
On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <paul(a)hoplahup.net>
wrote:
Hello Eduard,
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/**mailman/listinfo/devs<http://lists.xwiki.org/ma…
it's nice of you to see you take this further.
This issue has already been previously [1] discussed during the GSoC
> project, but I am not particularly happy with the chosen approach.
> When handling multiple languages, there are generally[2][3] 3 different
> approaches:
>
> 1) Indexing the content in a single field (like title, doccontent,
> etc.)
> - This has the advantage that queries are clear and fast
> - The disadvantage is that you can not run very well tuned analyzers on
>
the
> fields, having to resort to (at best) basic tokenization and
>
lowercasing.
> 2) Indexing the content in multiple fields, one field for each language
> (like title_en, title_fr, doccontent_en, doccontent_fr, etc.)
> - This has the advantage that you can easily specify (as dynamic
> fields)
> that *_en fields are of type text_en (and analyzed by an
>
english-centered
> chain of analyzers); *_fr of type text_fr (focused on french, etc.),
>
thus
> making the results much better.
>
I would add one more field here: title_ws and text_ws where the full
text
is analyzed just as words (using the whitespace-tokenizer?).
A match there would even be preferred to a match in the below
text-fields.
(maybe that would be called title and text?)
- The disadvantage is that querying such a schema is a pain. If you
> want
> all the results in all languages, you end up with a big and expensive
> query.
>
Why is this an issue?
Dismax does it for you for free (thanks to the "form" parameter that
gives weight to each of the fields).
This is an issue only if you start to have more than 100 languages or
so...
Lucene, the underlying engine of solr, handles thousands of clauses in a
query without an issue (this is how prefix-queries are handled... they
are
expanded to a query for any of the term that matches the prefix, a
setting
deep somewhere, which is about 2000 avoids this to explode).
Sure, Solr is great when you want to do simple queries like "XWiki Open
Source", however, since in XWiki we also expose the Solr/Lucene query
APIs
to the platform, there will be (as as it is currently with Lucene) a lot
of
extensions wanting to do search using this API. These extensions (like
the
search suggest for example, rest search, etc) want to do something like
"title:'Open Source' AND type:document AND doccontent:XWiki". Because
option 2) is so messy in it's fields, it would mean that the extension
would have to come up with a query like "title_en:'Open Source' AND
type:document AND doccontent_en:XWiki" (assuming that it is only limited
to
the current -- english or whatever -- language; what happens if it wants
to
do that no matter what language? It will have to specify each combination
possible because we can't use generic field names).
Solr's approach works for using it in your web application's search
input,
in a specific usecase, where you have precisely specified the default
search fields and their boosts inside your schema.xml. However, as a
search
API, using option 2) you are making the life of anyone else wanting to
use
the Solr search API really hard. Also, your search application will work
nicely when the user enters a simple query in the input field, but an
advanced user will suffer the same fate when trying to write an advanced
query, thus not relying on the default query (computed by solr based on
schema.xml).
Also, based on your note above regarding improvements like title_ws and
such, again, all of these are very well suited for the search application
use case, together with the default query that you configure in
schema.xml,
making the search results perform really well. However, what does all
these
fields mean to another extension wanting to do search? Will it have to
handle all these implementation details to query for title, content and
such? I`m not sure how well this would work in practice.
Unrealistic idea(?): perhaps we should come up with an abstract search
language (Solr/Lucene clone) that parses the searched fields andhides the
complexities of all the indexed fields, allowing to write simple queries
like "title:XWiki", while this gets translated to "title_en:XWiki OR
title_fr:XWiki OR title_de:XWiki..." :)
Am I approaching this wrong by trying to have both a tweakable/tweaked
search application AND a search API? Are the two not compatible? Do we
have
to sacrifice search result performance (no language-specific stuff) to be
able to have a usable API?
If you want just some language, you have to read the right fields
> (ex title_en) instead of just getting a clear
field name (title).
>
You have to be careful, this is really only if you want to be specific.
In this case, it is likely that you also do not want so much stemming.
My experience, which was before dismax on curriki.org, has made it so
that any query that is a bit specific is likely to not desire stemming.
Can you please elaborate on this? I`m not sure I understand the
problem.
-- Also, the schema.xml definition is a static one in this concern,
> requiring you to know beforehand which
languages you want to support
>
(for
> example when defining the default fields to search for). Adding a new
> language requires you to start editing the xml files by hand.
>
True but the available languages are almost all hand-coded.
You could generate the schema.xml based on the available languages if
not
hand-generated?
Basically I would have to output a zip with schema.xml, solrconfig.xml
and
then all the resources specific to all the selected languages (stopwords,
synonims, etc) for the languages that we can provide out of the box. For
other languages, the admin would have to get dirty with the xmls.
There's one catch with this approach which is new to me but seems to be
quite important to implement this approach: the
idf should be modified,
the
Similarity class should be, so that the total number of documents is the
total number of documents having that language.
See:
http://mail-archives.apache.**org/mod_mbox/lucene-solr-user/**
201211.mbox/%3Czarafa.**509ccb61.698a.**1d02345614818807(a)mail.**
openindex.io%3E<http://mail-archives.apache.org/mod_mbox/lucene-solr-use…
The solution sketched there sounds easy but I have not tried it.
3) Indexing the content in different Solr cores (indexes), one for each
> language. Each core requires it's on directory and configuration files.
> - The advantage is that queries are clean to write (like option 1) and
>
that
> you have a nice separation
> - The disadvantage is that it's difficult to get it right
>
(administrative
> issues) and then you also have the (considerable) problem of having to
>
fix
> the relevancy score of a query result that has entries from different
> cores; each core has it's own relevancy computed and does not consider
>
the
> others.
> - To make it even worst, it seems that you can not [5] also push to a
> remote Solr instance the configuration files when creating a new core
> programatically. However, if we are running an embedded Solr instance,
>
we
> could provide a way to generate the config files and write them to the
>
data
> directory.
>
Post-processing results is very very very dangerous as performance is at
risk (e.g. if a core does not answer)... I would tend to avoid that as
much
as possible.
Not really related, but this reminds me about the post processing that
I
do for checking view rights over the returned result, but that's another
discussion that we will probably need to have :)
Currently I have implemented option 1) in our existing Solr integration,
> which is also more or less compatible with
our existing Lucene queries,
>
but
> I would like to find a better solution that actually analyses the
>
content.
> During GSoC, option 2) was preferred but the implementation did not
> consider practical reasons like the ones described above (query
>
complexity,
> user configuration, etc.)
>
True, Savitha surfed the possibility of having different solr documents
per language.
I still could not be sure that this was not showing the document match
single in one language.
However, indicating which language it is matched into is probably
useful...
Already doing that.
Funnily, cross-language-retrieval is a mature research field but
retrieval for multilanguage user is not so!
On a related note, I have also watched an interesting presentation [3]
> about how Drupal handles its Solr integration and, particularly, a
>
plugin
> [4] that handles the multilingual aspect.
> The idea seen there is that you have this UI that helps you generate
> configuration files, depending you your needs. For instance, you
> (admin)
> check that you need search for language English, French and German and
>
the
> ui/extension gives you a zip with the configuration you need to use in
>
your
> (remote or embedded) solr instance. The configuration for each language
> comes preset with the analyzers you should use for it and the
> additional
> resources (stopwords.txt, synonims.txt, etc.).
> This approach helps with avoiding the need for admins to be forced to
>
edit
> xml files and could also still be useful for other cases, not only
>
option
> 2).
>
Generating sounds like an easy approach to me.
Yes, however I don`t like the fact that we can not do everything from
the
webapp and the admin needs to access the filesystem to install the given
configuration on the embedded/remote solr directory. Lucene does not have
this problem now. It just works with XWiki and everything is done from
XWiki UI. I feel that losing this commodity will not be very well
received
by users that now have some new install steps to get XWiki running.
Well, of course, for the embedded solr version, we could handle it like
we
do now and push the files directly from the webapp to the filesystem.
Since
embedded will be default, it should be OK and avoid the extra install
step.
Users with a remote solr machine should have the option to get the zip
instead.
Not sure if we can apply the new configuration without a restart, but
I`ll
have to look more into it. I know the multi-core architecture supports
something like this but will have to see the details.
All these problems basically come from the fact that there is no way to
> specify in the schema.xml that, based on the
value of a field (like the
> field "lang" that stores the document language), you want to run this
> or
> that group of analyzers.
>
Well, this is possible with ThreadLocal but is not necessarily a good
idea.
Also, it is very common that users formulate queries without formulating
their language and thus you need to "or" the user's queries through
multiple languages (e.g. given by the browser).
Perhaps a solution would be a custom kind of "AggregatorAnalyzer" that
> would call other analyzers at runtime, based on the value of the lang
> field. However, this solution could only be applied at index time, when
>
you
> have the lang information (in the solrDocument to be indexed), but when
>
you
> perform the query, you can not analyze the query text since you do not
>
know
> the language of the field you're querying (it was determined at runtime
>
-
> at index time) and thus do not know what operations to apply to the
>
query
> (to reduce it to the same form as the indexed values).
>
How would that look at query time?
That's what I was saying, that at query time, the searched term will
not
get analyzed by the right chain. When you search for a single language,
you
could add that language as a query filter and then you could apply the
right chain, but when searching in 2 or more (or no, meaning all)
languages
you are stuck.
I have also read another interesting analysis [6] on this problem that
> elaborates on the complexities and
limitations of each options. (Ignore
>
the
> Rosette stuff mentioned there)
>
> I have been thinking about this for some time now, but the solution is
> probably somewhere in between, finding an option that is acceptable
>
while
> not restrictive. I will probably also send a mail on the Solr list to
>
get
> some more input from there, but I get the feeling that whatever
>
solution we
> choose, it will most likely require the users to at least copy (or even
> edit) some files into some directories (configurations and/or jars),
>
since
> it does not seem to be easy/possible to do everything on-the-fly,
> programatically.
>
The only hard step is when changing the supported languages, I think.
In this case, when automatically generating the index, you need to warn
the user.
The admin UI should have a checkbox "use generated schema" or a textarea
for the schema.
Please see above regarding configuration generation. Basically, since
we
are going to support both embedded and remote solr instances, we could
support things like editing the schema from XWiki only for the embedded
instance, but not for the remote one. We might end up having separate UIs
for each case, since we might want to exploit the flexibility of the
embedded one as much as possible.
Those that want particular fields and tunings need to write their own
schema.
The same UI could also include whether to include a phonetic track or
not
(then require reindexing).
hope it helps.
Yes, very helpful so far. I`m counting on your expertise with
Lucene/Solr
on the details. My current approach is a practical one without previous
experience on the topic, so I`m still doing mostly guesswork in some
areas.
Thanks,
Eduard
paul
______________________________**_________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/**mailman/listinfo/devs<http://lists.xwiki.org/ma…
______________________________**_________________
29 Nov
29 Nov
4:34 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Le 28 nov. 2012 à 16:52, Eduard Moraru a écrit :
"title:something AND doccontent:somethingElse AND
lang:en" becomes
"(doccontent_en:something OR doccontent_fr:something OR ...) AND
(title_en:something OR title_fr:something OR ...) AND lang:en" (based on
the currently supported -- or just indexed -- languages)
Just a methodological note: we should not be programming queries by combining strings but
by combining Solr query objects which are rich, flexible, and easy to manipulate.
However, speaking this way in mails is by far the best way!
The mechanism you describe above is exactly how I would do it: my query expansion start
with a q parameter and a context.
Based on the context a weighted list of languages is extracted (context.language, then
browser languages).
I give the q parameter to the query-parser, with a whitespace analyzer. I obtain generally
a boolean-query (otherwise I wrap it). Then I recursively go through the components of the
boolean-queries.
Each TermQuery of the default field is expanded into a disjunction of the TermQuery in
each of the language (with their weight and with the right analyzer).
Same for each phrasequery, wildcardquery, and fuzzyquery (not sure).
Finding the analyzer defined in the Solr schema is not too hard.
I can share code if you wish.
Paul
27 Nov
27 Nov
4:44 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Hi Edy,
I'm not a huge fan of the title_fr title_en title_morelanguages approach as
indeed it seems to be quite complex at the query level. I was more leaning
towards multiple indexes if we can query them globally but I understand
this is complex too.
Now let's see the use cases that are hugely important:
1/ Make sure that if you decide your wiki is monolingual:
- the indexing uses the specific language analyzer
- make sure the query uses the specific language analyzer
- make sure the search looks in all content even if the language setting
of the document is wrongly set (consider all documents being of the
specific language)
2/ Allow is a wiki is multi-lingual:
- search in the language you decide (maybe the UI should display a
language choice for the query)
- search in content that is analyzed in the proper language when the
content is declared in this language
- allow to specify if you want to restrict your search to documents
declared in the language of your query, versus search more widely in all
documents accross languages. If you search in only the language of the
query only one document can show up but it should point to the right
translation that matches, if you search in multiple languages then you can
show individual translations.
- allow technical users to search for all documents across all languages
(where the language analysis does not really matter)
From an admin point of view it makes good sense for the
admin to be able to
specify in a multilingual wiki which language analysis should be
activated,
and then have this transmitted to SOLR to properly configure the engine.
Reindexing is ok when changing the configuration.
I believe in the end wether you use multiple fields with _fr _en or
multiple SOLR cores, as long as you can query accross SOLR cores is a bit
the same. If you cannot run a query merging multiple indexes then the first
solution is kind of absolutely necessary as it would be the only one
allowing to search across all languages.
Maybe a solution would be to create one index per language and index ALL
content regardless of it's language using the language analyzer of that
index. This would allow to have better results even though the users have
badly tagged the language of a document, and it's only the job of the UI to
limit the search to only the language of the query, or all documents.
So you could have a configuration in the admin that says:
1/ Create an English Index
2/ Create an additional French index
The UI would allow to search in English and French, + would add a language
restriction for the documents.
In the future if we are able to "detect" the language of the documents we
could add a lucene field with the "detected" language instead of the
"provided" language of the documents, therefore increasing the quality of
searches only on documents of a specific language.
This later solution would be the only one that would really work on file
attachements as we have no information about the specific language of file
attachements (or even XWiki objects) which are attached to the main
document and not to the translated document.
This later issues shows that a search on "only french content" should still
include the attachements because we have no idea if the attachments are
"french" or "english".
Ludovic
2012/11/26 Eduard Moraru <enygma2002(a)gmail.com>
Hi devs,
Any other input on this matter?
To summarize a bit, if we go with the multiple fields for each language, we
end up with an index like:
English version:
id: xwiki:Main.SomeDocument_en
language: en
space: Main
title_en: XWiki document
doccontent_en: This is some content
French version:
id: xwiki:Main.SomeDocument_fr
language: fr
space: Main
title_fr: XWiki document
doccontent_fr: This is some content
The Solr configuration is generated by some XWiki UI that returns a zip
that the admin has to unpack in his (remote) Solr instance. This could be
automated for the embedded instance. This operation is to be performed each
time an admin changes the indexed languages (rarely or even only once).
Querying such a schema is a bit tricky when you are interested in more than
one language, because you have to add all the clauses (title_en, title_fr,
etc.) specific to the languages you are interested in.
Some extra fields might also be added like title_ws (for whitespace
tokenization only) that have various approaches to the indexing operation,
with the aim of improving the relevancy.
One solution to simplify the query for API clients would be to use fields
like "title" and "doccontent" and to put as values very lightly (or
not at
all) analyzed content, as Paul suggested. This would allow applications to
write simple (and backwards compatible maybe) queries that will still work,
but will not catch some of the nuances of specific languages. As far as
I`ve seen until now, applications are not very interested in nuances, but
rather in filtering the results, a task for which this solution might be
well suited. Of course, nothing stops applications from using the *new* and
more expressive fields that are properly analized.
Thus, the search application will be the major beneficiary of these
analyzed fields (title_en, title_fr, etc.), while still allowing
applications to get their job done (trough generic, but less/not analized
fields like "title", "doccontent", etc.).
WDYT?
Thanks,
Eduard
On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru <enygma2002(a)gmail.com
APIs
and
integration,
(admin)
Yes, however I don`t like the fact that we can not do everything from the
webapp and the admin needs to access the filesystem to install the given
configuration on the embedded/remote solr directory. Lucene does not have
this problem now. It just works with XWiki and everything is done from
XWiki UI. I feel that losing this commodity will not be very well received
you
Please see above regarding configuration generation. Basically, since we
are going to support both embedded and remote solr instances, we could
support things like editing the schema from XWiki only for the embedded
instance, but not for the remote one. We might end up having separate UIs
for each case, since we might want to exploit the flexibility of the
embedded one as much as possible.
>
> Those that want particular fields and tunings need to write their own
> schema.
>
> The same UI could also include whether to include a phonetic track or not
areas.
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
wrote:
Hi Paul,
I was counting on your feedback :)
On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <paul(a)hoplahup.net>
wrote:
>
> Hello Eduard,
>
> it's nice of you to see you take this further.
>
> > This issue has already been previously [1] discussed during the GSoC
> > project, but I am not particularly happy with the chosen approach.
> > When handling multiple languages, there are generally[2][3] 3
different
> > approaches:
> >
> > 1) Indexing the content in a single field (like title, doccontent,
etc.)
> > - This has the advantage that queries
are clear and fast
> > - The disadvantage is that you can not run very well tuned analyzers
on
> the
> > fields, having to resort to (at best) basic tokenization and
> lowercasing.
> >
> > 2) Indexing the content in multiple fields, one field for each
language
> > (like title_en, title_fr, doccontent_en,
doccontent_fr, etc.)
> > - This has the advantage that you can easily specify (as dynamic
fields)
> > that *_en fields are of type text_en
(and analyzed by an
> english-centered
> > chain of analyzers); *_fr of type text_fr (focused on french, etc.),
> thus
> > making the results much better.
>
> I would add one more field here: title_ws and text_ws where the full
text
> is analyzed just as words (using the
whitespace-tokenizer?).
> A match there would even be preferred to a match in the below
text-fields.
>
> (maybe that would be called title and text?)
>
> > - The disadvantage is that querying such a schema is a pain. If you
want
> > all the results in all languages, you
end up with a big and expensive
> > query.
>
> Why is this an issue?
> Dismax does it for you for free (thanks to the "form" parameter that
> gives weight to each of the fields).
> This is an issue only if you start to have more than 100 languages or
> so...
> Lucene, the underlying engine of solr, handles thousands of clauses in a
> query without an issue (this is how prefix-queries are handled... they
are
> expanded to a query for any of the term that
matches the prefix, a
setting
deep
somewhere, which is about 2000 avoids this to explode).
Sure, Solr is great when you want to do simple queries like "XWiki Open
Source", however, since in XWiki we also expose the Solr/Lucene query to the platform, there will be (as as it is
currently with Lucene) a lot
of
extensions wanting to do search using this API.
These extensions (like
the
search suggest for example, rest search, etc)
want to do something like
"title:'Open Source' AND type:document AND doccontent:XWiki". Because
option 2) is so messy in it's fields, it would mean that the extension
would have to come up with a query like "title_en:'Open Source' AND
type:document AND doccontent_en:XWiki" (assuming that it is only limited
to
the current -- english or whatever -- language;
what happens if it wants
to
do that no matter what language? It will have to
specify each combination
possible because we can't use generic field names).
Solr's approach works for using it in your web application's search
input,
in a specific usecase, where you have precisely
specified the default
search fields and their boosts inside your schema.xml. However, as a
search
API, using option 2) you are making the life of
anyone else wanting to
use
the Solr search API really hard. Also, your
search application will work
nicely when the user enters a simple query in the input field, but an
advanced user will suffer the same fate when trying to write an advanced
query, thus not relying on the default query (computed by solr based on
schema.xml).
Also, based on your note above regarding improvements like title_ws and
such, again, all of these are very well suited for the search application
use case, together with the default query that you configure in
schema.xml,
making the search results perform really well.
However, what does all
these
fields mean to another extension wanting to do
search? Will it have to
handle all these implementation details to query for title, content and
such? I`m not sure how well this would work in practice.
Unrealistic idea(?): perhaps we should come up with an abstract search
language (Solr/Lucene clone) that parses the searched fields andhides the
complexities of all the indexed fields, allowing to write simple queries
like "title:XWiki", while this gets translated to "title_en:XWiki OR
title_fr:XWiki OR title_de:XWiki..." :)
Am I approaching this wrong by trying to have both a tweakable/tweaked
search application AND a search API? Are the two not compatible? Do we
have
to sacrifice search result performance (no
language-specific stuff) to be
able to have a usable API?
Can you please elaborate on this? I`m not sure I understand the problem.
>
> > -- Also, the schema.xml definition is a static one in this concern,
> > requiring you to know beforehand which languages you want to support
> (for
> > example when defining the default fields to search for). Adding a new
> > language requires you to start editing the xml files by hand.
>
> True but the available languages are almost all hand-coded.
> You could generate the schema.xml based on the available languages if
not
If you
want just some language, you have to read the right fields
(ex title_en) instead of just getting a clear field name (title).
You have to be careful, this is really only if you want to be specific.
In this case, it is likely that you also do not want so much stemming.
My experience, which was before dismax on curriki.org, has made it so
that any query that is a bit specific is likely to not desire stemming.
hand-generated?
Basically I would have to output a zip with schema.xml, solrconfig.xml then all the resources specific to all the
selected languages (stopwords,
synonims, etc) for the languages that we can provide out of the box. For
other languages, the admin would have to get dirty with the xmls.
>
> There's one catch with this approach which is new to me but seems to be
> quite important to implement this approach: the idf should be modified,
the
> Similarity class should be, so that the total
number of documents is the
> total number of documents having that language.
> See:
>
>
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
> The solution sketched there sounds easy but I
have not tried it.
>
> > 3) Indexing the content in different Solr cores (indexes), one for
each
> > language. Each core requires it's on
directory and configuration
files.
> > - The advantage is that queries are
clean to write (like option 1) and
> that
> > you have a nice separation
> > - The disadvantage is that it's difficult to get it right
> (administrative
> > issues) and then you also have the (considerable) problem of having to
> fix
> > the relevancy score of a query result that has entries from different
> > cores; each core has it's own relevancy computed and does not consider
> the
> > others.
> > - To make it even worst, it seems that you can not [5] also push to a
> > remote Solr instance the configuration files when creating a new core
> > programatically. However, if we are running an embedded Solr instance,
> we
> > could provide a way to generate the config files and write them to the
> data
> > directory.
>
> Post-processing results is very very very dangerous as performance is at
> risk (e.g. if a core does not answer)... I would tend to avoid that as
much
as
possible.
Not really related, but this reminds me about the post processing that I
do for checking view rights over the returned result, but that's another
discussion that we will probably need to have :)
>
> > Currently I have implemented option 1) in our existing Solr > > which is also more or less compatible
with our existing Lucene
queries,
but
Already doing that.
> Funnily, cross-language-retrieval is a mature research field but
> retrieval for multilanguage user is not so!
>
> > On a related note, I have also watched an interesting presentation [3]
> > about how Drupal handles its Solr integration and, particularly, a
> plugin
> > [4] that handles the multilingual aspect.
> > The idea seen there is that you have this UI that helps you generate
> > configuration files, depending you your needs. For instance, you I would like to find a better solution that
actually analyses the
content.
During GSoC, option 2) was preferred but the implementation did not
consider practical reasons like the ones described above (query
complexity,
user configuration, etc.)
True, Savitha surfed the possibility of having different solr documents
per language.
I still could not be sure that this was not showing the document match
single in one language.
However, indicating which language it is matched into is probably
useful...
> > check that you need search for language
English, French and German and
> the
> > ui/extension gives you a zip with the configuration you need to use in
> your
> > (remote or embedded) solr instance. The configuration for each
language
> > comes preset with the analyzers you
should use for it and the
additional
resources (stopwords.txt, synonims.txt, etc.).
This approach helps with avoiding the need for admins to be forced to
edit
xml files and could also still be useful for
other cases, not only
option
2).
Generating sounds like an easy approach to me.
by users that now have some new install steps to
get XWiki running.
Well, of course, for the embedded solr version, we could handle it like
we
do now and push the files directly from the
webapp to the filesystem.
Since
embedded will be default, it should be OK and
avoid the extra install
step.
Users with a remote solr machine should have the
option to get the zip
instead.
Not sure if we can apply the new configuration without a restart, but
I`ll
have to look more into it. I know the multi-core
architecture supports
something like this but will have to see the details.
>
> > All these problems basically come from the fact that there is no way
to
> > specify in the schema.xml that, based on
the value of a field (like
the
> > field "lang" that stores the
document language), you want to run this
or
> > that group of analyzers.
>
> Well, this is possible with ThreadLocal but is not necessarily a good
> idea.
> Also, it is very common that users formulate queries without formulating
> their language and thus you need to "or" the user's queries through
> multiple languages (e.g. given by the browser).
>
> > Perhaps a solution would be a custom kind of "AggregatorAnalyzer"
that
> > would call other analyzers at runtime, based on the value of the lang
> > field. However, this solution could only be applied at index time,
when
> you
> > have the lang information (in the solrDocument to be indexed), but
when
> you
> > perform the query, you can not analyze the query text since you do not
> know
> > the language of the field you're querying (it was determined at
runtime
-
That's what I was saying, that at query time, the searched term will not
get analyzed by the right chain. When you search for a single language, at index time) and thus do not know what
operations to apply to the
query
(to reduce it to the same form as the indexed
values).
How would that look at query time?
could add that language as a query filter and
then you could apply the
right chain, but when searching in 2 or more (or no, meaning all)
languages
you are stuck.
>
> > I have also read another interesting analysis [6] on this problem that
> > elaborates on the complexities and limitations of each options.
(Ignore
> the
> > Rosette stuff mentioned there)
> >
> > I have been thinking about this for some time now, but the solution is
> > probably somewhere in between, finding an option that is acceptable
> while
> > not restrictive. I will probably also send a mail on the Solr list to
> get
> > some more input from there, but I get the feeling that whatever
> solution we
> > choose, it will most likely require the users to at least copy (or
even
edit) some files into some directories (configurations
and/or jars),
since
it does not seem to be easy/possible to do
everything on-the-fly,
programatically.
The only hard step is when changing the supported languages, I think.
In this case, when automatically generating the index, you need to warn
the user.
The admin UI should have a checkbox "use generated schema" or a textarea
for the schema.
(then
require reindexing).
hope it helps.
Yes, very helpful so far. I`m counting on your expertise with Lucene/Solr
on the details. My current approach is a practical one without previous
experience on the topic, so I`m still doing mostly guesswork in some
Thanks,
Eduard
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
paul
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
28 Nov
28 Nov
3:41 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Hi Ludovic,
Thanks for the reply. Please read below...
On Tue, Nov 27, 2012 at 5:44 PM, Ludovic Dubost <ludovic(a)xwiki.com> wrote:
Hi Edy,
I'm not a huge fan of the title_fr title_en title_morelanguages approach as
indeed it seems to be quite complex at the query level. I was more leaning
towards multiple indexes if we can query them globally but I understand
this is complex too.
Now let's see the use cases that are hugely important:
1/ Make sure that if you decide your wiki is monolingual:
- the indexing uses the specific language analyzer
- make sure the query uses the specific language analyzer
- make sure the search looks in all content even if the language setting
of the document is wrongly set (consider all documents being of the
specific language)
You mean that, if the wiki is monolingual, we should ignore the language
filter and hardcode it to "All languages", right?
However, what would be the advantage of this? Why would be want to pollute
the results with irrelevant documents (caused by a probable recent
configuration change that went from multi-lingual to mono-lignual)? Wasn't
that the whole reason why the admin switched to mono-lingual?
2/ Allow is a wiki is multi-lingual:
- search in the language you decide (maybe the UI should display a
language choice for the query)
We already support this, by using the "Filtered Search" option and
selecting the language.
- search in content that is analyzed in the proper language when the
content is declared in this language
- allow to specify if you want to restrict your search to documents
declared in the language of your query, versus search more widely in all
documents accross languages. If you search in only the language of the
query only one document can show up but it should point to the right
translation that matches, if you search in multiple languages then you can
show individual translations.
I think this one is the same as the first bullet above.
- allow technical users to search for all documents
across all languages
(where the language analysis does not really matter)
Do you mean as an API?
What exactly do you mean by "language analysis does not really matter"? Any
Example?
From an admin point of view it makes good sense for the admin to be able to
specify in a multilingual wiki which language analysis should be activated,
and then have this transmitted to SOLR to properly configure the engine.
Reindexing is ok when changing the configuration.
I believe in the end wether you use multiple fields with _fr _en or
multiple SOLR cores, as long as you can query accross SOLR cores is a bit
the same. If you cannot run a query merging multiple indexes then the first
solution is kind of absolutely necessary as it would be the only one
allowing to search across all languages.
Maybe a solution would be to create one index per language and index ALL
content regardless of it's language using the language analyzer of that
index. This would allow to have better results even though the users have
badly tagged the language of a document, and it's only the job of the UI to
limit the search to only the language of the query, or all documents.
So you could have a configuration in the admin that
says:
1/ Create an English Index
2/ Create an additional French index
The UI would allow to search in English and French, + would add a language
restriction for the documents.
Applying the language specific analyzers (for Chinese, for example) to all
the documents will just create a mess for all the documents that do not
match the analyzer's language. I`m not sure the results for the
badly-indexed languages will make any sense to users.
Also, this is very similar to the multi-core approach (one core per
language), just that you also add documents that are indexed with the wrong
analyzers. We have the same problem regarding merging relevance scores
across indexes (cores) that is a big turn-off for the original multi-core
approach.
In the future if we are able to "detect" the language of the documents we
could add a lucene field with the "detected" language instead of the
"provided" language of the documents, therefore increasing the quality of
searches only on documents of a specific language.
In the previous discussions (on the GSoC thread) we agreed that language in
XWiki is known before-hard, so no recognition is required, at least not at
document level.
This later solution would be the only one that would really work on file
attachements as we have no information about the specific language of file
attachements (or even XWiki objects) which are attached to the main
document and not to the translated document.
Yes, this is a problem right now. AFAIU, the plan [1] is to support
translated objects and maybe attachments as well. Until then, we could
either:
1) Use the original document's language to index the attachment's content
2) Use a language detection library to try to detect the attachment
content's language and index it accordingly.
The above could also be applied for objects and their properties.
----------
[1] http://jira.xwiki.org/browse/XWIKI-69
This later issues shows that a search on "only french content" should still
include the attachements because we have no idea if the attachments are
"french" or "english".
(The paragraphs below discuss on what currently exists and what could be
done, ignoring the possible language detection mentioned above)
Right now a document also indexes the object's properties in a field called
"objcontent". I do this for all translations, thus duplicating the field's
value in all translations. I can do the same for attachments. The purpose
is, indeed, to be able to find document translations based on hits in their
objects/attachments. If a language filter is used and there is a hit in an
object, only one document is returned. If there are no language filters,
all translations will be returned.
However, if we search for the object/property/attachment itself, it will
only be assigned to one language: the language of the original document.
This means that if we search for all languages, the object itself will be
found too (there is no language filter used). If we add a language filter
that is different from the object/property/attachment's original document
language, the object/property/attachment will not be found.
Maybe we can come up with some processing of the query in the search
application, that applies the language filter only for documents:
((-type:"OBJECT" OR -type:"OBJECT_PROPERTY" OR
-type:"ATTACHMENT") OR
lang:"<userSelectedLanguage>") -- writing it like this because the
default
operand is AND in the query filter clause that we use in the Search
application.
The problem with this is that that, when a language filter is used, the
object/property/attachments that are now included in the results might not
have the specified language and will pollute the results.
Thanks,
Eduard
Ludovic
2012/11/26 Eduard Moraru <enygma2002(a)gmail.com>
APIs
> to the platform, there will be (as as it is currently with Lucene) a lot
work
the
be
and
> then all the resources specific to all the selected languages (stopwords,
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
and
I
documents
and
to
this
not
not
that
is
warn
Lucene/Solr
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
Hi devs,
Any other input on this matter?
To summarize a bit, if we go with the multiple fields for each language,
we
end up with an index like:
English version:
id: xwiki:Main.SomeDocument_en
language: en
space: Main
title_en: XWiki document
doccontent_en: This is some content
French version:
id: xwiki:Main.SomeDocument_fr
language: fr
space: Main
title_fr: XWiki document
doccontent_fr: This is some content
The Solr configuration is generated by some XWiki UI that returns a zip
that the admin has to unpack in his (remote) Solr instance. This could be
automated for the embedded instance. This operation is to be performed
each
time an admin changes the indexed languages
(rarely or even only once).
Querying such a schema is a bit tricky when you are interested in more
than
one language, because you have to add all the
clauses (title_en,
title_fr,
etc.) specific to the languages you are
interested in.
Some extra fields might also be added like title_ws (for whitespace
tokenization only) that have various approaches to the indexing
operation,
with the aim of improving the relevancy.
One solution to simplify the query for API clients would be to use fields
like "title" and "doccontent" and to put as values very lightly (or
not
at
all) analyzed content, as Paul suggested. This
would allow applications
to
write simple (and backwards compatible maybe)
queries that will still
work,
but will not catch some of the nuances of
specific languages. As far as
I`ve seen until now, applications are not very interested in nuances, but
rather in filtering the results, a task for which this solution might be
well suited. Of course, nothing stops applications from using the *new*
and
more expressive fields that are properly
analized.
Thus, the search application will be the major beneficiary of these
analyzed fields (title_en, title_fr, etc.), while still allowing
applications to get their job done (trough generic, but less/not analized
fields like "title", "doccontent", etc.).
WDYT?
Thanks,
Eduard
On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru <enygma2002(a)gmail.com
expensive
wrote:
Hi Paul,
I was counting on your feedback :)
On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <paul(a)hoplahup.net>
wrote:
>
> Hello Eduard,
>
> it's nice of you to see you take this further.
>
> > This issue has already been previously [1] discussed during the GSoC
> > project, but I am not particularly happy with the chosen approach.
> > When handling multiple languages, there are generally[2][3] 3
different
> > approaches:
> >
> > 1) Indexing the content in a single field (like title, doccontent,
etc.)
> > - This has the advantage that queries
are clear and fast
> > - The disadvantage is that you can not run very well tuned analyzers
on
> the
> > fields, having to resort to (at best) basic tokenization and
> lowercasing.
> >
> > 2) Indexing the content in multiple fields, one field for each
language
> > (like title_en, title_fr, doccontent_en,
doccontent_fr, etc.)
> > - This has the advantage that you can easily specify (as dynamic
fields)
> > that *_en fields are of type text_en
(and analyzed by an
> english-centered
> > chain of analyzers); *_fr of type text_fr (focused on french, etc.),
> thus
> > making the results much better.
>
> I would add one more field here: title_ws and text_ws where the full
text
> is analyzed just as words (using the
whitespace-tokenizer?).
> A match there would even be preferred to a match in the below
text-fields.
>
> (maybe that would be called title and text?)
>
> > - The disadvantage is that querying such a schema is a pain. If you
want
>> > all the results in all languages, you end up with a big and >> > query.
>>
>> Why is this an issue?
>> Dismax does it for you for free (thanks to the "form" parameter that
>> gives weight to each of the fields).
>> This is an issue only if you start to have more than 100 languages or
>> so...
>> Lucene, the underlying engine of solr, handles thousands of clauses
in a
> query
without an issue (this is how prefix-queries are handled... they
are
> expanded to a query for any of the term that
matches the prefix, a
setting
deep
somewhere, which is about 2000 avoids this to explode).
Sure, Solr is great when you want to do simple queries like "XWiki Open
Source", however, since in XWiki we also expose the Solr/Lucene query of
limited
extensions wanting to do search using this API.
These extensions (like
the
> search suggest for example, rest search, etc) want to do something like
> "title:'Open Source' AND type:document AND doccontent:XWiki".
Because
> option 2) is so messy in it's fields, it would mean that the extension
> would have to come up with a query like "title_en:'Open Source' AND
> type:document AND doccontent_en:XWiki" (assuming that it is only to
> the current -- english or whatever -- language; what happens if it
wants
to
> do that no matter what language? It will have to specify each
combination
possible
because we can't use generic field names).
Solr's approach works for using it in your web application's search
input,
in a specific usecase, where you have precisely
specified the default
search fields and their boosts inside your schema.xml. However, as a
search
API, using option 2) you are making the life of
anyone else wanting to
use
> the Solr search API really hard. Also, your search application will > nicely when the user enters a simple query
in the input field, but an
> advanced user will suffer the same fate when trying to write an
advanced
> query, thus not relying on the default query
(computed by solr based on
> schema.xml).
>
> Also, based on your note above regarding improvements like title_ws and
> such, again, all of these are very well suited for the search
application
use case,
together with the default query that you configure in
schema.xml,
making the search results perform really well.
However, what does all
these
> fields mean to another extension wanting to do search? Will it have to
> handle all these implementation details to query for title, content and
> such? I`m not sure how well this would work in practice.
>
> Unrealistic idea(?): perhaps we should come up with an abstract search
> language (Solr/Lucene clone) that parses the searched fields andhides > complexities of all the indexed fields,
allowing to write simple
queries
like
"title:XWiki", while this gets translated to "title_en:XWiki OR
title_fr:XWiki OR title_de:XWiki..." :)
Am I approaching this wrong by trying to have both a tweakable/tweaked
search application AND a search API? Are the two not compatible? Do we
have
> to sacrifice search result performance (no language-specific stuff) to > able to have a usable API?
>
>
>> > If you want just some language, you have to read the right fields
>> > (ex title_en) instead of just getting a clear field name (title).
>>
>> You have to be careful, this is really only if you want to be
specific.
>> In this case, it is likely that you also
do not want so much stemming.
>> My experience, which was before dismax on curriki.org, has made it so
>> that any query that is a bit specific is likely to not desire
stemming.
>>
>
> Can you please elaborate on this? I`m not sure I understand the
problem.
>
>
>>
>> > -- Also, the schema.xml definition is a static one in this concern,
>> > requiring you to know beforehand which languages you want to support
>> (for
>> > example when defining the default fields to search for). Adding a
new
> >
language requires you to start editing the xml files by hand.
>
> True but the available languages are almost all hand-coded.
> You could generate the schema.xml based on the available languages if
not
hand-generated?
Basically I would have to output a zip with schema.xml, solrconfig.xml > synonims, etc) for the languages that we can
provide out of the box.
For
> other languages, the admin would have to get
dirty with the xmls.
>
>
>>
>> There's one catch with this approach which is new to me but seems to
be
>> quite important to implement this
approach: the idf should be
modified,
the
>> Similarity class should be, so that the total number of documents is
the
> total
number of documents having that language.
> See:
>
>
> The
solution sketched there sounds easy but I have not tried it.
>
> > 3) Indexing the content in different Solr cores (indexes), one for
each
> > language. Each core requires it's on
directory and configuration
files.
>> > - The advantage is that queries are clean to write (like option 1) >> that
>> > you have a nice separation
>> > - The disadvantage is that it's difficult to get it right
>> (administrative
>> > issues) and then you also have the (considerable) problem of having
to
>> fix
>> > the relevancy score of a query result that has entries from
different
>> > cores; each core has it's own
relevancy computed and does not
consider
>> the
>> > others.
>> > - To make it even worst, it seems that you can not [5] also push to
a
>> > remote Solr instance the
configuration files when creating a new
core
>> > programatically. However, if we are
running an embedded Solr
instance,
>> we
>> > could provide a way to generate the config files and write them to
the
>> data
>> > directory.
>>
>> Post-processing results is very very very dangerous as performance is
at
> risk
(e.g. if a core does not answer)... I would tend to avoid that as
much
>> as possible.
>>
>
> Not really related, but this reminds me about the post processing that > do for checking view rights over the
returned result, but that's
another
discussion that we will probably need to have :)
>
> > Currently I have implemented option 1) in our existing Solr
integration,
> > which is also more or less compatible
with our existing Lucene
queries,
>> but
>> > I would like to find a better solution that actually analyses the
>> content.
>> >
>> > During GSoC, option 2) was preferred but the implementation did not
>> > consider practical reasons like the ones described above (query
>> complexity,
>> > user configuration, etc.)
>>
>> True, Savitha surfed the possibility of having different solr >> per language.
>> I still could not be sure that this was not showing the document match
>> single in one language.
>>
>> However, indicating which language it is matched into is probably
>> useful...
>>
>
> Already doing that.
>
>
>> Funnily, cross-language-retrieval is a mature research field but
>> retrieval for multilanguage user is not so!
>>
>> > On a related note, I have also watched an interesting presentation
[3]
> >
about how Drupal handles its Solr integration and, particularly, a
> plugin
> > [4] that handles the multilingual aspect.
> > The idea seen there is that you have this UI that helps you generate
> > configuration files, depending you your needs. For instance, you
(admin)
>> > check that you need search for language English, French and German >> the
>> > ui/extension gives you a zip with the configuration you need to use
in
> your
> > (remote or embedded) solr instance. The configuration for each
language
> > comes preset with the analyzers you
should use for it and the
additional
>> > resources (stopwords.txt, synonims.txt, etc.).
>> > This approach helps with avoiding the need for admins to be forced >> edit
>> > xml files and could also still be useful for other cases, not only
>> option
>> > 2).
>>
>> Generating sounds like an easy approach to me.
>>
>
> Yes, however I don`t like the fact that we can not do everything from
the
> webapp and the admin needs to access the
filesystem to install the
given
> configuration on the embedded/remote solr
directory. Lucene does not
have
this
problem now. It just works with XWiki and everything is done from
XWiki UI. I feel that losing this commodity will not be very well
received
by users that now have some new install steps to
get XWiki running.
Well, of course, for the embedded solr version, we could handle it like
we
do now and push the files directly from the
webapp to the filesystem.
Since
embedded will be default, it should be OK and
avoid the extra install
step.
Users with a remote solr machine should have the
option to get the zip
instead.
Not sure if we can apply the new configuration without a restart, but
I`ll
have to look more into it. I know the multi-core
architecture supports
something like this but will have to see the details.
>
> > All these problems basically come from the fact that there is no way
to
> > specify in the schema.xml that, based on
the value of a field (like
the
>> > field "lang" that stores the document language), you want to run
or
>> > that group of analyzers.
>>
>> Well, this is possible with ThreadLocal but is not necessarily a good
>> idea.
>> Also, it is very common that users formulate queries without
formulating
>> their language and thus you need to
"or" the user's queries through
>> multiple languages (e.g. given by the browser).
>>
>> > Perhaps a solution would be a custom kind of "AggregatorAnalyzer"
that
>> > would call other analyzers at
runtime, based on the value of the
lang
> >
field. However, this solution could only be applied at index time,
when
> you
> > have the lang information (in the solrDocument to be indexed), but
when
>> you
>> > perform the query, you can not analyze the query text since you do > know
> > the language of the field you're querying (it was determined at
runtime
>> -
>> > at index time) and thus do not know what operations to apply to the
>> query
>> > (to reduce it to the same form as the indexed values).
>>
>> How would that look at query time?
>>
>
> That's what I was saying, that at query time, the searched term will get
analyzed by the right chain. When you search for a single language,
you
could add that language as a query filter and
then you could apply the
right chain, but when searching in 2 or more (or no, meaning all)
languages
> you are stuck.
>
>>
>> > I have also read another interesting analysis [6] on this problem > >
elaborates on the complexities and limitations of each options.
(Ignore
>> the
>> > Rosette stuff mentioned there)
>> >
>> > I have been thinking about this for some time now, but the solution >> > probably somewhere in between,
finding an option that is acceptable
>> while
>> > not restrictive. I will probably also send a mail on the Solr list
to
> get
> > some more input from there, but I get the feeling that whatever
> solution we
> > choose, it will most likely require the users to at least copy (or
even
>> > edit) some files into some directories (configurations and/or jars),
>> since
>> > it does not seem to be easy/possible to do everything on-the-fly,
>> > programatically.
>>
>> The only hard step is when changing the supported languages, I think.
>> In this case, when automatically generating the index, you need to >> the user.
>> The admin UI should have a checkbox "use generated schema" or a
textarea
>> for the schema.
>>
>
> Please see above regarding configuration generation. Basically, since
we
> are going to support both embedded and
remote solr instances, we could
> support things like editing the schema from XWiki only for the embedded
> instance, but not for the remote one. We might end up having separate
UIs
for each
case, since we might want to exploit the flexibility of the
embedded one as much as possible.
>
> Those that want particular fields and tunings need to write their own
> schema.
>
> The same UI could also include whether to include a phonetic track or
not
>> (then require reindexing).
>
>
>> hope it helps.
>>
>
> Yes, very helpful so far. I`m counting on your expertise with on the
details. My current approach is a practical one without previous
experience on the topic, so I`m still doing mostly guesswork in some
areas.
Thanks,
Eduard
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
paul
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
4:35 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
2012/11/28 Eduard Moraru <enygma2002(a)gmail.com>
Sure. I was just repeating what is important.
Not specifically as an API
I undestand the issue here, but in most cases the user will say "french
search" on "french content", he will only expand to non french content if
he was not satisfied by his search. What is just allowed here is to also
search in "french" inside all content. That would cover content that would
have the bad language setting as well as any other content. The results
might be a bit noisy but I don't think it's a big issue.
This is not a good solution. If I understand correctly we could not end up
not searching in a french attachment because the original document is in
marked "en".
I'm for Paul's solution to index objects and attachments in each
translation (if we have separate entities for translated documents). I
understand that in the title_fr,content_fr approach this problem does not
happen.
Hi Ludovic,
Thanks for the reply. Please read below...
On Tue, Nov 27, 2012 at 5:44 PM, Ludovic Dubost <ludovic(a)xwiki.com> wrote:
If the user is monolingual, we can safely ignore the language setting of
each document and only the "main" document will be shown anyway to the user.
So we should make sure we search on ALL documents that are available to the
user.
The user might have to "reindex" to make sure this is properly taken into
account by the search engine.
What is important here, is that even if the wiki is set to "fr", then even
if documents have "en" for the main language they will still show up in the
search. The opposite would be bad.
Hi Edy,
I'm not a huge fan of the title_fr title_en title_morelanguages approach
as
indeed it seems to be quite complex at the query
level. I was more
leaning
towards multiple indexes if we can query them
globally but I understand
this is complex too.
Now let's see the use cases that are hugely important:
1/ Make sure that if you decide your wiki is monolingual:
- the indexing uses the specific language analyzer
- make sure the query uses the specific language analyzer
- make sure the search looks in all content even if the language
setting
of the document is wrongly set (consider all
documents being of the
specific language)
You mean that, if the wiki is monolingual, we should ignore the language
filter and hardcode it to "All languages", right?
However, what would be the advantage of this? Why would be want to pollute
the results with irrelevant documents (caused by a probable recent
configuration change that went from multi-lingual to mono-lignual)? Wasn't
that the whole reason why the admin switched to mono-lingual?
2/ Allow is a wiki is multi-lingual:
- search in the language you decide (maybe the UI should display a
language choice for the query)
We already support this, by using the "Filtered Search" option and
selecting the language.
- search in content that is analyzed in the proper language when the
"French" but look in the "French" + "English" dataset.
For me if both
documents with the same page name match, both should come out separately.
content is declared in this language
- allow to specify if you want to restrict your search to documents
declared in the language of your query, versus search more widely in all
documents accross languages. If you search in only the language of the
query only one document can show up but it should point to the right
translation that matches, if you search in multiple languages then you
can
show individual translations.
I think this one is the same as the first bullet above.
No I think it's different. The user can decide to make his search in
- allow technical users to search for all
documents across all
languages
(where the language analysis does not really
matter)
Do you mean as an API?
What exactly do you mean by "language analysis does not really matter"? Any
Example?
I mean here that as a technical user your objective is to make sure your
search spans ALL content in the wiki. In this case you don't care about
stemming and such.
The standard UI could take this into account by choosing "any language
search" and with data set "all languages". We just need to make sure that
this won't exclude any content of the search
(Here is an example of a case where such exclusion might occur. Suppose you
have done only "French" and "English" indexing and there is
"German"
content also in your wiki but since you have not asked for German search in
your wiki you don't have title_de and content_de fields or don't have a
german specific solr index (in the other method), then your german content
would not be indexed at all ?)
From an admin point of view it makes good sense for the admin to be able
to
specify in a multilingual wiki which language
analysis should be
activated,
and then have this transmitted to SOLR to
properly configure the engine.
Reindexing is ok when changing the configuration.
I believe in the end wether you use multiple fields with _fr _en or
multiple SOLR cores, as long as you can query accross SOLR cores is a bit
the same. If you cannot run a query merging multiple indexes then the
first
solution is kind of absolutely necessary as it
would be the only one
allowing to search across all languages.
Maybe a solution would be to create one index per language and index ALL
content regardless of it's language using the language analyzer of that
index. This would allow to have better results even though the users have
badly tagged the language of a document, and it's only the job of the UI
to
limit the search to only the language of the
query, or all documents.
So you could have a configuration in the admin
that says:
1/ Create an English Index
2/ Create an additional French index
The UI would allow to search in English and French, + would add a
language
restriction for the documents.
Applying the language specific analyzers (for Chinese, for example) to all
the documents will just create a mess for all the documents that do not
match the analyzer's language. I`m not sure the results for the
badly-indexed languages will make any sense to users.
Also, this is very similar to the multi-core approach
(one core per
language), just that you also add documents that are indexed with the wrong
analyzers. We have the same problem regarding merging relevance scores
across indexes (cores) that is a big turn-off for the original multi-core
approach.
This is a more serious issue. If it's hard to merge the results spanning
over
multiple cores this could be a showstopper. However the solution of
having only one Lucene document for all languages is not so cool either as
it would make it difficult to know which ones of the languages has matched
and present them separately with separate scores.
It's really the core issue to decide on. What are the benefits and
drawbacks of the different solutions. For each solution is there something
in the UI that you cannot do ?
So far I've heard:
1/ Presenting different scores for documents in different languages with
the same doc name if the title_fr,content_fr method is used
2/ Merging scores accross indexes in multicore approach
Other ? Can we list them in a wiki page ?
In the future if we are able to "detect" the language of the documents we
could add a lucene field with the "detected" language instead of the
"provided" language of the documents, therefore increasing the quality of
searches only on documents of a specific language.
In the previous discussions (on the GSoC thread) we agreed that language in
XWiki is known before-hard, so no recognition is required, at least not at
document level.
Let's forget this
This later solution would be the only one that would really work on file
attachements as we have no information about the specific language of
file
attachements (or even XWiki objects) which are
attached to the main
document and not to the translated document.
Yes, this is a problem right now. AFAIU, the plan [1] is to support
translated objects and maybe attachments as well. Until then, we could
either:
1) Use the original document's language to index the attachment's content
2) Use a language detection library to try to detect
the attachment
content's language and index it accordingly.
Not sure we can for now
The above could also be applied for objects and their
properties.
----------
[1] http://jira.xwiki.org/browse/XWIKI-69
It seems we have to do this for now
This later issues shows that a search on "only french content" should
still
include the attachements because we have no idea
if the attachments are
"french" or "english".
(The paragraphs below discuss on what currently exists and what could be
done, ignoring the possible language detection mentioned above)
Right now a document also indexes the object's properties in a field called
"objcontent". I do this for all translations, thus duplicating the field's
value in all translations. I can do the same for attachments. The purpose
is, indeed, to be able to find document translations based on hits in their
objects/attachments. If a language filter is used and there is a hit in an
object, only one document is returned. If there are no language filters,
all translations will be returned.
However, if we search for the object/property/attachment itself, it will
only be assigned to one language: the language of the original document.
This means that if we search for all languages, the object itself will be
found too (there is no language filter used). If we add a language filter
that is different from the object/property/attachment's original document
language, the object/property/attachment will not be found.
Maybe we can come up with some processing of the query in the search
application, that applies the language filter only for documents:
((-type:"OBJECT" OR -type:"OBJECT_PROPERTY" OR
-type:"ATTACHMENT") OR
lang:"<userSelectedLanguage>") -- writing it like this because the
default
operand is AND in the query filter clause that we use in the Search
application.
The problem with this is that that, when a language filter is used, the
object/property/attachments that are now included in the results might not
have the specified language and will pollute the results.
I'm not sure I understand. We have an "objcontent" field for each
translation that has the full content of objects and properties, but
individual object fields we don't have in each translations ?
The more I see all the issues, the more I lean towards a separate index per
language solution. The reason I do is that the main need is for a non
English user to have very relevant results in his own language. Therefore
we need to make sure that all content that the users have published has the
chance to be analyzed using the non English language analyzer. So indexing
all objects and attachments with the relevant language analyzer is the
solution. This is also why I proposed to index all content in this specific
index regardless of the language declared, which would only be used in the
UI to limit searches to the specific language.
In this view:
0/ There would be a language specific index per language with the objects
and attachments indexed only in the language of the index
1/ The user chooses the language in which he searches
2/ Automatically that sets the index to be used to be the "french" index
3/ Automatically that presets to limit the span of the search to declared
"french" documents
4/ The user can decide to go for non french documents at his own risks
knowing that the results might be weird because of wrong analysis (this is
what happens today with english analysis over french documents)
The benefit here is that you don't have a merging score over multiple index
issue, since you would never have to do a search across multiple indexes.
Searches are still simple to write. By default results are quite relevant
since you limit the search the french declared documents (this would be the
same as limiting your search to title_fr, content_fr) and still cover what
needs to be covered (objects and attachments).
Another benefit is that this falls back gracefully to monolingual as you
just have to have one index in the language declared for the monolingual
wiki.
The drawback is that the indexing is more costly and there is duplicated
content in the index. Howerver it is the Admin that say which languages he
wants available and he takes responsibility of the ressources this needs.
Could this solution work ?
Ludovic
Thanks,
Eduard
fields
but
analized
or
they
lot
> of
> > extensions wanting to do search using this API. These extensions (like
to
on
to
tweakable/tweaked
concern,
if
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
and
> >> that
> >> > you have a nice separation
> >> > - The disadvantage is that it's difficult to get it right
> >> (administrative
> >> > issues) and then you also have the (considerable) problem of
having
match
generate
and
> >> the
> >> > ui/extension gives you a zip with the configuration you need to
use
good
not
> >> know
> >> > the language of the field you're querying (it was determined at
> runtime
> >> -
> >> > at index time) and thus do not know what operations to apply to
the
language,
that
> >> > elaborates on the complexities and limitations of each options.
> (Ignore
> >> the
> >> > Rosette stuff mentioned there)
> >> >
> >> > I have been thinking about this for some time now, but the
solution
jars),
could
own
previous
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
Ludovic
2012/11/26 Eduard Moraru <enygma2002(a)gmail.com>
> Hi devs,
>
> Any other input on this matter?
>
> To summarize a bit, if we go with the multiple fields for each
language,
we
> end up with an index like:
>
> English version:
> id: xwiki:Main.SomeDocument_en
> language: en
> space: Main
> title_en: XWiki document
> doccontent_en: This is some content
>
> French version:
> id: xwiki:Main.SomeDocument_fr
> language: fr
> space: Main
> title_fr: XWiki document
> doccontent_fr: This is some content
>
> The Solr configuration is generated by some XWiki UI that returns a zip
> that the admin has to unpack in his (remote) Solr instance. This could
be
automated
for the embedded instance. This operation is to be performed
each
time an admin changes the indexed languages
(rarely or even only once).
Querying such a schema is a bit tricky when you are interested in more
than
one language, because you have to add all the
clauses (title_en,
title_fr,
etc.) specific to the languages you are
interested in.
Some extra fields might also be added like title_ws (for whitespace
tokenization only) that have various approaches to the indexing
operation,
> with the aim of improving the relevancy.
>
> One solution to simplify the query for API clients would be to use like
"title" and "doccontent" and to put as values very lightly (or not
at
all) analyzed content, as Paul suggested. This
would allow applications
to
write simple (and backwards compatible maybe)
queries that will still
work,
> but will not catch some of the nuances of specific languages. As far as
> I`ve seen until now, applications are not very interested in nuances, > rather in filtering the results, a task for
which this solution might
be
well
suited. Of course, nothing stops applications from using the *new*
and
> more expressive fields that are properly analized.
>
> Thus, the search application will be the major beneficiary of these
> analyzed fields (title_en, title_fr, etc.), while still allowing
> applications to get their job done (trough generic, but less/not > fields like "title",
"doccontent", etc.).
>
> WDYT?
>
> Thanks,
> Eduard
>
>
>
>
> On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru <enygma2002(a)gmail.com
> >wrote:
>
> > Hi Paul,
> >
> > I was counting on your feedback :)
> >
> > On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <paul(a)hoplahup.net>
> wrote:
> >
> >>
> >> Hello Eduard,
> >>
> >> it's nice of you to see you take this further.
> >>
> >> > This issue has already been previously [1] discussed during the
GSoC
> >> > project, but I am not
particularly happy with the chosen approach.
> >> > When handling multiple languages, there are generally[2][3] 3
> different
> >> > approaches:
> >> >
> >> > 1) Indexing the content in a single field (like title, doccontent,
> etc.)
> >> > - This has the advantage that queries are clear and fast
> >> > - The disadvantage is that you can not run very well tuned
analyzers
> on
> >> the
> >> > fields, having to resort to (at best) basic tokenization and
> >> lowercasing.
> >> >
> >> > 2) Indexing the content in multiple fields, one field for each
> language
> >> > (like title_en, title_fr, doccontent_en, doccontent_fr, etc.)
> >> > - This has the advantage that you can easily specify (as dynamic
> fields)
> >> > that *_en fields are of type text_en (and analyzed by an
> >> english-centered
> >> > chain of analyzers); *_fr of type text_fr (focused on french,
etc.),
> >> thus
> >> > making the results much better.
> >>
> >> I would add one more field here: title_ws and text_ws where the full
> text
> >> is analyzed just as words (using the whitespace-tokenizer?).
> >> A match there would even be preferred to a match in the below
> text-fields.
> >>
> >> (maybe that would be called title and text?)
> >>
> >> > - The disadvantage is that querying such a schema is a pain. If
you
want
>> > all the results in all languages, you end up with a big and
expensive
> >> > query.
> >>
> >> Why is this an issue?
> >> Dismax does it for you for free (thanks to the "form" parameter
that
> >> gives weight to each of the fields).
> >> This is an issue only if you start to have more than 100 languages >>
so...
>> Lucene, the underlying engine of solr, handles thousands of clauses
in a
> >> query without an issue (this is how prefix-queries are handled... > are
> >> expanded to a query for any of the term that matches the prefix, a
> setting
> >> deep somewhere, which is about 2000 avoids this to explode).
> >>
> >
> > Sure, Solr is great when you want to do simple queries like "XWiki
Open
Source", however, since in XWiki we also expose
the Solr/Lucene query
APIs
> to the platform, there will be (as as it is currently with Lucene) a > the
> > search suggest for example, rest search, etc) want to do something
like
> > "title:'Open Source' AND
type:document AND doccontent:XWiki". Because
> > option 2) is so messy in it's fields, it would mean that the
extension
>
would have to come up with a query like "title_en:'Open Source' AND
> type:document AND doccontent_en:XWiki" (assuming that it is only
limited
to
> the current -- english or whatever -- language; what happens if it
wants
to
> do that no matter what language? It will have to specify each
combination
> > possible because we can't use generic field names).
> >
> > Solr's approach works for using it in your web application's search
> input,
> > in a specific usecase, where you have precisely specified the default
> > search fields and their boosts inside your schema.xml. However, as a
> search
> > API, using option 2) you are making the life of anyone else wanting use
> the Solr search API really hard. Also, your search application will
work
> nicely when the user enters a simple query
in the input field, but an
> advanced user will suffer the same fate when trying to write an
advanced
> > query, thus not relying on the default query (computed by solr based > > schema.xml).
> >
> > Also, based on your note above regarding improvements like title_ws
and
>
such, again, all of these are very well suited for the search
application
> > use case, together with the default query that you configure in
> schema.xml,
> > making the search results perform really well. However, what does all
> these
> > fields mean to another extension wanting to do search? Will it have > > handle all these implementation details
to query for title, content
and
> > such? I`m not sure how well this would
work in practice.
> >
> > Unrealistic idea(?): perhaps we should come up with an abstract
search
>
language (Solr/Lucene clone) that parses the searched fields andhides
the
> complexities of all the indexed fields,
allowing to write simple
queries
> > like "title:XWiki", while this gets translated to
"title_en:XWiki OR
> > title_fr:XWiki OR title_de:XWiki..." :)
> >
> > Am I approaching this wrong by trying to have both a > > search application AND a search API?
Are the two not compatible? Do
we
> have
> > to sacrifice search result performance (no language-specific stuff)
to
be
stemming.
> able to have a usable API?
>
>
>> > If you want just some language, you have to read the right fields
>> > (ex title_en) instead of just getting a clear field name (title).
>>
>> You have to be careful, this is really only if you want to be
specific.
> >> In this case, it is likely that you also do not want so much > >> My experience, which was before
dismax on curriki.org, has made it
so
>>
that any query that is a bit specific is likely to not desire
stemming.
>>
>
> Can you please elaborate on this? I`m not sure I understand the
problem.
> >
> >
> >>
> >> > -- Also, the schema.xml definition is a static one in this > >> > requiring you to know
beforehand which languages you want to
support
>>
(for
>> > example when defining the default fields to search for). Adding a
new
> >> > language requires you to start editing the xml files by hand.
> >>
> >> True but the available languages are almost all hand-coded.
> >> You could generate the schema.xml based on the available languages > not
> >> hand-generated?
> >>
> >
> > Basically I would have to output a zip with schema.xml,
solrconfig.xml
and
> then all the resources specific to all the selected languages
(stopwords,
> synonims, etc) for the languages that we can
provide out of the box.
For
> other languages, the admin would have to get
dirty with the xmls.
>
>
>>
>> There's one catch with this approach which is new to me but seems to
be
>> quite important to implement this
approach: the idf should be
modified,
the
>> Similarity class should be, so that the total number of documents is
the
> total
number of documents having that language.
> See:
>
>
> The solution sketched there sounds easy but I have
not tried it.
>
> > 3) Indexing the content in different Solr cores (indexes), one for
each
> > language. Each core requires it's on
directory and configuration
files.
>> > - The advantage is that queries are clean to write (like option 1) to
to
>> fix
>> > the relevancy score of a query result that has entries from
different
>> > cores; each core has it's own
relevancy computed and does not
consider
> >> the
> >> > others.
> >> > - To make it even worst, it seems that you can not [5] also push
a
is
>> > remote Solr instance the
configuration files when creating a new
core
>> > programatically. However, if we are
running an embedded Solr
instance,
>> we
>> > could provide a way to generate the config files and write them to
the
> >> data
> >> > directory.
> >>
> >> Post-processing results is very very very dangerous as performance at
> >> risk (e.g. if a core does not answer)... I would tend to avoid that
as
> much
> >> as possible.
> >>
> >
> > Not really related, but this reminds me about the post processing
that
I
not
> do for checking view rights over the
returned result, but that's
another
> > discussion that we will probably need to have :)
> >
> >
> >>
> >> > Currently I have implemented option 1) in our existing Solr
> integration,
> >> > which is also more or less compatible with our existing Lucene
> queries,
> >> but
> >> > I would like to find a better solution that actually analyses the
> >> content.
> >> >
> >> > During GSoC, option 2) was preferred but the implementation did >>
> consider practical reasons like the ones described above (query
>> complexity,
>> > user configuration, etc.)
>>
>> True, Savitha surfed the possibility of having different solr
documents
> >> per language.
> >> I still could not be sure that this was not showing the document >>
single in one language.
>>
>> However, indicating which language it is matched into is probably
>> useful...
>>
>
> Already doing that.
>
>
>> Funnily, cross-language-retrieval is a mature research field but
>> retrieval for multilanguage user is not so!
>>
>> > On a related note, I have also watched an interesting presentation
[3]
> >> > about how Drupal handles its Solr integration and, particularly, a
> >> plugin
> >> > [4] that handles the multilingual aspect.
> >> > The idea seen there is that you have this UI that helps you > > configuration files, depending you your
needs. For instance, you
(admin)
>> > check that you need search for language English, French and German in
to
from
> your
> > (remote or embedded) solr instance. The configuration for each
language
> > comes preset with the analyzers you
should use for it and the
additional
>> > resources (stopwords.txt, synonims.txt, etc.).
>> > This approach helps with avoiding the need for admins to be forced >> edit
>> > xml files and could also still be useful for other cases, not only
>> option
>> > 2).
>>
>> Generating sounds like an easy approach to me.
>>
>
> Yes, however I don`t like the fact that we can not do everything from
the
> webapp and the admin needs to access the
filesystem to install the
given
> configuration on the embedded/remote solr
directory. Lucene does not
have
> > this problem now. It just works with XWiki and everything is done > > XWiki UI. I feel that losing this
commodity will not be very well
> received
> > by users that now have some new install steps to get XWiki running.
> >
> > Well, of course, for the embedded solr version, we could handle it
like
> we
> > do now and push the files directly from the webapp to the filesystem.
> Since
> > embedded will be default, it should be OK and avoid the extra install
> step.
> > Users with a remote solr machine should have the option to get the
zip
> > instead.
> >
> > Not sure if we can apply the new configuration without a restart, but
> I`ll
> > have to look more into it. I know the multi-core architecture
supports
> > something like this but will have to
see the details.
> >
> >
> >>
> >> > All these problems basically come from the fact that there is no
way
> to
> >> > specify in the schema.xml that, based on the value of a field
(like
the
>> > field "lang" that stores the document language), you want to run
this
> or
> >> > that group of analyzers.
> >>
> >> Well, this is possible with ThreadLocal but is not necessarily a >>
idea.
>> Also, it is very common that users formulate queries without
formulating
>> their language and thus you need to
"or" the user's queries through
>> multiple languages (e.g. given by the browser).
>>
>> > Perhaps a solution would be a custom kind of "AggregatorAnalyzer"
that
>> > would call other analyzers at
runtime, based on the value of the
lang
> >
field. However, this solution could only be applied at index time,
when
> you
> > have the lang information (in the solrDocument to be indexed), but
when
>> you
>> > perform the query, you can not analyze the query text since you do >>
query
>> > (to reduce it to the same form as the indexed values).
>>
>> How would that look at query time?
>>
>
> That's what I was saying, that at query time, the searched term will
not
> > get analyzed by the right chain. When you search for a single > you
> > could add that language as a query filter and then you could apply
the
right chain, but when searching in 2 or more (or no,
meaning all)
languages
> you are stuck.
>
>>
>> > I have also read another interesting analysis [6] on this problem is
> >> > probably somewhere in between, finding an option that is
acceptable
>>
while
>> > not restrictive. I will probably also send a mail on the Solr list
to
> >> get
> >> > some more input from there, but I get the feeling that whatever
> >> solution we
> >> > choose, it will most likely require the users to at least copy (or
> even
> >> > edit) some files into some directories (configurations and/or > >> since
> >> > it does not seem to be easy/possible to do everything on-the-fly,
> >> > programatically.
> >>
> >> The only hard step is when changing the supported languages, I
think.
>>
In this case, when automatically generating the index, you need to
warn
>> the user.
>> The admin UI should have a checkbox "use generated schema" or a
textarea
>> for the schema.
>>
>
> Please see above regarding configuration generation. Basically, since
we
> > are going to support both embedded and remote solr instances, we > > support things like editing the schema
from XWiki only for the
embedded
>
instance, but not for the remote one. We might end up having separate
UIs
> > for each case, since we might want to exploit the flexibility of the
> > embedded one as much as possible.
> >
> >
> >>
> >> Those that want particular fields and tunings need to write their > >> schema.
> >>
> >> The same UI could also include whether to include a phonetic track
or
not
>> (then require reindexing).
>
>
>> hope it helps.
>>
>
> Yes, very helpful so far. I`m counting on your expertise with
Lucene/Solr
> > on the details. My current approach is a practical one without experience on the topic, so I`m still doing mostly
guesswork in some
areas.
Thanks,
Eduard
> paul
> _______________________________________________
> devs mailing list
> devs(a)xwiki.org
> http://lists.xwiki.org/mailman/listinfo/devs
>
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
29 Nov
29 Nov
3:47 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Hi,
On Wed, Nov 28, 2012 at 5:35 PM, Ludovic Dubost <ludovic(a)xwiki.com> wrote:
Sure. I was just repeating what is important.
"French" but look in the "French" + "English" dataset. For
me if both
documents with the same page name match, both should come out separately.
It's the same as in the UI allows you to specify which language to search
in, but it also allows you to specify that you want to search in "All
languages", that is what I meant.
Not specifically as an API
I understand now what you mean. Well, to avoid this, we can use only one
notion and that is the one of "supported languages". All the supported
languages should be indexed in order for the Search feature to work as
expected (and not have the case where you have a document that is not
indexed because of its language). I am not sure if there are many examples
where admins would choose to index just some of the supported languages and
not index all of them.
WDYT?
ALL
What I think you are actually asking is that all languages are queried, but
that results in the requested language get boosted/elevated and thus are
displayed before all other language results.
Perhaps this can be achieved by using the lang field as a query field and
not as a query filter. This means that it will influence the score. Couple
that with a higher boost for the lang field and results in the requested
language should be scored better than all the rest (because there will
always be a hit for the lang field resulting in a better score). There
might still be some results from other languages that could have a better
score than results from the requested language (because they just have more
hits on diferent fields), but it should work in most cases.
However, I`m not sure we want this. Users might results from other
languages as a bug. Problems might also be harder to debug.
Instead, we could do something like Google and if we see a small number of
results, we could suggest that the user searches in all languages.
In any case, I`m not sure this is such a big issue right now, since it's
basically an optimisation that we could choose to do, or let the user do it
himself, since he has the tools (UI).
I agree. The one-document-for-all-languages solution was proposed by Paul,
but I don`t think it's the way to go. We are currently only considering the
one-document-per-language direction.
"objcontent" stores the properties (format: "propName:propValue") of
each
object inside the original document (multi-valued field).
Besides XWikiDocuments, we also index Objects, Properties and Attachments
as Lucene/Solr first-class documents. Each of these entries has the wiki,
space, name and lang fields set from the document to which they belong to.
This is what I meant above with "if we search for the
object/property/attachment itself"
So to reiterate, the idea was that if you want to search for an indexed
Object (type:"OBJECT"), you will *have* to avoid setting a language filter,
or you might not find the object you are looking for, since it is indexed
under the language of original document.
---
While writing this down, I just thought of an elegant idea to fix this. The
lang field could be set to multi-valued. This means that, when we index
objects, properties and attachments, we could set in the lang field all the
values corresponding to all the existing translations of the owning
document. Example:
An Object:
id: xwiki:Main.Page^XWiki.Class[0] <-- (object reference)
class: XWiki.Class
wiki: xwiki
space: Main
name: Page
lang: en <-- proposal to make it multi-valued would look like this,
stored like a list of values.
fr
de
type: OBJECT
Note that this solution would also affect document languages as well (since
they share the same schema), but we will just put one value and it will not
affect queries. Queries will still be written like: "lang:en"
If we apply this solution, even if a language filter exists in the user's
query, it will still hit the object because the lang field of the object
contains the value.
2012/11/28 Eduard Moraru <enygma2002(a)gmail.com>
Why/How can you end up with documents of other languages than the current
default one if the wiki is mono-lingual?
Hi Ludovic,
Thanks for the reply. Please read below...
On Tue, Nov 27, 2012 at 5:44 PM, Ludovic Dubost <ludovic(a)xwiki.com>
wrote:
> Hi Edy,
>
> I'm not a huge fan of the title_fr title_en title_morelanguages
approach
as
pollute
indeed it seems to be quite complex at the query
level. I was more
leaning
towards multiple indexes if we can query them
globally but I understand
this is complex too.
Now let's see the use cases that are hugely important:
1/ Make sure that if you decide your wiki is monolingual:
- the indexing uses the specific language analyzer
- make sure the query uses the specific language analyzer
- make sure the search looks in all content even if the language
setting
of the document is wrongly set (consider all
documents being of the
specific language)
You mean that, if the wiki is monolingual, we should ignore the language
filter and hardcode it to "All languages", right?
However, what would be the advantage of this? Why would be want to the results with irrelevant documents (caused by
a probable recent
configuration change that went from multi-lingual to mono-lignual)?
Wasn't
that the whole reason why the admin switched to
mono-lingual?
If the user is monolingual, we can safely ignore the language setting of
each document and only the "main" document will be shown anyway to the
user.
So we should make sure we search on ALL documents that are available to the
user.
The user might have to "reindex" to make sure this is properly taken into
account by the search engine.
What is important here, is that even if the wiki is set to "fr", then even
if documents have "en" for the main language they will still show up in the
search. The opposite would be bad.
2/ Allow is a wiki is multi-lingual:
- search in the language you decide (maybe the UI should display a
language choice for the query)
We already support this, by using the "Filtered Search" option and
selecting the language.
- search in content that is analyzed in the proper language when the
> content is declared in this language
> - allow to specify if you want to restrict your search to documents
> declared in the language of your query, versus search more widely in
all
documents
accross languages. If you search in only the language of the
query only one document can show up but it should point to the right
translation that matches, if you search in multiple languages then you
can
show individual translations.
I think this one is the same as the first bullet above.
No I think it's different. The user can decide to make his search in - allow technical users to search for all
documents across all
languages
(where the language analysis does not really
matter)
Do you mean as an API?
What exactly do you mean by "language analysis does not really matter"?
Any
Example?
I mean here that as a technical user your objective is to make sure your
search spans ALL content in the wiki. In this case you don't care about
stemming and such.
The standard UI could take this into account by choosing "any language
search" and with data set "all languages". We just need to make sure that
this won't exclude any content of the search
(Here is an example of a case where such exclusion might occur. Suppose you
have done only "French" and "English" indexing and there is
"German"
content also in your wiki but since you have not asked for German search in
your wiki you don't have title_de and content_de fields or don't have a
german specific solr index (in the other method), then your german content
would not be indexed at all ?)
>
> From an admin point of view it makes good sense for the admin to be
able
to
engine.
specify in a multilingual wiki which language
analysis should be
activated,
> and then have this transmitted to SOLR to properly configure the > Reindexing is ok when changing the
configuration.
>
> I believe in the end wether you use multiple fields with _fr _en or
> multiple SOLR cores, as long as you can query accross SOLR cores is a
bit
the same.
If you cannot run a query merging multiple indexes then the
first
> solution is kind of absolutely necessary as it would be the only one
> allowing to search across all languages.
>
> Maybe a solution would be to create one index per language and index > content regardless of it's language
using the language analyzer of that
> index. This would allow to have better results even though the users
have
> badly tagged the language of a document, and
it's only the job of the
UI
to
all
limit the search to only the language of the
query, or all documents.
So you could have a configuration in the admin
that says:
1/ Create an English Index
2/ Create an additional French index
The UI would allow to search in English and French, + would add a
language
restriction for the documents.
Applying the language specific analyzers (for Chinese, for example) to the documents will just create a mess for all the
documents that do not
match the analyzer's language. I`m not sure the results for the
badly-indexed languages will make any sense to users.
I undestand the issue here, but in most cases the user will say "french
search" on "french content", he will only expand to non french content if
he was not satisfied by his search. What is just allowed here is to also
search in "french" inside all content. That would cover content that would
have the bad language setting as well as any other content. The results
might be a bit noisy but I don't think it's a big issue.
Also, this is very similar to the multi-core
approach (one core per
language), just that you also add documents that are indexed with the
wrong
analyzers. We have the same problem regarding
merging relevance scores
across indexes (cores) that is a big turn-off for the original multi-core
approach.
This is a more serious issue. If it's hard to merge the results spanning
over
multiple cores this could be a showstopper. However the solution of
having only one Lucene document for all languages is not so cool either as
it would make it difficult to know which ones of the languages has matched
and present them separately with separate scores.
It's really the core issue to decide on. What are
the benefits and
drawbacks of the different solutions. For each solution is there something
in the UI that you cannot do ?
So far I've heard:
1/ Presenting different scores for documents in different languages with
the same doc name if the title_fr,content_fr method is used
Since we are not considering the one-document-for-all-languages solution,
this is not an issue.
2/ Merging scores accross indexes in multicore approach
Other ? Can we list them in a wiki page ?
in
This is not a good solution. If I understand correctly we could not end up
not searching in a french attachment because the original document is in
marked "en".
I'm for Paul's solution to index objects and attachments in each
translation (if we have separate entities for translated documents).
Yep, we already do this for objects, just need to do it for attachments as
well.
>
> In the future if we are able to "detect" the language of the documents
we
> could add a lucene field with the
"detected" language instead of the
> "provided" language of the documents, therefore increasing the quality
of
searches
only on documents of a specific language.
In the previous discussions (on the GSoC thread) we agreed that language XWiki is known before-hard, so no recognition is
required, at least not
at
document level.
Let's forget this
>
> This later solution would be the only one that would really work on
file
attachements as we have no information about the specific language of
file
attachements (or even XWiki objects) which are
attached to the main
document and not to the translated document.
Yes, this is a problem right now. AFAIU, the plan [1] is to support
translated objects and maybe attachments as well. Until then, we could
either:
1) Use the original document's language to index the attachment's content
I
understand that in the title_fr,content_fr approach this problem does not
happen.
This problem is not related to how we handle multiple translations
(separate indexes or not), as long as each translation is a separate
entity. Basically this last part is the problem. If the French translation
is a separate entity from the original document (i.e. that is in English),
any object/attachment *index field* of the English original version will
need to be duplicated into the French translation as well, or the French
translation risks not to get a hit on the object/attachment.
2) Use a language detection library to try to
detect the attachment
content's language and index it accordingly.
Not sure we can for now
The above could also be applied for objects and
their properties.
----------
[1] http://jira.xwiki.org/browse/XWIKI-69
called
This later issues shows that a search on "only french content" should
still
include the attachements because we have no idea
if the attachments are
"french" or "english".
(The paragraphs below discuss on what currently exists and what could be
done, ignoring the possible language detection mentioned above)
Right now a document also indexes the object's properties in a field "objcontent". I do this for all
translations, thus duplicating the
field's
value in all translations. I can do the same for
attachments. The purpose
is, indeed, to be able to find document translations based on hits in
their
objects/attachments. If a language filter is used
and there is a hit in
an
object, only one document is returned. If there
are no language filters,
all translations will be returned.
It seems we have to do this for now
However, if we search for the object/property/attachment itself, it will
only be assigned to one language: the language of the original document.
This means that if we search for all languages, the object itself will be
found too (there is no language filter used). If we add a language filter
that is different from the object/property/attachment's original document
language, the object/property/attachment will not be found.
Maybe we can come up with some processing of the query in the search
application, that applies the language filter only for documents:
((-type:"OBJECT" OR -type:"OBJECT_PROPERTY" OR
-type:"ATTACHMENT") OR
lang:"<userSelectedLanguage>") -- writing it like this because the
default
operand is AND in the query filter clause that we
use in the Search
application.
The problem with this is that that, when a language filter is used, the
object/property/attachments that are now included in the results might
not
have the specified language and will pollute the
results.
I'm not sure I understand. We have an "objcontent" field for each
translation that has the full content of objects and properties, but
individual object fields we don't have in each translations ?
The more I see all the issues, the more I lean towards
a separate index per
language solution.
Again, these issues are not related to this, just to the fact that Document
translations are first-class Lucene documents.
The reason I do is that the main need is for a non
English user to have very relevant results in his own language. Therefore
we need to make sure that all content that the users have published has the
chance to be analyzed using the non English language analyzer. So indexing
all objects and attachments with the relevant language analyzer is the
solution. This is also why I proposed to index all content in this specific
index regardless of the language declared, which would only be used in the
UI to limit searches to the specific language.
In this view:
0/ There would be a language specific index per language with the objects
and attachments indexed only in the language of the index
1/ The user chooses the language in which he searches
2/ Automatically that sets the index to be used to be the "french" index
3/ Automatically that presets to limit the span of the search to declared
"french" documents
4/ The user can decide to go for non french documents at his own risks
knowing that the results might be weird because of wrong analysis (this is
what happens today with english analysis over french documents)
The benefit here is that you don't have a merging score over multiple index
issue, since you would never have to do a search across multiple indexes.
Searches are still simple to write. By default results are quite relevant
since you limit the search the french declared documents (this would be the
same as limiting your search to title_fr, content_fr) and still cover what
needs to be covered (objects and attachments).
Another benefit is that this falls back gracefully to monolingual as you
just have to have one index in the language declared for the monolingual
wiki.
The drawback is that the indexing is more costly and there is duplicated
content in the index. Howerver it is the Admin that say which languages he
wants available and he takes responsibility of the ressources this needs.
Could this solution work ?
Unfortunately I am still not convinced by this approach. Besides the
complexities of managing multiple cores (each with it's own schema and
config files), the user is exposed to a lot of unnecessary badly indexed
data that will make him stay away from the search feature, as it has
happened with Lucene so far.
I believe this discussion thread has come up with some nice solutions to
most of our problems and that the multiple-fields direction is one that can
give us relevant results, properly indexed content for all languages (even
when searching in all languages) and a good solution for objects,
attachments and properties (though, again, this is not related to this
specific choice).
I will try to make a summary of the ideas from this thread and put them
into a wiki page that documents the design/progress of the multi-lingual
related work.
Thank you very much for now and, of course, this does not mean that the
discussion is over in any way :)
-Eduard
Ludovic
not
*new*
enygma2002(a)gmail.com
approach.
full
that
clauses
query
Because
default
an
andhides
fields
stemming.
> > >> My experience, which was before dismax on curriki.org, has made
it
box.
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
to
that
match
> > >> single in one language.
> > >>
> > >> However, indicating which language it is matched into is probably
> > >> useful...
> > >>
> > >
> > > Already doing that.
> > >
> > >
> > >> Funnily, cross-language-retrieval is a mature research field but
> > >> retrieval for multilanguage user is not so!
> > >>
> > >> > On a related note, I have also watched an interesting
presentation
German
forced
filesystem.
but
good
> > >> idea.
> > >> Also, it is very common that users formulate queries without
> formulating
> > >> their language and thus you need to "or" the user's
queries through
will
problem
list
on-the-fly,
since
separate
previous
> > > experience on the topic, so I`m still doing mostly guesswork in
some
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
Thanks,
Eduard
zip
Ludovic
2012/11/26 Eduard Moraru <enygma2002(a)gmail.com>
> Hi devs,
>
> Any other input on this matter?
>
> To summarize a bit, if we go with the multiple fields for each
language,
> we
> > end up with an index like:
> >
> > English version:
> > id: xwiki:Main.SomeDocument_en
> > language: en
> > space: Main
> > title_en: XWiki document
> > doccontent_en: This is some content
> >
> > French version:
> > id: xwiki:Main.SomeDocument_fr
> > language: fr
> > space: Main
> > title_fr: XWiki document
> > doccontent_fr: This is some content
> >
> > The Solr configuration is generated by some XWiki UI that returns a > > that the admin has to unpack in his
(remote) Solr instance. This
could
be
> > automated for the embedded instance. This operation is to be
performed
> each
> > time an admin changes the indexed languages (rarely or even only
once).
> >
> > Querying such a schema is a bit tricky when you are interested in
more
than
fields
> > like "title" and "doccontent" and to put as values very
lightly (or one language, because you have to add all the
clauses (title_en,
title_fr,
etc.) specific to the languages you are
interested in.
Some extra fields might also be added like title_ws (for whitespace
tokenization only) that have various approaches to the indexing
operation,
> with the aim of improving the relevancy.
>
> One solution to simplify the query for API clients would be to use > at
> > all) analyzed content, as Paul suggested. This would allow
applications
> to
> > write simple (and backwards compatible maybe) queries that will still
> work,
> > but will not catch some of the nuances of specific languages. As far
as
> I`ve
seen until now, applications are not very interested in nuances,
but
> rather in filtering the results, a task for
which this solution might
be
> > well suited. Of course, nothing stops applications from using the and
> more expressive fields that are properly analized.
>
> Thus, the search application will be the major beneficiary of these
> analyzed fields (title_en, title_fr, etc.), while still allowing
> applications to get their job done (trough generic, but less/not
analized
> > fields like "title", "doccontent", etc.).
> >
> > WDYT?
> >
> > Thanks,
> > Eduard
> >
> >
> >
> >
> > On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru < >
>wrote:
>
> > Hi Paul,
> >
> > I was counting on your feedback :)
> >
> > On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <paul(a)hoplahup.net
> wrote:
> >
> >>
> >> Hello Eduard,
> >>
> >> it's nice of you to see you take this further.
> >>
> >> > This issue has already been previously [1] discussed during the
GSoC
> > >> > project, but I am not particularly happy with the chosen > > >> > When handling multiple
languages, there are generally[2][3] 3
> > different
> > >> > approaches:
> > >> >
> > >> > 1) Indexing the content in a single field (like title,
doccontent,
>
etc.)
> >> > - This has the advantage that queries are clear and fast
> >> > - The disadvantage is that you can not run very well tuned
analyzers
> on
> >> the
> >> > fields, having to resort to (at best) basic tokenization and
> >> lowercasing.
> >> >
> >> > 2) Indexing the content in multiple fields, one field for each
> language
> >> > (like title_en, title_fr, doccontent_en, doccontent_fr, etc.)
> >> > - This has the advantage that you can easily specify (as dynamic
> fields)
> >> > that *_en fields are of type text_en (and analyzed by an
> >> english-centered
> >> > chain of analyzers); *_fr of type text_fr (focused on french,
etc.),
> > >> thus
> > >> > making the results much better.
> > >>
> > >> I would add one more field here: title_ws and text_ws where the >
text
> >> is analyzed just as words (using the whitespace-tokenizer?).
> >> A match there would even be preferred to a match in the below
> text-fields.
> >>
> >> (maybe that would be called title and text?)
> >>
> >> > - The disadvantage is that querying such a schema is a pain. If
you
> > want
> > >> > all the results in all languages, you end up with a big and
> expensive
> > >> > query.
> > >>
> > >> Why is this an issue?
> > >> Dismax does it for you for free (thanks to the "form"
parameter >
>> gives weight to each of the fields).
> >> This is an issue only if you start to have more than 100 languages
or
> > >> so...
> > >> Lucene, the underlying engine of solr, handles thousands of in a
> >> query without an issue (this is how prefix-queries are handled...
they
> are
> >> expanded to a query for any of the term that matches the prefix, a
> setting
> >> deep somewhere, which is about 2000 avoids this to explode).
> >>
> >
> > Sure, Solr is great when you want to do simple queries like "XWiki
Open
> > > Source", however, since in XWiki we also expose the Solr/Lucene
> > APIs
> > > to the platform, there will be (as as it is currently with Lucene)
a
lot
> of
> > extensions wanting to do search using this API. These extensions
(like
> the
> > search suggest for example, rest search, etc) want to do something
like
> > > "title:'Open Source' AND type:document AND
doccontent:XWiki". > >
option 2) is so messy in it's fields, it would mean that the
extension
> > > would have to come up with a query like "title_en:'Open
Source' AND
> > > type:document AND doccontent_en:XWiki" (assuming that it is only
> limited
> > to
> > > the current -- english or whatever -- language; what happens if it
> wants
> > to
> > > do that no matter what language? It will have to specify each
> combination
> > > possible because we can't use generic field names).
> > >
> > > Solr's approach works for using it in your web application's
search
> > input,
> > > in a specific usecase, where you have precisely specified the > > > search fields and their boosts
inside your schema.xml. However, as
a
>
search
> > API, using option 2) you are making the life of anyone else wanting
to
> > use
> > > the Solr search API really hard. Also, your search application will
> work
> > > nicely when the user enters a simple query in the input field, but > > > advanced user will suffer the same
fate when trying to write an
> advanced
> > > query, thus not relying on the default query (computed by solr
based
on
all
> > schema.xml).
> >
> > Also, based on your note above regarding improvements like title_ws
and
> > > such, again, all of these are very well suited for the search
> application
> > > use case, together with the default query that you configure in
> > schema.xml,
> > > making the search results perform really well. However, what does >
these
> > fields mean to another extension wanting to do search? Will it have
to
> > handle all these implementation details
to query for title, content
and
> > such? I`m not sure how well this would
work in practice.
> >
> > Unrealistic idea(?): perhaps we should come up with an abstract
search
> > > language (Solr/Lucene clone) that parses the searched fields > the
> > > complexities of all the indexed fields, allowing to write simple
> queries
> > > like "title:XWiki", while this gets translated to
"title_en:XWiki
OR
> >
title_fr:XWiki OR title_de:XWiki..." :)
> >
> > Am I approaching this wrong by trying to have both a
tweakable/tweaked
> > search application AND a search API?
Are the two not compatible? Do
we
> have
> > to sacrifice search result performance (no language-specific stuff)
to
> be
> > > able to have a usable API?
> > >
> > >
> > >> > If you want just some language, you have to read the right > > >> > (ex title_en) instead of
just getting a clear field name
(title).
>>
>> You have to be careful, this is really only if you want to be
specific.
> >> In this case, it is likely that you also do not want so much so
concern,
a
>>
that any query that is a bit specific is likely to not desire
stemming.
>>
>
> Can you please elaborate on this? I`m not sure I understand the
problem.
> >
> >
> >>
> >> > -- Also, the schema.xml definition is a static one in this > >> > requiring you to know
beforehand which languages you want to
support
> > >> (for
> > >> > example when defining the default fields to search for). Adding
new
> >> > language requires you to start editing the xml files by hand.
> >>
> >> True but the available languages are almost all hand-coded.
> >> You could generate the schema.xml based on the available languages
if
> not
> >> hand-generated?
> >>
> >
> > Basically I would have to output a zip with schema.xml,
solrconfig.xml
> > and
> > > then all the resources specific to all the selected languages
> (stopwords,
> > > synonims, etc) for the languages that we can provide out of the > For
> > > other languages, the admin would have to get dirty with the xmls.
> > >
> > >
> > >>
> > >> There's one catch with this approach which is new to me but seems
to
> be
> > >> quite important to implement this approach: the idf should be
> modified,
> > the
> > >> Similarity class should be, so that the total number of documents
is
the
>> total number of documents having that
language.
>> See:
>>
>>
> > >> The solution sketched there
sounds easy but I have not tried it.
> > >>
> > >> > 3) Indexing the content in different Solr cores (indexes), one
for
> > each
> > >> > language. Each core requires it's on directory and
configuration
> > files.
> > >> > - The advantage is that queries are clean to write (like option
1)
and
> >> that
> >> > you have a nice separation
> >> > - The disadvantage is that it's difficult to get it right
> >> (administrative
> >> > issues) and then you also have the (considerable) problem of
having
to
to
> a
> > >> > remote Solr instance the configuration files when creating a new
> core
> > >> > programatically. However, if we are running an embedded Solr
> instance,
> > >> we
> > >> > could provide a way to generate the config files and write them
>> fix
>> > the relevancy score of a query result that has entries from
different
>> > cores; each core has it's own
relevancy computed and does not
consider
> >> the
> >> > others.
> >> > - To make it even worst, it seems that you can not [5] also push
the
> >> data
> >> > directory.
> >>
> >> Post-processing results is very very very dangerous as performance
is
> at
> > >> risk (e.g. if a core does not answer)... I would tend to avoid as
the
> much
> >> as possible.
> >>
> >
> > Not really related, but this reminds me about the post processing
that
> I
> > > do for checking view rights over the returned result, but that's
> another
> > > discussion that we will probably need to have :)
> > >
> > >
> > >>
> > >> > Currently I have implemented option 1) in our existing Solr
> > integration,
> > >> > which is also more or less compatible with our existing Lucene
> > queries,
> > >> but
> > >> > I would like to find a better solution that actually analyses
>
>> content.
> >> >
> >> > During GSoC, option 2) was preferred but the implementation did
not
>>
> consider practical reasons like the ones described above (query
>> complexity,
>> > user configuration, etc.)
>>
>> True, Savitha surfed the possibility of having different solr
documents
> >> per language.
> >> I still could not be sure that this was not showing the document > [3]
> > >> > about how Drupal handles its Solr integration and,
particularly, a
>
>> plugin
> >> > [4] that handles the multilingual aspect.
> >> > The idea seen there is that you have this UI that helps you
generate
> > >> > configuration files, depending you your needs. For instance, you
> > (admin)
> > >> > check that you need search for language English, French and
and
> >> the
> >> > ui/extension gives you a zip with the configuration you need to
use
> in
> > >> your
> > >> > (remote or embedded) solr instance. The configuration for each
> > language
> > >> > comes preset with the analyzers you should use for it and the
> > additional
> > >> > resources (stopwords.txt, synonims.txt, etc.).
> > >> > This approach helps with avoiding the need for admins to be
> to
> > >> edit
> > >> > xml files and could also still be useful for other cases, not
only
> > >> option
> > >> > 2).
> > >>
> > >> Generating sounds like an easy approach to me.
> > >>
> > >
> > > Yes, however I don`t like the fact that we can not do everything
from
> the
> > > webapp and the admin needs to access the filesystem to install the
> given
> > > configuration on the embedded/remote solr directory. Lucene does
not
have
> > this problem now. It just works with XWiki and everything is done
from
> > XWiki UI. I feel that losing this
commodity will not be very well
> received
> > by users that now have some new install steps to get XWiki running.
> >
> > Well, of course, for the embedded solr version, we could handle it
like
> > we
> > > do now and push the files directly from the webapp to the > > Since
> > > embedded will be default, it should be OK and avoid the extra
install
>
step.
> > Users with a remote solr machine should have the option to get the
zip
> > > instead.
> > >
> > > Not sure if we can apply the new configuration without a restart, >
I`ll
> > have to look more into it. I know the multi-core architecture
supports
> > something like this but will have to
see the details.
> >
> >
> >>
> >> > All these problems basically come from the fact that there is no
way
> to
> >> > specify in the schema.xml that, based on the value of a field
(like
the
>> > field "lang" that stores the document language), you want to run
this
> or
> >> > that group of analyzers.
> >>
> >> Well, this is possible with ThreadLocal but is not necessarily a > > >> multiple languages (e.g. given
by the browser).
> > >>
> > >> > Perhaps a solution would be a custom kind of
"AggregatorAnalyzer"
> that
> > >> > would call other analyzers at runtime, based on the value of the
> lang
> > >> > field. However, this solution could only be applied at index
time,
> > when
> > >> you
> > >> > have the lang information (in the solrDocument to be indexed),
but
> > when
> > >> you
> > >> > perform the query, you can not analyze the query text since you
do
not
> >> know
> >> > the language of the field you're querying (it was determined at
> runtime
> >> -
> >> > at index time) and thus do not know what operations to apply to
the
> > >> query
> > >> > (to reduce it to the same form as the indexed values).
> > >>
> > >> How would that look at query time?
> > >>
> > >
> > > That's what I was saying, that at query time, the searched term not
> > get analyzed by the right chain. When you search for a single
language,
> you
> > could add that language as a query filter and then you could apply
the
> > > right chain, but when searching in 2 or more (or no, meaning all)
> > languages
> > > you are stuck.
> > >
> > >>
> > >> > I have also read another interesting analysis [6] on this that
> >> > elaborates on the complexities and limitations of each options.
> (Ignore
> >> the
> >> > Rosette stuff mentioned there)
> >> >
> >> > I have been thinking about this for some time now, but the
solution
is
> >> > probably somewhere in between, finding an option that is
acceptable
> > >> while
> > >> > not restrictive. I will probably also send a mail on the Solr
> to
> > >> get
> > >> > some more input from there, but I get the feeling that whatever
> > >> solution we
> > >> > choose, it will most likely require the users to at least copy
(or
>
even
> >> > edit) some files into some directories (configurations and/or
jars),
> > >> since
> > >> > it does not seem to be easy/possible to do everything >
>> > programatically.
> >>
> >> The only hard step is when changing the supported languages, I
think.
> > >> In this case, when automatically generating the index, you need to
> warn
> > >> the user.
> > >> The admin UI should have a checkbox "use generated schema"
or a
> textarea
> > >> for the schema.
> > >>
> > >
> > > Please see above regarding configuration generation. Basically, we
> > are going to support both embedded and remote solr instances, we
could
> > support things like editing the schema
from XWiki only for the
embedded
> > > instance, but not for the remote one. We might end up having > UIs
> > > for each case, since we might want to exploit the flexibility of
the
> >
embedded one as much as possible.
> >
> >
> >>
> >> Those that want particular fields and tunings need to write their
own
> >> schema.
> >>
> >> The same UI could also include whether to include a phonetic track
or
not
>> (then require reindexing).
>
>
>> hope it helps.
>>
>
> Yes, very helpful so far. I`m counting on your expertise with
Lucene/Solr
> > on the details. My current approach is a practical one without areas.
>
> Thanks,
> Eduard
>
>
>> paul
>> _______________________________________________
>> devs mailing list
>> devs(a)xwiki.org
>> http://lists.xwiki.org/mailman/listinfo/devs
>>
>
>
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
3:54 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
On Thu, Nov 29, 2012 at 3:47 PM, Eduard Moraru <enygma2002(a)gmail.com> wrote:
Sure. I was just repeating what is important.
the
It's the same as in the UI allows you to specify which language to search
in, but it also allows you to specify that you want to search in "All
languages", that is what I meant.
Not specifically as an API
you
ALL
> > content regardless of it's language using the language analyzer of
that
would
multi-core
This is not a good solution. If I understand correctly we could not end up
are
purpose
filters,
will
"objcontent" stores the properties (format: "propName:propValue") of
each
object inside the original document (multi-valued field).
Besides XWikiDocuments, we also index Objects, Properties and Attachments
as Lucene/Solr first-class documents. Each of these entries has the wiki,
space, name and lang fields set from the document to which they belong to.
This is what I meant above with "if we search for the
object/property/attachment itself"
So to reiterate, the idea was that if you want to search for an indexed
Object (type:"OBJECT"), you will *have* to avoid setting a language filter,
or you might not find the object you are looking for, since it is indexed
under the language of original document.
---
While writing this down, I just thought of an elegant idea to fix this. The
lang field could be set to multi-valued. This means that, when we index
objects, properties and attachments, we could set in the lang field all the
values corresponding to all the existing translations of the owning
document. Example:
An Object:
id: xwiki:Main.Page^XWiki.Class[0] <-- (object reference)
class: XWiki.Class
wiki: xwiki
space: Main
name: Page
lang: en <-- proposal to make it multi-valued would look like this,
stored like a list of values.
fr
de
type: OBJECT
Note that this solution would also affect document languages as well (since
they share the same schema), but we will just put one value and it will not
affect queries. Queries will still be written like: "lang:en"
If we apply this solution, even if a language filter exists in the user's
query, it will still hit the object because the lang field of the object
contains the value.
enygma2002(a)gmail.com
> > > >wrote:
> > >
> > > > Hi Paul,
> > > >
> > > > I was counting on your feedback :)
> > > >
> > > > On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <
paul(a)hoplahup.net
dynamic
full
> > > text
> > > >> is analyzed just as words (using the whitespace-tokenizer?).
> > > >> A match there would even be preferred to a match in the below
> > > text-fields.
> > > >>
> > > >> (maybe that would be called title and text?)
> > > >>
> > > >> > - The disadvantage is that querying such a schema is a pain.
If
languages
handled...
Lucene)
AND
as
have
Do
it
> so
> > > >> that any query that is a bit specific is likely to not desire
> > stemming.
> > > >>
> > > >
> > > > Can you please elaborate on this? I`m not sure I understand the
> > problem.
> > > >
> > > >
> > > >>
> > > >> > -- Also, the schema.xml definition is a static one in this
> concern,
> > > >> > requiring you to know beforehand which languages you want
to
> support
> > > >> (for
> > > >> > example when defining the default fields to search for).
Adding
seems
configuration
did
you
to
the
running.
the
no
through
the
you
apply
options.
list
> > to
> > > >> get
> > > >> > some more input from there, but I get the feeling that
whatever
on-the-fly,
> > > >> > programatically.
> > > >>
> > > >> The only hard step is when changing the supported languages, I
> think.
> > > >> In this case, when automatically generating the index, you need
to
separate
their
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Thomas Mortagne
Hi,
On Wed, Nov 28, 2012 at 5:35 PM, Ludovic Dubost <ludovic(a)xwiki.com> wrote:
the
When importing a XAR for example.
2012/11/28 Eduard Moraru
<enygma2002(a)gmail.com>
understand
Hi Ludovic,
Thanks for the reply. Please read below...
On Tue, Nov 27, 2012 at 5:44 PM, Ludovic Dubost <ludovic(a)xwiki.com>
wrote:
> Hi Edy,
>
> I'm not a huge fan of the title_fr title_en title_morelanguages
approach
> as
> > indeed it seems to be quite complex at the query level. I was more
> leaning
> > towards multiple indexes if we can query them globally but I > > this is complex too.
> >
> > Now let's see the use cases that are hugely important:
> >
> > 1/ Make sure that if you decide your wiki is monolingual:
> > - the indexing uses the specific language analyzer
> > - make sure the query uses the specific language analyzer
> > - make sure the search looks in all content even if the language
> setting
> > of the document is wrongly set (consider all documents being of the
> > specific language)
> >
>
> You mean that, if the wiki is monolingual, we should ignore the
language
filter
and hardcode it to "All languages", right?
However, what would be the advantage of this? Why would be want to
pollute
the results with irrelevant documents (caused by
a probable recent
configuration change that went from multi-lingual to mono-lignual)?
Wasn't
that the whole reason why the admin switched to
mono-lingual?
If the user is monolingual, we can safely ignore the language setting of
each document and only the "main" document will be shown anyway to the
user.
So we should make sure we search on ALL documents that are available to user.
The user might have to "reindex" to make sure this is properly taken into
account by the search engine.
What is important here, is that even if the wiki is set to "fr", then
even
if documents have "en" for the main
language they will still show up in
the
search. The opposite would be bad.
Why/How can you end up with documents of other languages than the current
default one if the wiki is mono-lingual?
2/ Allow is a wiki is multi-lingual:
- search in the language you decide (maybe the UI should display a
language choice for the query)
We already support this, by using the "Filtered Search" option and
selecting the language.
- search in content that is analyzed in the proper language when the
> content is declared in this language
> - allow to specify if you want to restrict your search to documents
> declared in the language of your query, versus search more widely in
all
> > documents accross languages. If you search in only the language of > > query only one document can show up but
it should point to the right
> > translation that matches, if you search in multiple languages then
you
can
"French" but look in the "French" + "English" dataset. For
me if both
documents with the same page name match, both should come out separately.
show individual translations.
I think this one is the same as the first bullet above.
No I think it's different. The user can decide to make his search in - allow technical users to search for all
documents across all
languages
(where the language analysis does not really
matter)
Do you mean as an API?
What exactly do you mean by "language analysis does not really matter"?
Any
Example?
I mean here that as a technical user your objective is to make sure your
search spans ALL content in the wiki. In this case you don't care about
stemming and such.
The standard UI could take this into account by choosing "any language
search" and with data set "all languages". We just need to make sure that
this won't exclude any content of the search
(Here is an example of a case where such exclusion might occur. Suppose have done only "French" and
"English" indexing and there is "German"
content also in your wiki but since you have not asked for German search
in
your wiki you don't have title_de and
content_de fields or don't have a
german specific solr index (in the other method), then your german
content
would not be indexed at all ?)
I understand now what you mean. Well, to avoid this, we can use only one
notion and that is the one of "supported languages". All the supported
languages should be indexed in order for the Search feature to work as
expected (and not have the case where you have a document that is not
indexed because of its language). I am not sure if there are many examples
where admins would choose to index just some of the supported languages and
not index all of them.
WDYT?
>
> From an admin point of view it makes good sense for the admin to be
able
to
engine.
specify in a multilingual wiki which language
analysis should be
activated,
> and then have this transmitted to SOLR to properly configure the > Reindexing is ok when changing the
configuration.
>
> I believe in the end wether you use multiple fields with _fr _en or
> multiple SOLR cores, as long as you can query accross SOLR cores is a
bit
the same.
If you cannot run a query merging multiple indexes then the
first
> solution is kind of absolutely necessary as it would be the only one
> allowing to search across all languages.
>
> Maybe a solution would be to create one index per language and index >
index. This would allow to have better results even though the users
have
> badly tagged the language of a document, and
it's only the job of the
UI
to
all
limit the search to only the language of the
query, or all documents.
So you could have a configuration in the admin
that says:
1/ Create an English Index
2/ Create an additional French index
The UI would allow to search in English and French, + would add a
language
restriction for the documents.
Applying the language specific analyzers (for Chinese, for example) to the documents will just create a mess for all the
documents that do not
match the analyzer's language. I`m not sure the results for the
badly-indexed languages will make any sense to users.
I undestand the issue here, but in most cases the user will say "french
search" on "french content", he will only expand to non french content if
he was not satisfied by his search. What is just allowed here is to also
search in "french" inside all content. That would cover content that have the bad language setting as well as any
other content. The results
might be a bit noisy but I don't think it's a big issue.
What I think you are actually asking is that all languages are queried, but
that results in the requested language get boosted/elevated and thus are
displayed before all other language results.
Perhaps this can be achieved by using the lang field as a query field and
not as a query filter. This means that it will influence the score. Couple
that with a higher boost for the lang field and results in the requested
language should be scored better than all the rest (because there will
always be a hit for the lang field resulting in a better score). There
might still be some results from other languages that could have a better
score than results from the requested language (because they just have more
hits on diferent fields), but it should work in most cases.
However, I`m not sure we want this. Users might results from other
languages as a bug. Problems might also be harder to debug.
Instead, we could do something like Google and if we see a small number of
results, we could suggest that the user searches in all languages.
In any case, I`m not sure this is such a big issue right now, since it's
basically an optimisation that we could choose to do, or let the user do it
himself, since he has the tools (UI).
Also, this is very similar to the multi-core
approach (one core per
language), just that you also add documents that are indexed with the
wrong
> analyzers. We have the same problem regarding merging relevance scores
> across indexes (cores) that is a big turn-off for the original > approach.
>
> This is a more serious issue. If it's hard to merge the results
spanning
over multiple cores this could be a showstopper.
However the solution of
having only one Lucene document for all languages is not so cool either
as
it would make it difficult to know which ones of
the languages has
matched
and present them separately with separate
scores.
I agree. The one-document-for-all-languages solution was proposed by Paul,
but I don`t think it's the way to go. We are currently only considering the
one-document-per-language direction.
It's really the core issue to decide on. What
are the benefits and
drawbacks of the different solutions. For each solution is there
something
in the UI that you cannot do ?
So far I've heard:
1/ Presenting different scores for documents in different languages with
the same doc name if the title_fr,content_fr method is used
Since we are not considering the one-document-for-all-languages solution,
this is not an issue.
2/ Merging scores accross indexes in multicore approach
Other ? Can we list them in a wiki page ?
> >
> > In the future if we are able to "detect" the language of the
documents
we
> > could add a lucene field with the "detected" language instead of the
> > "provided" language of the documents, therefore increasing the
quality
of
> > searches only on documents of a specific language.
> >
>
> In the previous discussions (on the GSoC thread) we agreed that
language
in
content
XWiki is known before-hard, so no recognition is
required, at least not
at
document level.
Let's forget this
>
> This later solution would be the only one that would really work on
file
> > attachements as we have no information about the specific language of
> file
> > attachements (or even XWiki objects) which are attached to the main
> > document and not to the translated document.
> >
>
> Yes, this is a problem right now. AFAIU, the plan [1] is to support
> translated objects and maybe attachments as well. Until then, we could
> either:
> 1) Use the original document's language to index the attachment's not searching in a french attachment because the
original document is in
marked "en".
I'm for Paul's solution to index objects and attachments in each
translation (if we have separate entities for translated documents).
Yep, we already do this for objects, just need to do it for attachments as
well.
I
understand that in the title_fr,content_fr approach this problem does not
happen.
This problem is not related to how we handle multiple translations
(separate indexes or not), as long as each translation is a separate
entity. Basically this last part is the problem. If the French translation
is a separate entity from the original document (i.e. that is in English),
any object/attachment *index field* of the English original version will
need to be duplicated into the French translation as well, or the French
translation risks not to get a hit on the object/attachment.
2) Use a language detection library to try to
detect the attachment
content's language and index it accordingly.
Not sure we can for now
> The above could also be applied for objects and their properties.
> ----------
> [1] http://jira.xwiki.org/browse/XWIKI-69
>
>
> >
> > This later issues shows that a search on "only french content"
should
> still
> > include the attachements because we have no idea if the attachments > > "french" or
"english".
> >
>
> (The paragraphs below discuss on what currently exists and what could
be
done,
ignoring the possible language detection mentioned above)
Right now a document also indexes the object's properties in a field
called
"objcontent". I do this for all
translations, thus duplicating the
field's
> value in all translations. I can do the same for attachments. The is,
indeed, to be able to find document translations based on hits in
their
objects/attachments. If a language filter is used
and there is a hit in
an
> object, only one document is returned. If there are no language all
translations will be returned.
It seems we have to do this for now
>
> However, if we search for the object/property/attachment itself, it > only be assigned to one language: the
language of the original
document.
> This means that if we search for all
languages, the object itself will
be
> found too (there is no language filter
used). If we add a language
filter
> that is different from the
object/property/attachment's original
document
language,
the object/property/attachment will not be found.
Maybe we can come up with some processing of the query in the search
application, that applies the language filter only for documents:
((-type:"OBJECT" OR -type:"OBJECT_PROPERTY" OR
-type:"ATTACHMENT") OR
lang:"<userSelectedLanguage>") -- writing it like this because the
default
operand is AND in the query filter clause that we
use in the Search
application.
The problem with this is that that, when a language filter is used, the
object/property/attachments that are now included in the results might
not
have the specified language and will pollute the
results.
I'm not sure I understand. We have an "objcontent" field for each
translation that has the full content of objects and properties, but
individual object fields we don't have in each translations ?
The more I see all the issues, the more I lean
towards a separate index
per
language solution.
Again, these issues are not related to this, just to the fact that Document
translations are first-class Lucene documents.
The reason I do is that the main need is for a
non
English user to have very relevant results in his own language. Therefore
we need to make sure that all content that the users have published has
the
chance to be analyzed using the non English
language analyzer. So
indexing
all objects and attachments with the relevant
language analyzer is the
solution. This is also why I proposed to index all content in this
specific
index regardless of the language declared, which
would only be used in
the
UI to limit searches to the specific language.
In this view:
0/ There would be a language specific index per language with the objects
and attachments indexed only in the language of the index
1/ The user chooses the language in which he searches
2/ Automatically that sets the index to be used to be the "french" index
3/ Automatically that presets to limit the span of the search to declared
"french" documents
4/ The user can decide to go for non french documents at his own risks
knowing that the results might be weird because of wrong analysis (this
is
what happens today with english analysis over
french documents)
The benefit here is that you don't have a merging score over multiple
index
issue, since you would never have to do a search
across multiple indexes.
Searches are still simple to write. By default results are quite relevant
since you limit the search the french declared documents (this would be
the
same as limiting your search to title_fr,
content_fr) and still cover
what
needs to be covered (objects and attachments).
Another benefit is that this falls back gracefully to monolingual as you
just have to have one index in the language declared for the monolingual
wiki.
The drawback is that the indexing is more costly and there is duplicated
content in the index. Howerver it is the Admin that say which languages
he
wants available and he takes responsibility of
the ressources this needs.
Could this solution work ?
Unfortunately I am still not convinced by this approach. Besides the
complexities of managing multiple cores (each with it's own schema and
config files), the user is exposed to a lot of unnecessary badly indexed
data that will make him stay away from the search feature, as it has
happened with Lucene so far.
I believe this discussion thread has come up with some nice solutions to
most of our problems and that the multiple-fields direction is one that can
give us relevant results, properly indexed content for all languages (even
when searching in all languages) and a good solution for objects,
attachments and properties (though, again, this is not related to this
specific choice).
I will try to make a summary of the ideas from this thread and put them
into a wiki page that documents the design/progress of the multi-lingual
related work.
Thank you very much for now and, of course, this does not mean that the
discussion is over in any way :)
-Eduard
Ludovic
not
still
Thanks,
Eduard
zip
Ludovic
2012/11/26 Eduard Moraru <enygma2002(a)gmail.com>
> Hi devs,
>
> Any other input on this matter?
>
> To summarize a bit, if we go with the multiple fields for each
language,
> we
> > end up with an index like:
> >
> > English version:
> > id: xwiki:Main.SomeDocument_en
> > language: en
> > space: Main
> > title_en: XWiki document
> > doccontent_en: This is some content
> >
> > French version:
> > id: xwiki:Main.SomeDocument_fr
> > language: fr
> > space: Main
> > title_fr: XWiki document
> > doccontent_fr: This is some content
> >
> > The Solr configuration is generated by some XWiki UI that returns a > > that the admin has to unpack in his
(remote) Solr instance. This
could
be
> > automated for the embedded instance. This operation is to be
performed
> each
> > time an admin changes the indexed languages (rarely or even only
once).
> >
> > Querying such a schema is a bit tricky when you are interested in
more
than
> one language, because you have to add all the clauses (title_en,
title_fr,
> etc.) specific to the languages you are interested in.
>
> Some extra fields might also be added like title_ws (for whitespace
> tokenization only) that have various approaches to the indexing
operation,
> with the aim of improving the relevancy.
>
> One solution to simplify the query for API clients would be to use
fields
> > like "title" and "doccontent" and to put as values very
lightly (or > at
> > all) analyzed content, as Paul suggested. This would allow
applications
> > to
> > > write simple (and backwards compatible maybe) queries that will > > work,
> > > but will not catch some of the nuances of specific languages. As
far
as
> > > I`ve seen until now, applications are not very interested in
nuances,
> but
> > > rather in filtering the results, a task for which this solution
might
be
> > well suited. Of course, nothing stops applications from using the
*new*
and
> more expressive fields that are properly analized.
>
> Thus, the search application will be the major beneficiary of these
> analyzed fields (title_en, title_fr, etc.), while still allowing
> applications to get their job done (trough generic, but less/not
analized
> > fields like "title", "doccontent", etc.).
> >
> > WDYT?
> >
> > Thanks,
> > Eduard
> >
> >
> >
> >
> > On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru < >
> > > wrote:
> > > >
> > > >>
> > > >> Hello Eduard,
> > > >>
> > > >> it's nice of you to see you take this further.
> > > >>
> > > >> > This issue has already been previously [1] discussed during
the
GSoC
> > >> > project, but I am not particularly happy with the chosen
approach.
> > >> > When handling multiple
languages, there are generally[2][3] 3
> > different
> > >> > approaches:
> > >> >
> > >> > 1) Indexing the content in a single field (like title,
doccontent,
> > > etc.)
> > > >> > - This has the advantage that queries are clear and fast
> > > >> > - The disadvantage is that you can not run very well tuned
> analyzers
> > > on
> > > >> the
> > > >> > fields, having to resort to (at best) basic tokenization
and
> > > >> lowercasing.
> > > >> >
> > > >> > 2) Indexing the content in multiple fields, one field for
each
> > > language
> > > >> > (like title_en, title_fr, doccontent_en, doccontent_fr,
etc.)
> > > >> > - This has the advantage that you can easily specify (as
> fields)
> >> > that *_en fields are of type text_en (and analyzed by an
> >> english-centered
> >> > chain of analyzers); *_fr of type text_fr (focused on french,
etc.),
> > >> thus
> > >> > making the results much better.
> > >>
> > >> I would add one more field here: title_ws and text_ws where the you
> > want
> > >> > all the results in all languages, you end up with a big and
> expensive
> > >> > query.
> > >>
> > >> Why is this an issue?
> > >> Dismax does it for you for free (thanks to the "form"
parameter
that
> > > >> gives weight to each of the fields).
> > > >> This is an issue only if you start to have more than 100 or
> > >> so...
> > >> Lucene, the underlying engine of solr, handles thousands of
clauses
> > in a
> > > >> query without an issue (this is how prefix-queries are > they
> > > are
> > > >> expanded to a query for any of the term that matches the
prefix, a
> > > setting
> > > >> deep somewhere, which is about 2000 avoids this to explode).
> > > >>
> > > >
> > > > Sure, Solr is great when you want to do simple queries like
"XWiki
Open
> > > Source", however, since in XWiki we also expose the Solr/Lucene
query
> > > APIs
> > > > to the platform, there will be (as as it is currently with a
> > lot
> > > of
> > > > extensions wanting to do search using this API. These extensions
> (like
> > > the
> > > > search suggest for example, rest search, etc) want to do
something
like
> > > "title:'Open Source' AND type:document AND
doccontent:XWiki".
Because
> > > > option 2) is so messy in it's fields, it would mean that the
> extension
> > > > would have to come up with a query like "title_en:'Open
Source' > > > > type:document AND
doccontent_en:XWiki" (assuming that it is only
> > limited
> > > to
> > > > the current -- english or whatever -- language; what happens if
it
> > wants
> > > to
> > > > do that no matter what language? It will have to specify each
> > combination
> > > > possible because we can't use generic field names).
> > > >
> > > > Solr's approach works for using it in your web application's
search
> >
input,
> > > in a specific usecase, where you have precisely specified the
default
> > > > search fields and their boosts inside your schema.xml. However,
a
> > > search
> > > > API, using option 2) you are making the life of anyone else
wanting
> to
> > > use
> > > > the Solr search API really hard. Also, your search application
will
> > work
> > > > nicely when the user enters a simple query in the input field,
but
an
title_ws
> > > advanced user will suffer the same
fate when trying to write an
> advanced
> > > query, thus not relying on the default query (computed by solr
based
> on
> > > > schema.xml).
> > > >
> > > > Also, based on your note above regarding improvements like and
> > > such, again, all of these are very well suited for the search
> application
> > > use case, together with the default query that you configure in
> > schema.xml,
> > > making the search results perform really well. However, what does
all
> > > these
> > > > fields mean to another extension wanting to do search? Will it > to
> > > > handle all these implementation details to query for title,
content
and
andhides
> > such? I`m not sure how well this would
work in practice.
> >
> > Unrealistic idea(?): perhaps we should come up with an abstract
search
> > > language (Solr/Lucene clone) that parses the searched fields > the
> > > complexities of all the indexed fields, allowing to write simple
> queries
> > > like "title:XWiki", while this gets translated to
"title_en:XWiki
OR
> > > > title_fr:XWiki OR title_de:XWiki..." :)
> > > >
> > > > Am I approaching this wrong by trying to have both a
> tweakable/tweaked
> > > > search application AND a search API? Are the two not compatible?
> we
> > > have
> > > > to sacrifice search result performance (no language-specific
stuff)
to
> be
> > > able to have a usable API?
> > >
> > >
> > >> > If you want just some language, you have to read the right
fields
> > >> > (ex title_en) instead of
just getting a clear field name
(title).
>
>>
> >> You have to be careful, this is really only if you want to be
specific.
> >> In this case, it is likely that you also do not want so much
stemming.
> > >> My experience, which was before dismax on curriki.org, has made a
> > new
> > > >> > language requires you to start editing the xml files by
hand.
> > > >>
> > > >> True but the available languages are almost all hand-coded.
> > > >> You could generate the schema.xml based on the available
languages
if
box.
> > For
> > > > other languages, the admin would have to get dirty with the xmls.
> > > >
> > > >
> > > >>
> > > >> There's one catch with this approach which is new to me but
> not
> >> hand-generated?
> >>
> >
> > Basically I would have to output a zip with schema.xml,
solrconfig.xml
> > and
> > > then all the resources specific to all the selected languages
> (stopwords,
> > > synonims, etc) for the languages that we can provide out of the to
> > be
> > > >> quite important to implement this approach: the idf should be
> > modified,
> > > the
> > > >> Similarity class should be, so that the total number of
documents
is
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%3Cza…
the
> >> total number of documents having that language.
> >> See:
> >>
> >>
>
> >
>> The solution sketched there sounds easy but I have not tried it.
> > >>
> > >> > 3) Indexing the content in different Solr cores (indexes), one
for
> > > each
> > > >> > language. Each core requires it's on directory and
> > > files.
> > > >> > - The advantage is that queries are clean to write (like
option
1)
> > and
> > > >> that
> > > >> > you have a nice separation
> > > >> > - The disadvantage is that it's difficult to get it
right
> > > >> (administrative
> > > >> > issues) and then you also have the (considerable) problem
of
> having
> > to
> > > >> fix
> > > >> > the relevancy score of a query result that has entries from
> > different
> > > >> > cores; each core has it's own relevancy computed and
does not
> > consider
> > > >> the
> > > >> > others.
> > > >> > - To make it even worst, it seems that you can not [5] also
push
> to
> > a
> > > >> > remote Solr instance the configuration files when creating a
new
> > core
> > > >> > programatically. However, if we are running an embedded
Solr
> > instance,
> > > >> we
> > > >> > could provide a way to generate the config files and write
them
to
> > the
> > > >> data
> > > >> > directory.
> > > >>
> > > >> Post-processing results is very very very dangerous as
performance
is
> at
> > >> risk (e.g. if a core does not answer)... I would tend to avoid
that
as
the
> > > >> content.
> > > >> >
> > > >> > During GSoC, option 2) was preferred but the implementation
> much
> >> as possible.
> >>
> >
> > Not really related, but this reminds me about the post processing
that
> I
> > > do for checking view rights over the returned result, but that's
> another
> > > discussion that we will probably need to have :)
> > >
> > >
> > >>
> > >> > Currently I have implemented option 1) in our existing Solr
> > integration,
> > >> > which is also more or less compatible with our existing Lucene
> > queries,
> > >> but
> > >> > I would like to find a better solution that actually analyses
> not
> > > >> > consider practical reasons like the ones described above
(query
> > > >> complexity,
> > > >> > user configuration, etc.)
> > > >>
> > > >> True, Savitha surfed the possibility of having different solr
> > documents
> > > >> per language.
> > > >> I still could not be sure that this was not showing the document
> match
> > > >> single in one language.
> > > >>
> > > >> However, indicating which language it is matched into is
probably
> >
>> useful...
> > >>
> > >
> > > Already doing that.
> > >
> > >
> > >> Funnily, cross-language-retrieval is a mature research field but
> > >> retrieval for multilanguage user is not so!
> > >>
> > >> > On a related note, I have also watched an interesting
presentation
> [3]
> > >> > about how Drupal handles its Solr integration and,
particularly, a
> > > >> plugin
> > > >> > [4] that handles the multilingual aspect.
> > > >> > The idea seen there is that you have this UI that helps you
> generate
> > > >> > configuration files, depending you your needs. For instance,
> >
(admin)
> > >> > check that you need search for language English, French and
German
> > and
> > > >> the
> > > >> > ui/extension gives you a zip with the configuration you need
use
> in
> > >> your
> > >> > (remote or embedded) solr instance. The configuration for each
> > language
> > >> > comes preset with the analyzers you should use for it and the
> > additional
> > >> > resources (stopwords.txt, synonims.txt, etc.).
> > >> > This approach helps with avoiding the need for admins to be
forced
> to
> > >> edit
> > >> > xml files and could also still be useful for other cases, not
only
> > >> option
> > >> > 2).
> > >>
> > >> Generating sounds like an easy approach to me.
> > >>
> > >
> > > Yes, however I don`t like the fact that we can not do everything
from
> > the
> > > > webapp and the admin needs to access the filesystem to install >
given
> > > configuration on the embedded/remote solr directory. Lucene does
not
> > have
> > > > this problem now. It just works with XWiki and everything is done
> from
> > > > XWiki UI. I feel that losing this commodity will not be very well
> > > received
> > > > by users that now have some new install steps to get XWiki > > > >
> > > > Well, of course, for the embedded solr version, we could handle
it
like
> > we
> > > do now and push the files directly from the webapp to the
filesystem.
> > Since
> > > embedded will be default, it should be OK and avoid the extra
install
> > > step.
> > > > Users with a remote solr machine should have the option to get zip
> > > instead.
> > >
> > > Not sure if we can apply the new configuration without a restart,
but
> > > I`ll
> > > > have to look more into it. I know the multi-core architecture
> supports
> > > > something like this but will have to see the details.
> > > >
> > > >
> > > >>
> > > >> > All these problems basically come from the fact that there
is > way
> > > to
> > > >> > specify in the schema.xml that, based on the value of a
field
> (like
> > > the
> > > >> > field "lang" that stores the document language),
you want to
run
this
> or
> >> > that group of analyzers.
> >>
> >> Well, this is possible with ThreadLocal but is not necessarily a
good
> > >> idea.
> > >> Also, it is very common that users formulate queries without
> formulating
> > >> their language and thus you need to "or" the user's
queries > > >> multiple languages (e.g. given
by the browser).
> > >>
> > >> > Perhaps a solution would be a custom kind of
"AggregatorAnalyzer"
> > that
> > > >> > would call other analyzers at runtime, based on the value of
>
lang
> > >> > field. However, this solution could only be applied at index
time,
> > when
> > >> you
> > >> > have the lang information (in the solrDocument to be indexed),
but
> > > when
> > > >> you
> > > >> > perform the query, you can not analyze the query text since
do
> > not
> > > >> know
> > > >> > the language of the field you're querying (it was
determined
at
> > > runtime
> > > >> -
> > > >> > at index time) and thus do not know what operations to apply
to
the
> > >> query
> > >> > (to reduce it to the same form as the indexed values).
> > >>
> > >> How would that look at query time?
> > >>
> > >
> > > That's what I was saying, that at query time, the searched term
will
> > not
> > > > get analyzed by the right chain. When you search for a single
> language,
> > > you
> > > > could add that language as a query filter and then you could the
> > > right chain, but when searching in 2 or more (or no, meaning all)
> > languages
> > > you are stuck.
> > >
> > >>
> > >> > I have also read another interesting analysis [6] on this
problem
> > that
> > > >> > elaborates on the complexities and limitations of each
> (Ignore
> >> the
> >> > Rosette stuff mentioned there)
> >> >
> >> > I have been thinking about this for some time now, but the
solution
is
> >> > probably somewhere in between, finding an option that is
acceptable
> > >> while
> > >> > not restrictive. I will probably also send a mail on the Solr
> >
>> solution we
> > >> > choose, it will most likely require the users to at least copy
(or
>
even
> >> > edit) some files into some directories (configurations and/or
jars),
> > >> since
> > >> > it does not seem to be easy/possible to do everything >
warn
> > >> the user.
> > >> The admin UI should have a checkbox "use generated schema"
or a
> textarea
> > >> for the schema.
> > >>
> > >
> > > Please see above regarding configuration generation. Basically,
since
we
> > are going to support both embedded and remote solr instances, we
could
> > support things like editing the schema
from XWiki only for the
embedded
> > > instance, but not for the remote one. We might end up having > UIs
> > > for each case, since we might want to exploit the flexibility of
the
> > > > embedded one as much as possible.
> > > >
> > > >
> > > >>
> > > >> Those that want particular fields and tunings need to write
> own
> > > >> schema.
> > > >>
> > > >> The same UI could also include whether to include a phonetic
track
or
some
> not
> >> (then require reindexing).
> >
> >
> >> hope it helps.
> >>
> >
> > Yes, very helpful so far. I`m counting on your expertise with
Lucene/Solr
> > on the details. My current approach is a practical one without
previous
> > > experience on the topic, so I`m still doing mostly guesswork in >
areas.
> >
> > Thanks,
> > Eduard
> >
> >
> >> paul
> >> _______________________________________________
> >> devs mailing list
> >> devs(a)xwiki.org
> >> http://lists.xwiki.org/mailman/listinfo/devs
> >>
> >
> >
> _______________________________________________
> devs mailing list
> devs(a)xwiki.org
> http://lists.xwiki.org/mailman/listinfo/devs
>
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
4:45 p.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
Why/How can you end up with documents of other
languages than the current
default one if the wiki is mono-lingual?
by changing mind.
>> No I think it's different. The user can
decide to make his search in
> "French" but look in the "French" + "English" dataset.
For me if both
> documents with the same page name match, both should come out separately.
This can seriously be a problem considering that many browsers come, by default with many
supported languages.
paul
28 Nov
28 Nov
8:57 a.m.
New subject: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search
To summarize a bit, if we go with the multiple fields
for each language, we
end up with an index like:
English version:
id: xwiki:Main.SomeDocument_en
language: en
space: Main
title_en: XWiki document
doccontent_en: This is some content
French version:
id: xwiki:Main.SomeDocument_fr
language: fr
space: Main
title_fr: XWiki document
doccontent_fr: This is some content
Careful Eduard,
that means that searching for common words (e.g "direction") would show you the
same XWiki document as two different search results. I think you would want to combine the
two in one Solr document. This is the old problem of Savitha which, I think, she has not
solved.
I've always believed that the fact that these two documents exist implies that they
both have the same meaning and are translations of each other.
I like the idea to carry the whole analysis of a language defined by the document language
(it could be an object-field also).
Some extra fields might also be added like title_ws
(for whitespace
tokenization only) that have various approaches to the indexing operation,
with the aim of improving the relevancy.
One solution to simplify the query for API clients would be to use fields
like "title" and "doccontent" and to put as values very lightly (or
not at
all) analyzed content, as Paul suggested. This would allow applications to
write simple (and backwards compatible maybe) queries that will still work,
but will not catch some of the nuances of specific languages.
It's a good idea to go backwards compatible with such a predictable behaviour as the
whitespace analyzer.
Le 27 nov. 2012 à 16:27, Jerome Velociter a écrit :
Thus, the
search application will be the major beneficiary of these
analyzed fields (title_en, title_fr, etc.), while still allowing
applications to get their job done (trough generic, but less/not analized
fields like "title", "doccontent", etc.).
I think for applications this complexity/implementations details would benefit being
hidden behind a "query builder" interface of some sort, WDYT ? Maybe a solution would be to create one index per
language and index ALL
content regardless of it's language using the language analyzer of that
index.
I fear that this will bring zillions of false positive because many people have a long
list of supported language.
Stemming is quite aggressive sometimes... for example searching for sitting will find all
"sit", but this should not be the case if choosing French alone (then only the
gathering of people is meant).
If a browser indicates fr de as languages, and searches for sitting, this would find
documents with attachments that contain this "sit" in any language, even though
searching for English was not activated.
This later solution would be the only one that would
really work on file
attachements as we have no information about the specific language of file
attachements (or even XWiki objects) which are attached to the main
document and not to the translated document.
It is not entirely true that object fields and attachments do not carry a language.
But I agree that there may be installations where the admin would prefer that attachments
and objects are considered multilingual. Note that your solution is also doable with the
multi-field approach of above and does not require several indices.
There's the wiki language which one could apply to attachments and objects.
There could be object properties doing this (e.g. Curriki has this), it would need to be
customizable.
There's the language of documents of sections in several file formats (e.g. in PDF or
word files): while the current extractors do not honour this (I think), it could be used
to switch analyzers.
Again an option?
("index attachments in all languages", "index object fields in all
languages")
paul
4309
days inactive
4319
days old
13 comments
5 participants
participants (5)
-
Eduard Moraru -
Jerome Velociter -
Ludovic Dubost -
Paul Libbrecht -
Thomas Mortagne