28 Nov
2012
28 Nov
'12
8:57 a.m.
To summarize a bit, if we go with the multiple fields
for each language, we
end up with an index like:
English version:
id: xwiki:Main.SomeDocument_en
language: en
space: Main
title_en: XWiki document
doccontent_en: This is some content
French version:
id: xwiki:Main.SomeDocument_fr
language: fr
space: Main
title_fr: XWiki document
doccontent_fr: This is some content
Careful Eduard,
that means that searching for common words (e.g "direction") would show you the
same XWiki document as two different search results. I think you would want to combine the
two in one Solr document. This is the old problem of Savitha which, I think, she has not
solved.
I've always believed that the fact that these two documents exist implies that they
both have the same meaning and are translations of each other.
I like the idea to carry the whole analysis of a language defined by the document language
(it could be an object-field also).
Some extra fields might also be added like title_ws
(for whitespace
tokenization only) that have various approaches to the indexing operation,
with the aim of improving the relevancy.
One solution to simplify the query for API clients would be to use fields
like "title" and "doccontent" and to put as values very lightly (or
not at
all) analyzed content, as Paul suggested. This would allow applications to
write simple (and backwards compatible maybe) queries that will still work,
but will not catch some of the nuances of specific languages.
It's a good idea to go backwards compatible with such a predictable behaviour as the
whitespace analyzer.
Le 27 nov. 2012 à 16:27, Jerome Velociter a écrit :
Thus, the
search application will be the major beneficiary of these
analyzed fields (title_en, title_fr, etc.), while still allowing
applications to get their job done (trough generic, but less/not analized
fields like "title", "doccontent", etc.).
I think for applications this complexity/implementations details would benefit being
hidden behind a "query builder" interface of some sort, WDYT ? Maybe a solution would be to create one index per
language and index ALL
content regardless of it's language using the language analyzer of that
index.
I fear that this will bring zillions of false positive because many people have a long
list of supported language.
Stemming is quite aggressive sometimes... for example searching for sitting will find all
"sit", but this should not be the case if choosing French alone (then only the
gathering of people is meant).
If a browser indicates fr de as languages, and searches for sitting, this would find
documents with attachments that contain this "sit" in any language, even though
searching for English was not activated.
This later solution would be the only one that would
really work on file
attachements as we have no information about the specific language of file
attachements (or even XWiki objects) which are attached to the main
document and not to the translated document.
It is not entirely true that object fields and attachments do not carry a language.
But I agree that there may be installations where the admin would prefer that attachments
and objects are considered multilingual. Note that your solution is also doable with the
multi-field approach of above and does not require several indices.
There's the wiki language which one could apply to attachments and objects.
There could be object properties doing this (e.g. Curriki has this), it would need to be
customizable.
There's the language of documents of sections in several file formats (e.g. in PDF or
word files): while the current extractors do not honour this (I think), it could be used
to switch analyzers.
Again an option?
("index attachments in all languages", "index object fields in all
languages")
paul