Re: [xwiki-devs] [DISCUSSION] Handling document translations in Solr Search

2012/11/28 Eduard Moraru <enygma2002(a)gmail.com> If the user is monolingual, we can safely ignore the language setting of each document and only the "main" document will be shown anyway to the user. So we should make sure we search on ALL documents that are available to the user. The user might have to "reindex" to make sure this is properly taken into account by the search engine. What is important here, is that even if the wiki is set to "fr", then even if documents have "en" for the main language they will still show up in the search. The opposite would be bad. Sure. I was just repeating what is important. "French" but look in the "French" + "English" dataset. For me if both documents with the same page name match, both should come out separately. Not specifically as an API I mean here that as a technical user your objective is to make sure your search spans ALL content in the wiki. In this case you don't care about stemming and such. The standard UI could take this into account by choosing "any language search" and with data set "all languages". We just need to make sure that this won't exclude any content of the search (Here is an example of a case where such exclusion might occur. Suppose you have done only "French" and "English" indexing and there is "German" content also in your wiki but since you have not asked for German search in your wiki you don't have title_de and content_de fields or don't have a german specific solr index (in the other method), then your german content would not be indexed at all ?) I undestand the issue here, but in most cases the user will say "french search" on "french content", he will only expand to non french content if he was not satisfied by his search. What is just allowed here is to also search in "french" inside all content. That would cover content that would have the bad language setting as well as any other content. The results might be a bit noisy but I don't think it's a big issue. over multiple cores this could be a showstopper. However the solution of having only one Lucene document for all languages is not so cool either as it would make it difficult to know which ones of the languages has matched and present them separately with separate scores. It's really the core issue to decide on. What are the benefits and drawbacks of the different solutions. For each solution is there something in the UI that you cannot do ? So far I've heard: 1/ Presenting different scores for documents in different languages with the same doc name if the title_fr,content_fr method is used 2/ Merging scores accross indexes in multicore approach Other ? Can we list them in a wiki page ? This is not a good solution. If I understand correctly we could not end up not searching in a french attachment because the original document is in marked "en". I'm for Paul's solution to index objects and attachments in each translation (if we have separate entities for translated documents). I understand that in the title_fr,content_fr approach this problem does not happen. It seems we have to do this for now I'm not sure I understand. We have an "objcontent" field for each translation that has the full content of objects and properties, but individual object fields we don't have in each translations ? The more I see all the issues, the more I lean towards a separate index per language solution. The reason I do is that the main need is for a non English user to have very relevant results in his own language. Therefore we need to make sure that all content that the users have published has the chance to be analyzed using the non English language analyzer. So indexing all objects and attachments with the relevant language analyzer is the solution. This is also why I proposed to index all content in this specific index regardless of the language declared, which would only be used in the UI to limit searches to the specific language. In this view: 0/ There would be a language specific index per language with the objects and attachments indexed only in the language of the index 1/ The user chooses the language in which he searches 2/ Automatically that sets the index to be used to be the "french" index 3/ Automatically that presets to limit the span of the search to declared "french" documents 4/ The user can decide to go for non french documents at his own risks knowing that the results might be weird because of wrong analysis (this is what happens today with english analysis over french documents) The benefit here is that you don't have a merging score over multiple index issue, since you would never have to do a search across multiple indexes. Searches are still simple to write. By default results are quite relevant since you limit the search the french declared documents (this would be the same as limiting your search to title_fr, content_fr) and still cover what needs to be covered (objects and attachments). Another benefit is that this falls back gracefully to monolingual as you just have to have one index in the language declared for the monolingual wiki. The drawback is that the indexing is more costly and there is duplicated content in the index. Howerver it is the Admin that say which languages he wants available and he takes responsibility of the ressources this needs. Could this solution work ? Ludovic -- Ludovic Dubost Founder and CEO Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost