[xwiki-devs] about lucene plugin
Hello XWiki devs, I've been playing a bit to adapt the xwiki lucene plugin, actually the curriki lucene plugin which is very similar, to the tastes of Intergeo. One thing that surprised me is that most of the fields in the lucene plugin are stored fields. This sounds like a big overkill when you think that all the values of the fields of a document are loaded when you have this document as match. Not surprisingly, the curriki search result was almost only using fullname, using lucene directly is somewhat faster, I think it's good practice. Could this be configurable? (i.e. the names of the fields that are stored) One powerful way to make it configurable would to migrate to solr which bases on lucene but uses a schema that describes well type of the fields. Three other pluggeabilities I would wish: - the search-result class really would wish to be subclassable, the configuration could define a search-result-factory or something such which would help (a lot) as a cursor exploiting the hits list - I deem it very important to also have query expansion but maybe that can be inside the rest of the application - what is indexed how should also be subclasses (a pluggeable factory for ObjectData?) Finally, I wanted to use different analyzers per language, using the Snowball family of analyzers so that stemming really does happen. I kind of have it now but I would like to test it in unit tests and, well... how can I do that? Is there a way to get a "unit-test xwiki environment" (context, xwiki) where my unit tests would create a few documents then query them? Is there active work on this plugin? Which patch is most interesting for the others? paul
Hi Paul, On Tue, Apr 7, 2009 at 9:12 PM, Paul Libbrecht <[email protected]> wrote:
Hello XWiki devs,
I've been playing a bit to adapt the xwiki lucene plugin, actually the curriki lucene plugin which is very similar, to the tastes of Intergeo.
One thing that surprised me is that most of the fields in the lucene plugin are stored fields. This sounds like a big overkill when you think that all the values of the fields of a document are loaded when you have this document as match.
Not surprisingly, the curriki search result was almost only using fullname, using lucene directly is somewhat faster, I think it's good practice. Could this be configurable? (i.e. the names of the fields that are stored) One powerful way to make it configurable would to migrate to solr which bases on lucene but uses a schema that describes well type of the fields.
Three other pluggeabilities I would wish: - the search-result class really would wish to be subclassable, the configuration could define a search-result-factory or something such which would help (a lot) as a cursor exploiting the hits list - I deem it very important to also have query expansion but maybe that can be inside the rest of the application - what is indexed how should also be subclasses (a pluggeable factory for ObjectData?)
Finally, I wanted to use different analyzers per language, using the Snowball family of analyzers so that stemming really does happen.
I kind of have it now but I would like to test it in unit tests and, well... how can I do that? Is there a way to get a "unit-test xwiki environment" (context, xwiki) where my unit tests would create a few documents then query them?
Is there active work on this plugin?
I'm not able to answer your technical questions, but I wanted to let you know that we plan on working on the Lucene plugin in an upcoming release, hopefully during the summer. We're not sure yet whether it's better to rewrite the plugin or to make iterative improvements to the existing one. To sum up: there will soon (= in the coming months, not tomorrow morning) be active work on the plugin and any help & suggestions are welcome as to what is the best way to make it work optimally for XWiki :-) Guillaume
Which patch is most interesting for the others?
paul _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Guillaume Lerouge Product Manager - XWiki Skype ID : wikibc http://guillaumelerouge.com/
Guillaume, that is good news! I would really ove to apache SOLR to provide the possibility to tune finer the indexing and query process, thereby also enabling the extraction of a remote search server (keeping the plugin for the scheduling of updates and access to solr. One of the features Vincent noted on Compass was the ability to do POJOs... I am not very convinced here since a normal xwiki application developer is actually doing... XOJOs and xwiki-lucene follows this rather well. In any cases, I've noted that solr supports indexing bean properties. Among the bigger advantages is also that solr is a proven search tool in large industry projects, supports distribution and replication, as well as highlighting. It originated from CNET where it is still used. In terms of API it's rather simpler than Lucene. Between rewrite and updates, I would keep at least the queue and scheduling part which is pretty clean and useful to me. paul Le 07-avr.-09 à 21:49, Guillaume Lerouge a écrit :
I'm not able to answer your technical questions, but I wanted to let you know that we plan on working on the Lucene plugin in an upcoming release, hopefully during the summer. We're not sure yet whether it's better to rewrite the plugin or to make iterative improvements to the existing one. To sum up: there will soon (= in the coming months, not tomorrow morning) be active work on the plugin and any help & suggestions are welcome as to what is the best way to make it work optimally for XWiki :-)
Paul Libbrecht wrote:
I would really ove to apache SOLR to provide the possibility to tune finer the indexing and query process, thereby also enabling the extraction of a remote search server (keeping the plugin for the scheduling of updates and access to solr.
One of the features Vincent noted on Compass was the ability to do POJOs... I am not very convinced here since a normal xwiki application developer is actually doing... XOJOs and xwiki-lucene follows this rather well. In any cases, I've noted that solr supports indexing bean properties.
Among the bigger advantages is also that solr is a proven search tool in large industry projects, supports distribution and replication, as well as highlighting. It originated from CNET where it is still used.
In terms of API it's rather simpler than Lucene.
Between rewrite and updates, I would keep at least the queue and scheduling part which is pretty clean and useful to me.
I looked at SOLR a while back, and I didn't like it. Maybe I didn't see all the ways it can be used, so correct me if I'm wrong: it only works as a standalone webapp talking over HTTP. If that is the case, then I don't like that, since it implies conversion to XML and parsing back, opening HTTP connections, network transfer. And in the end it still gets to Lucene. Using Lucene internally involves just a simple method call. Much faster. -- Sergiu Dumitriu http://purl.org/net/sergiu/
Yes, I had the same feeling a while back but it is really really different. solrj is what you want to look at, e.g: http://wiki.apache.org/solr/Solrj there the solrserver can be either an embedded server (xwiki could make this the default) or be an external one talked to over http. It is true the default solr tutorial shows only the http access. What it brings however is declarative field configuration and highlighting... both which are somewhat hard with the current plugin. paul PS: Solr is even a default component of Zend framework... Le 08-avr.-09 à 12:47, Sergiu Dumitriu a écrit :
I looked at SOLR a while back, and I didn't like it. Maybe I didn't see all the ways it can be used, so correct me if I'm wrong: it only works as a standalone webapp talking over HTTP. If that is the case, then I don't like that, since it implies conversion to XML and parsing back, opening HTTP connections, network transfer. And in the end it still gets to Lucene. Using Lucene internally involves just a simple method call. Much faster.
participants (3)
-
Guillaume Lerouge -
Paul Libbrecht -
Sergiu Dumitriu