[xwiki-devs] [GSOC] SOLR Search component
Hi all, I have come up with some basic set of API's for Solr search component. I have made it generic such that it can use solr or lucene as the backend. I have also documented in the DEsign proposal. http://dev.xwiki.org/xwiki/bin/view/Design/SOLRSearchIntegration *getBackend* public String getBackend() *Returns:* Returns the name of the backend which is currently in use. It could be Lucene or Solr. *rebuildIndex* * * *public int rebuildIndex(com.xpn.xwiki.api.XWiki wiki, com.xpn.xwiki.api.Context context)* Starts the rebuild of the whole index *Parameters:* wiki context *Returns:* Number of documents scheduled for indexing. -1 in case of errors * * *getSearchResults* * * * * * * * * public searchResponse *getSearchResults*(java.lang.String query, java.lang.string languages, com.xpn.xwiki.api.XWiki wiki) Searches the configured Indexes using the specified query for documents in the given languages. With virtual wikis enabled in your xwiki installation this will deliver results from all virtuall wikis. *Parameters : * query - query string given by the user languages - comma separated list of language codes to search in, may be null to search all languages. Language codes can be: - default for content having no specific language information - lower case 2- letter language codes like en, es , fr. *Returns:* a searchResponse instance containing the results (i.e) the response objects. *getSearchResults* public searchResponse *getSearchResults*(java.lang.String query, java.lang.String virtualWikiNames java.lang.string languages, com.xpn.xwiki.api.XWiki wiki) Searches the configured Indexes using the specified query for documents in the given languages belonging to one of the given virtual wikis. *Parameters : * query - query string given by the user languages - comma separated list of language codes to search in, may be null to search all languages. Language codes can be: - default for content having no specific language information - lower case 2- letter language codes like en, es , fr. virtualWikiNames - Names of the virtual wikis to search in. May be null for global search. *Returns:* a searchResponse instance containing the results (i.e) the response objects. I'm not sure on whether to expose the startIndex as API *Admin Module:* * * I has some thoughts on API's for admin settings where one could check the precision and recall results based on some known documents and could tweak the boost values accordingly. Thanking you, Savitha.
On 05/25/2012 04:48 PM, savitha sundaramurthy wrote:
Hi all,
I have come up with some basic set of API's for Solr search component. I have made it generic such that it can use solr or lucene as the backend. I have also documented in the DEsign proposal.
http://dev.xwiki.org/xwiki/bin/view/Design/SOLRSearchIntegration
*getBackend*
public String getBackend()
*Returns:*
Returns the name of the backend which is currently in use. It could be Lucene or Solr.
Not only, it could be internal Solr, remote Solr, internal Lucene, distributed Lucene. I'm not convinced this method is useful/needed.
*rebuildIndex*
* *
*public int rebuildIndex(com.xpn.xwiki.api.XWiki wiki, com.xpn.xwiki.api.Context context)*
Starts the rebuild of the whole index
*Parameters:*
wiki
context
*Returns:*
Number of documents scheduled for indexing. -1 in case of errors
Using the XWiki and XWikiContext classes is not encouraged, since these are old APIs being replaced by components. The method shouldn't receive any of these parameters, and it should get access to the needed components as injected dependencies. However, there are other parameters that could be useful, specifying what to index, although I'd rather use different methods: * public int rebuildFarmIndex() rebuilds the index for the whole wiki farm * public int rebuildWikiIndex() rebuilds the index for the current wiki * public int indexDocuments(List<DocumentReference> documents) reindexes the given set of documents
* * *getSearchResults* * * * * * * * *
public searchResponse *getSearchResults*(java.lang.String query,
java.lang.string languages,
com.xpn.xwiki.api.XWiki wiki)
Searches the configured Indexes using the specified query for documents in the given languages.
With virtual wikis enabled in your xwiki installation this will deliver results from all virtuall wikis.
*Parameters : *
query - query string given by the user
languages - comma separated list of language codes to search in, may be null to search all languages. Language codes can be:
- default for content having no specific language information - lower case 2- letter language codes like en, es , fr.
*Returns:*
a searchResponse instance containing the results (i.e) the response objects.
Again, the XWiki parameter shouldn't be there. I'd rather call the method "search" instead of "getSearchResults". Instead of a comma separated list of languages, I'd rather use a List<String>. I'm not sure that by default results should come from all wikis. Let's leave it as you proposed for the moment, and we'll think about it later.
*getSearchResults*
public searchResponse *getSearchResults*(java.lang.String query,
java.lang.String virtualWikiNames
java.lang.string languages,
com.xpn.xwiki.api.XWiki wiki)
Searches the configured Indexes using the specified query for documents in the given languages belonging to one of the given virtual wikis.
*Parameters : *
query - query string given by the user
languages - comma separated list of language codes to search in, may be null to search all languages. Language codes can be:
- default for content having no specific language information - lower case 2- letter language codes like en, es , fr.
virtualWikiNames - Names of the virtual wikis to search in. May be null for global search.
*Returns:* a searchResponse instance containing the results (i.e) the response objects.
Again, no XWiki parameter, and List<String> instead of String.
I'm not sure on whether to expose the startIndex as API
Yes, offset and limit should be part of the APIs.
*Admin Module:* * * I has some thoughts on API's for admin settings where one could check the precision and recall results based on some known documents and could tweak the boost values accordingly.
That's for later, let's start with the search first.
Thanking you, Savitha.
-- Sergiu Dumitriu http://purl.org/net/sergiu/
Hello Savitha, sorry to be slow at responding. Le 25 mai 2012 à 22:48, savitha sundaramurthy a écrit :
I have come up with some basic set of API's for Solr search component. I have made it generic such that it can use solr or lucene as the backend. I have also documented in the DEsign proposal.
http://dev.xwiki.org/xwiki/bin/view/Design/SOLRSearchIntegration
*getBackend* public String getBackend() *Returns:* Returns the name of the backend which is currently in use. It could be Lucene or Solr.
It would be useful to give use cases of things here. Wouldn't it make sense to rather have getImplementation?
*rebuildIndex* *public int rebuildIndex(com.xpn.xwiki.api.XWiki wiki, com.xpn.xwiki.api.Context context)*
Number of documents scheduled for indexing. -1 in case of errors
This is insufficient. In a big wiki such as Curriki.org, it can take several days to re-index. Aside of the proposal of Sergiu about the possible parameters, I'd suggest you create an "IndexerProcess" class with such APIs as: - getQueueSize - getPreQueueSize (sometimes, indexing processes have multiple queues) - getNextIndexerProcess (if another request was filed) - getEstimatedCompletionDate - getIndexingSpeed - getLastTenIndexedDocuments I also think we want to re-index based on an iterator of doc-names.
*getSearchResults* public SearchResponse *getSearchResults*(java.lang.String query, java.lang.string languages, com.xpn.xwiki.api.XWiki wiki)*Returns:* public searchResponse *getSearchResults*(java.lang.String query, java.lang.String virtualWikiNames, java.lang.string languages, com.xpn.xwiki.api.XWiki wiki) Searches the configured Indexes using the specified query for documents in the given languages belonging to one of the given virtual wikis.
*Parameters : *
query - query string given by the user
languages - comma separated list of language codes to search in, may be null to search all languages. Language codes can be:
- default for content having no specific language information - lower case 2- letter language codes like en, es , fr.
virtualWikiNames - Names of the virtual wikis to search in. May be null for global search.
*Returns:* a searchResponse instance containing the results (i.e) the response objects.
[...] a searchResponse instance containing the results (i.e) the response objects.
You need to document things here. I believe a simple inspiration is probably the structure of a SolrResponse. Methods such as getStart, getTotalResults, getSolrDocument(int_ (a SolrDocument!), getXWikiDocument, next. Think about this would be implemented in velocity. Consider the PageTool of SolrItas (which supports views into displaying or not links to previous or next pages of results).
I'm not sure on whether to expose the startIndex as API
Now that Sergiu has written, I understand the question!! It is crucial that each method has a start int and a maxResults! Maybe one or two commodity query methods can ignore it (and default to 0 and 100) but do not let buffers get filled by huge result-sets (and discourage any attempt to "go through all documents"). Also, please also provide search methods with query objects. This is the sole and only possibility to write an application whose search can be really tweaked preventing query-parts injections (similar to SQL injection). search(SolrQuery query, String wikis, List<String> langs, int start, int maxDocs) (... or Object query??) parseQuery(String q, List<String> langs, String queryModule) (queryModule would be something corresponding to a part of solrconfig.xml, maybe this is too much but this allows to have different solr-tunings to be exploited for different search-types)
*Admin Module:* I has some thoughts on API's for admin settings where one could check the precision and recall results based on some known documents and could tweak the boost values accordingly.
I believe I will need more customizability in the API but maybe that can be offered in the Admin module where such things as "ReIndexingPolicy" or "IndexDataFactory" can be configured for particular applications to be either Java classes or Groovy pages. paul
participants (3)
-
Paul Libbrecht -
savitha sundaramurthy -
Sergiu Dumitriu