[xwiki-devs] Handling of property values indexing in Solr
Hello devs, This mail following a discussion we've had with Eduard on IRC concerning the indexing of object property values. On the current Solr implementation as well as on our lucene plugin, all property values are stored as text/strings. I've expressed the idea that we probably want to store each object's property in a field that matches the XClass property field type. For example, store integers in integer field types, double as doubles, etc. My personnal use case is to store geo objects (for example long/lat coordinates), but I think this has value for other types, numbers for example (it means you can use those numbers as such when querying for instance). Now this will increase the complexity of querying since you would have for example property_text, property_integer, property_double, etc. vs. just propertvalue. Again, I think this complexity should be hidden by the "expending API" Paul mentionned in the mail regarding document translations. WDYT ? Jerome
Jérôme, you definitely have my +1. And that's yet another feature of an automated schema generation. One big plus of this is that indexing can optimize on these types for storage and retrieval. I'm fearing, however, that it adds to the work of solr implementation. I do not understand why you would have property_text and property_integer... why not have latitude, and latitude is a float? I think you are aiming more and more at a flexible query expansion routine, correct? (one that flexibly accepts field values depending on type and that expands across languages for multilang queriers?). This goes along making the query expansion a default first-class citizen and go away of the limited "a single string for a query". paul Le 28 nov. 2012 à 18:07, Jerome Velociter a écrit :
Hello devs,
This mail following a discussion we've had with Eduard on IRC concerning the indexing of object property values. On the current Solr implementation as well as on our lucene plugin, all property values are stored as text/strings. I've expressed the idea that we probably want to store each object's property in a field that matches the XClass property field type. For example, store integers in integer field types, double as doubles, etc. My personnal use case is to store geo objects (for example long/lat coordinates), but I think this has value for other types, numbers for example (it means you can use those numbers as such when querying for instance). Now this will increase the complexity of querying since you would have for example property_text, property_integer, property_double, etc. vs. just propertvalue. Again, I think this complexity should be hidden by the "expending API" Paul mentionned in the mail regarding document translations.
WDYT ?
Jerome _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Paul On 11/28/2012 09:24 PM, Paul Libbrecht wrote:
Jérôme,
you definitely have my +1. And that's yet another feature of an automated schema generation.
I'm not sure where the schema generation lays here. I'm proposing to have all XWiki field types mapped correctly in the schema.xml, not to generate those at run-tim. There's still the question of custom field types, though, now that XWiki supports them (via components implementing MetaClass).
One big plus of this is that indexing can optimize on these types for storage and retrieval.
I'm fearing, however, that it adds to the work of solr implementation. I do not understand why you would have property_text and property_integer... why not have latitude, and latitude is a float?
I took the example of integers because it's an object type we have in XWiki already and that is not indexed as such in lucene/solr. For lat/lon I leaning towards doing what is described here : http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4#Indexing. My idea is not to have field mappings per XWiki class, but per field type. Is that what you meant as well ?
I think you are aiming more and more at a flexible query expansion routine, correct? (one that flexibly accepts field values depending on type and that expands across languages for multilang queriers?). This goes along making the query expansion a default first-class citizen and go away of the limited "a single string for a query".
Yes that definitely make sense. Note that we could still have support for a single string query (and will need to actually), and have it expand according to settings, etc. Thanks for your feedback Jerome
paul
Le 28 nov. 2012 à 18:07, Jerome Velociter a écrit :
Hello devs,
This mail following a discussion we've had with Eduard on IRC concerning the indexing of object property values. On the current Solr implementation as well as on our lucene plugin, all property values are stored as text/strings. I've expressed the idea that we probably want to store each object's property in a field that matches the XClass property field type. For example, store integers in integer field types, double as doubles, etc. My personnal use case is to store geo objects (for example long/lat coordinates), but I think this has value for other types, numbers for example (it means you can use those numbers as such when querying for instance). Now this will increase the complexity of querying since you would have for example property_text, property_integer, property_double, etc. vs. just propertvalue. Again, I think this complexity should be hidden by the "expending API" Paul mentionned in the mail regarding document translations.
WDYT ?
Jerome _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Some more thoughts : - We could introduce something like a "PropertyFieldResolver" component role that for a passed BaseProperty determines what field it should be indexed in. Then in ObjectPropertySolrMetadataExtractor we loop over the responses and index accordingly. - Idea : for fields that have a special type (numbers, geohash, etc.), double index them both as text and their own type. It grows the index size more than necessary but simplifies querying. This might affect the document's score too, so not so sure it's a good idea. Any opinions ? - Do we want dynamic field types declaration ? This would mean generating solrconfig.xml and maintaining it in sync for the embedded veresion. Maybe we will need this anyway for translations. If we don't it means we have to include more types in the default config (for example geohash type) and we lose in extensibility/modularism. Jerome. On 11/28/2012 06:07 PM, Jerome Velociter wrote:
Hello devs,
This mail following a discussion we've had with Eduard on IRC concerning the indexing of object property values. On the current Solr implementation as well as on our lucene plugin, all property values are stored as text/strings. I've expressed the idea that we probably want to store each object's property in a field that matches the XClass property field type. For example, store integers in integer field types, double as doubles, etc. My personnal use case is to store geo objects (for example long/lat coordinates), but I think this has value for other types, numbers for example (it means you can use those numbers as such when querying for instance). Now this will increase the complexity of querying since you would have for example property_text, property_integer, property_double, etc. vs. just propertvalue. Again, I think this complexity should be hidden by the "expending API" Paul mentionned in the mail regarding document translations.
WDYT ?
Jerome _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
participants (2)
-
Jerome Velociter -
Paul Libbrecht