28 Nov
2012
28 Nov
'12
6:07 p.m.
Hello devs,
This mail following a discussion we've had with Eduard on IRC concerning
the indexing of object property values. On the current Solr
implementation as well as on our lucene plugin, all property values are
stored as text/strings. I've expressed the idea that we probably want to
store each object's property in a field that matches the XClass property
field type. For example, store integers in integer field types, double
as doubles, etc.
My personnal use case is to store geo objects (for example long/lat
coordinates), but I think this has value for other types, numbers for
example (it means you can use those numbers as such when querying for
instance).
Now this will increase the complexity of querying since you would have
for example property_text, property_integer, property_double, etc. vs.
just propertvalue. Again, I think this complexity should be hidden by
the "expending API" Paul mentionned in the mail regarding document
translations.
WDYT ?
Jerome
28 Nov
28 Nov
9:24 p.m.
Jérôme,
you definitely have my +1.
And that's yet another feature of an automated schema generation.
One big plus of this is that indexing can optimize on these types for storage and
retrieval.
I'm fearing, however, that it adds to the work of solr implementation.
I do not understand why you would have property_text and property_integer... why not have
latitude, and latitude is a float?
I think you are aiming more and more at a flexible query expansion routine, correct?
(one that flexibly accepts field values depending on type and that expands across
languages for multilang queriers?). This goes along making the query expansion a default
first-class citizen and go away of the limited "a single string for a query".
paul
Le 28 nov. 2012 à 18:07, Jerome Velociter a écrit :
Hello devs,
This mail following a discussion we've had with Eduard on IRC concerning the indexing
of object property values. On the current Solr implementation as well as on our lucene
plugin, all property values are stored as text/strings. I've expressed the idea that
we probably want to store each object's property in a field that matches the XClass
property field type. For example, store integers in integer field types, double as
doubles, etc.
My personnal use case is to store geo objects (for example long/lat coordinates), but I
think this has value for other types, numbers for example (it means you can use those
numbers as such when querying for instance).
Now this will increase the complexity of querying since you would have for example
property_text, property_integer, property_double, etc. vs. just propertvalue. Again, I
think this complexity should be hidden by the "expending API" Paul mentionned in
the mail regarding document translations.
WDYT ?
Jerome
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
9:36 p.m.
Hi Paul
On 11/28/2012 09:24 PM, Paul Libbrecht wrote:
Jérôme,
you definitely have my +1.
And that's yet another feature of an automated schema generation.
I'm not sure where the schema generation lays here. I'm proposing to
have all XWiki field types mapped correctly in the schema.xml, not to
generate those at run-tim. There's still the question of custom field
types, though, now that XWiki supports them (via components implementing
MetaClass).
One big plus of this is that indexing can optimize on
these types for storage and retrieval.
I'm fearing, however, that it adds to the work of solr implementation.
I do not understand why you would have property_text and property_integer... why not have
latitude, and latitude is a float?
I took the example of integers because it's an object type we have in
XWiki already and that is not indexed as such in lucene/solr. For
lat/lon I leaning towards doing what is described here :
http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4#Indexing.
My idea is not to have field mappings per XWiki class, but per field
type. Is that what you meant as well ?
I think you are aiming more and more at a flexible query expansion routine, correct?
(one that flexibly accepts field values depending on type and that expands across
languages for multilang queriers?). This goes along making the query expansion a default
first-class citizen and go away of the limited "a single string for a query".
Yes that definitely make sense. Note that we could still have support
for a single string query (and will need to actually), and have it
expand according to settings, etc.
Thanks for your feedback
Jerome
paul
Le 28 nov. 2012 à 18:07, Jerome Velociter a écrit :
Hello devs,
This mail following a discussion we've had with Eduard on IRC concerning the indexing
of object property values. On the current Solr implementation as well as on our lucene
plugin, all property values are stored as text/strings. I've expressed the idea that
we probably want to store each object's property in a field that matches the XClass
property field type. For example, store integers in integer field types, double as
doubles, etc.
My personnal use case is to store geo objects (for example long/lat coordinates), but I
think this has value for other types, numbers for example (it means you can use those
numbers as such when querying for instance).
Now this will increase the complexity of querying since you would have for example
property_text, property_integer, property_double, etc. vs. just propertvalue. Again, I
think this complexity should be hidden by the "expending API" Paul mentionned in
the mail regarding document translations.
WDYT ?
Jerome
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
1 Dec
1 Dec
4:33 p.m.
Some more thoughts :
- We could introduce something like a "PropertyFieldResolver" component
role that for a passed BaseProperty determines what field it should be
indexed in. Then in ObjectPropertySolrMetadataExtractor we loop over the
responses and index accordingly.
- Idea : for fields that have a special type (numbers, geohash, etc.),
double index them both as text and their own type. It grows the index
size more than necessary but simplifies querying. This might affect the
document's score too, so not so sure it's a good idea. Any opinions ?
- Do we want dynamic field types declaration ? This would mean
generating solrconfig.xml and maintaining it in sync for the embedded
veresion. Maybe we will need this anyway for translations. If we don't
it means we have to include more types in the default config (for
example geohash type) and we lose in extensibility/modularism.
Jerome.
On 11/28/2012 06:07 PM, Jerome Velociter wrote:
Hello devs,
This mail following a discussion we've had with Eduard on IRC
concerning the indexing of object property values. On the current Solr
implementation as well as on our lucene plugin, all property values
are stored as text/strings. I've expressed the idea that we probably
want to store each object's property in a field that matches the
XClass property field type. For example, store integers in integer
field types, double as doubles, etc.
My personnal use case is to store geo objects (for example long/lat
coordinates), but I think this has value for other types, numbers for
example (it means you can use those numbers as such when querying for
instance).
Now this will increase the complexity of querying since you would have
for example property_text, property_integer, property_double, etc. vs.
just propertvalue. Again, I think this complexity should be hidden by
the "expending API" Paul mentionned in the mail regarding document
translations.
WDYT ?
Jerome
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
4403
days inactive
4406
days old
3 comments
2 participants
participants (2)
-
Jerome Velociter -
Paul Libbrecht