30 Jun
2012
30 Jun
'12
3:27 p.m.
Hi All,
I have come up with the basic Solr Search Component. Have included
features such as hit highlighting.
You can find a running instance of the component here.
http://ec2-50-19-181-163.compute-1.amazonaws.com:8080/xwiki/bin/view/Search…
Below is the link to the documentation and the progress of the project.
http://dev.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/SOLRsearchcomponent
API
http://dev.xwiki.org/xwiki/bin/view/Design/SOLRSearchIntegration
Basic multi lingual support, and basic facet search are in progress to be
implemented for the coming week. Kindly give your suggestions.
--
Thanks a lot,
Savitha.s
30 Jun
30 Jun
8:26 p.m.
Hi,
This is nice progress. I've had a look and I have a few remarks:
1/ Some weird results
It seems the results are not always ok. For instance this page
http://ec2-50-19-181-163.compute-1.amazonaws.com:8080/xwiki/bin/view/Search…
comes up if I search for "SearchTest" but it does not come up for
"liste"
Also these 2 searches says 6 and 1 results and show only 3 and 0 results.
2/ Avanced queries
I was also wondering if we can use advanced queries.
I've been trying
SearchTest +space:SearchTest
and this does not seem to work.
3/ It's important that we end up with at least the same features as in
lucene. For instance being able to query all the fields we could query
in lucene is important. For instance:
object:XWiki.XWikiUsers
should return only users
Ordering and Scoring is also something that existed in lucene. How
would this work in SOLR ?
4/ we also want of course the advantages of SOLR, which means
facetting. Tags, Spaces, Wikis can be interesting facets
5/ in terms of multilingual search (in case of a multilingual wiki) we
need to make sure that you can say that you make a search in a
specific language and the correct stemmer is used (if stemming is used
at indexing time we need to index the content in each language with
the correct stemmer). I saw that you did some things with languages so
maybe SOLR has also other ways to handle this.
Keep up the good work
Ludovic
2012/6/30 savitha sundaramurthy <savitha.s131(a)gmail.com>om>:
Hi All,
I have come up with the basic Solr Search Component. Have included
features such as hit highlighting.
You can find a running instance of the component here.
http://ec2-50-19-181-163.compute-1.amazonaws.com:8080/xwiki/bin/view/Search…
Below is the link to the documentation and the progress of the project.
http://dev.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/SOLRsearchcomponent
API
http://dev.xwiki.org/xwiki/bin/view/Design/SOLRSearchIntegration
Basic multi lingual support, and basic facet search are in progress to be
implemented for the coming week. Kindly give your suggestions.
--
Thanks a lot,
Savitha.s
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
11:03 p.m.
Ah, feedback! This is really good.
Le 30 juin 2012 à 20:26, Ludovic Dubost a écrit :
This is nice progress. I've had a look and I have
a few remarks:
1/ Some weird results
It seems the results are not always ok. For instance this page
http://ec2-50-19-181-163.compute-1.amazonaws.com:8080/xwiki/bin/view/Search…
comes up if I search for "SearchTest" but it does not come up for
"liste"
Also these 2 searches says 6 and 1 results and show only 3 and 0 results.
This is due to the multi-lingual document (one document in four languages).
The multilinguality is, I think, on top of Savitha's priority.
2/ Avanced queries
I was also wondering if we can use advanced queries.
I've been trying
SearchTest +space:SearchTest
and this does not seem to work.
There's a good reason for this: the syntax for search currently in use is
"Dismax". This is a query-parser that is rather less technical, so it avoids
such issues as considering an apostrophe as a separator (an issue that was reported).
The queries you are suggesting, which I think can be useful, only work with the Lucene
Query-Parser, and not with dismax. This will be configurable but I am not sure which one
should be the default.
3/ It's important that we end up with at least the
same features as in
lucene.
Mmmh, not *all* of the features.
E.g. that all fields are stored is really not desired (and almost never used in search
results).
For instance being able to query all the fields we
could query
in lucene is important. For instance:
object:XWiki.XWikiUsers
should return only users
Something of this sort will be needed to achieve the advanced search scenario.
Ordering and Scoring is also something that existed in
lucene. How
would this work in SOLR ?
A score is already displayed currently.
4/ we also want of course the advantages of SOLR,
which means
facetting. Tags, Spaces, Wikis can be interesting facets
The reason multilingual documents have been a problem thus far is that Savitha is also
trying to make the language a facet which is really interesting but is raising an amount
of difficulties.
5/ in terms of multilingual search (in case of a
multilingual wiki) we
need to make sure that you can say that you make a search in a
specific language and the correct stemmer is used (if stemming is used
at indexing time we need to index the content in each language with
the correct stemmer). I saw that you did some things with languages so
maybe SOLR has also other ways to handle this.
If you look into the source, you can see some of that.
Solr can do this very nicely declaratively with the schema.xml and solrconfig.xml.
Part of Savitha's intent was to offer an adminstrative UI to manipulate this but
I'd personally prefer editing files manually. Or maybe we even have to invent an
extended schema syntax for XWiki-Solr (thus indicating that a field of solr, of this and
that type, tokenization and storage, if fed by a property x/yz of an xwiki document).
paul
2 Jul
2 Jul
11:12 a.m.
Hi,
looks very cool!
Indeed, I guess stemming still needs some improvements. For instance
searching for "embeter":
http://ec2-50-19-181-163.compute-1.amazonaws.com:8080/xwiki/bin/view/Search…
did not return any results although searching for "embêter":
http://ec2-50-19-181-163.compute-1.amazonaws.com:8080/xwiki/bin/view/Search…
returns 2 results whereas both should be the same.
Keep up the good work!
Guillaume
On Sat, Jun 30, 2012 at 11:03 PM, Paul Libbrecht <paul(a)hoplahup.net> wrote:
Ah, feedback! This is really good.
Le 30 juin 2012 à 20:26, Ludovic Dubost a écrit :
This is nice progress. I've had a look and I
have a few remarks:
1/ Some weird results
It seems the results are not always ok. For instance this page
http://ec2-50-19-181-163.compute-1.amazonaws.com:8080/xwiki/bin/view/Search…
comes up if I search for "SearchTest"
but it does not come up for "liste"
Also these 2 searches says 6 and 1 results and show only 3 and 0 results.
This is due to the multi-lingual document (one document in four languages).
The multilinguality is, I think, on top of Savitha's priority.
2/ Avanced queries
I was also wondering if we can use advanced queries.
I've been trying
SearchTest +space:SearchTest
and this does not seem to work.
There's a good reason for this: the syntax for search currently in use is
"Dismax". This is a query-parser that is rather less technical, so it
avoids such issues as considering an apostrophe as a separator (an issue
that was reported).
The queries you are suggesting, which I think can be useful, only work
with the Lucene Query-Parser, and not with dismax. This will be
configurable but I am not sure which one should be the default.
3/ It's important that we end up with at
least the same features as in
lucene.
Mmmh, not *all* of the features.
E.g. that all fields are stored is really not desired (and almost never
used in search results).
For instance being able to query all the fields
we could query
in lucene is important. For instance:
object:XWiki.XWikiUsers
should return only users
Something of this sort will be needed to achieve the advanced search
scenario.
Ordering and Scoring is also something that
existed in lucene. How
would this work in SOLR ?
A score is already displayed currently.
4/ we also want of course the advantages of SOLR,
which means
facetting. Tags, Spaces, Wikis can be interesting facets
The reason multilingual documents have been a problem thus far is that
Savitha is also trying to make the language a facet which is really
interesting but is raising an amount of difficulties.
5/ in terms of multilingual search (in case of a
multilingual wiki) we
need to make sure that you can say that you make a search in a
specific language and the correct stemmer is used (if stemming is used
at indexing time we need to index the content in each language with
the correct stemmer). I saw that you did some things with languages so
maybe SOLR has also other ways to handle this.
If you look into the source, you can see some of that.
Solr can do this very nicely declaratively with the schema.xml and
solrconfig.xml.
Part of Savitha's intent was to offer an adminstrative UI to manipulate
this but I'd personally prefer editing files manually. Or maybe we even
have to invent an extended schema syntax for XWiki-Solr (thus indicating
that a field of solr, of this and that type, tokenization and storage, if
fed by a property x/yz of an xwiki document).
paul
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
4 Jul
4 Jul
5:47 a.m.
Hi Ludovic and Paul,
On Sun, Jul 1, 2012 at 2:33 AM, Paul Libbrecht <paul(a)hoplahup.net> wrote:
It is possible and the parser am using is Extended Dismax and it supports
+/- and fielded queries. To support multilingual content indexing am using
a different set of fields. For english [ title_en, space_en, fulltext_en
...], French [ title_fr, space_fr, fulltext_fr ..], now when the user
searches for space:SearchText it may not work. To support this I should do
a bit of preprocessing to the query before calling solr server. I am
working on this too.
Yes, am currently considering Facets. To start with we can have Spaces and
Wikis. The new DocumentModelBridge and DocumentAccessBridge has some
missing API to access author, created date, modified date which can be
other interesting facets.
Ah, feedback! This is really good.
Le 30 juin 2012 à 20:26, Ludovic Dubost a écrit :
The embedded solr server running on the machine doesnt have ability to
handle multiple languages at this time. So it's returnng weird results. I
am working on it. Yes, paul its on my top priority and researching more
about it.
This is nice progress. I've had a look and I
have a few remarks:
1/ Some weird results
It seems the results are not always ok. For instance this page
http://ec2-50-19-181-163.compute-1.amazonaws.com:8080/xwiki/bin/view/Search…
comes up if I search for "SearchTest"
but it does not come up for "liste"
Also these 2 searches says 6 and 1 results and show only 3 and 0 results.
This is due to the multi-lingual document (one document in four languages).
The multilinguality is, I think, on top of Savitha's priority.
2/ Avanced queries
I was also wondering if we can use advanced queries.
I've been trying
SearchTest +space:SearchTest
and this does not seem to work.
There's a good reason for this: the syntax for search currently in use is
"Dismax". This is a query-parser that is rather less technical, so it
avoids such issues as considering an apostrophe as a separator (an issue
that was reported).
The queries you are suggesting, which I think can be useful, only work
with the Lucene Query-Parser, and not with dismax. This will be
configurable but I am not sure which one should be the default.
3/ It's important that we end up with at
least the same features as in
lucene.
Mmmh, not *all* of the features.
E.g. that all fields are stored is really not desired (and almost never
used in search results).
For instance being able to query all the fields
we could query
in lucene is important. For instance:
object:XWiki.XWikiUsers
should return only users
Something of this sort will be needed to achieve the advanced search
scenario.
Ordering and Scoring is also something that
existed in lucene. How
would this work in SOLR ?
A score is already displayed currently.
4/ we also want of course the advantages of SOLR,
which means
facetting. Tags, Spaces, Wikis can be interesting facets
The reason multilingual documents have been a problem thus far is that
Savitha is also trying to make the language a facet which is really
interesting but is raising an amount of difficulties.
5/ in terms of multilingual search (in case of a multilingual wiki) we
manually.
need to make sure that you can say that you make
a search in a
specific language and the correct stemmer is used (if stemming is used
at indexing time we need to index the content in each language with
the correct stemmer). I saw that you did some things with languages so
maybe SOLR has also other ways to handle this.
If you look into the source, you can see some of that.
Solr can do this very nicely declaratively with the schema.xml and
solrconfig.xml.
Part of Savitha's intent was to offer an adminstrative UI to manipulate
this but I'd personally prefer editing files manually. Or maybe we even
have to invent an extended schema syntax for XWiki-Solr (thus indicating
that a field of solr, of this and that type, tokenization and storage, if
fed by a property x/yz of an xwiki document).
As a start am going to keep it simple by editing the schema files paul
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
best regards,
Savitha.s
5 Jul
5 Jul
9:27 p.m.
Le 4 juil. 2012 à 05:47, savitha sundaramurthy a écrit :
Savitha,
this description reminds me of an old problem I had: having a whole infrastructure with
fixed analyzers. At the time I had simply written an analyzers that used thread-local to
recap the language and fork to the right analyzer at the tokenStream method. However, I
think there is more modern to this: MultiplexingAnalyzer or something like this. The
classics seem to indicated that using a suffix with the language is useful (and allows the
administrator to configure which analyzer is used for which language).
Paul
This is due to
the multi-lingual document (one document in four languages).
The multilinguality is, I think, on top of Savitha's priority.
The embedded solr server running on the machine doesnt have ability to
handle multiple languages at this time. So it's returnng weird results. I
am working on it. Yes, paul its on my top priority and researching more
about it.
6 Jul
6 Jul
9:22 p.m.
On Thu, Jul 5, 2012 at 12:27 PM, Paul Libbrecht <paul(a)hoplahup.net> wrote:
Le 4 juil. 2012 à 05:47, savitha sundaramurthy a écrit :
Savitha,
this description reminds me of an old problem I had: having a whole
infrastructure with fixed analyzers. At the time I had simply written an
analyzers that used thread-local to recap the language and fork to the
right analyzer at the tokenStream method. However, I think there is more
modern to this: MultiplexingAnalyzer or something like this. The classics
seem to indicated that using a suffix with the language is useful (and
allows the administrator to configure which analyzer is used for which
language).
I'm using dynamic fields(_en, _es, _fr etc...) to configure the fields in
schema.xml. As you said configuring this way is lot easier.
--
Thanks,
Savi
> This is due to the multi-lingual document
(one document in four
languages).
The
multilinguality is, I think, on top of Savitha's priority.
The embedded solr server running on the machine doesnt have ability to
handle multiple languages at this time. So it's returnng weird results. I
am working on it. Yes, paul its on my top priority and researching more
about it.
4552
days inactive
4558
days old
6 comments
4 participants
participants (4)
-
Guillaume Lerouge -
Ludovic Dubost -
Paul Libbrecht -
savitha sundaramurthy