Hi, this looks very very promising. We've been struggling with this kind of basic search-related issues for a loooong time and it looks like Solr has everything that we need already built-in to address many of them. I look forward to seeing more of this, it's very cool! Guillaume On Thu, Apr 5, 2012 at 10:52, Paul Libbrecht <[email protected]> wrote:

Ok, so this is a little script reporting usage. It looks ok but I need to touch software.

I could not build your project because of checkstyle errors. Could you fix that?

Also, I insist on forking for me to see the diffs.

And please remember my request for readme to explain a typical deployment.

thanks in advance

paul

PS: what's your regular work schedule?

Le 5 avr. 2012 à 04:57, savitha sundaramurthy a écrit :

...

Hi Paul,

Here is the link to the document

https://docs.google.com/document/pub?id=1-2GqvjtzS_a8TRKDNsom9egDZHTT5xIzUf1...

...

Thanks a lot, Savitha

...

On Wed, Apr 4, 2012 at 12:03 PM, Paul Libbrecht <[email protected]>

wrote:

...

...

...

Hi Savitha,

things are moving. This is good.

Do you have a (temporary) server address where this could be tested? (maybe best over IRC?). Or, how could I build your code and inject it on a fresh XWiki?

Le 4 avr. 2012 à 18:46, savitha sundaramurthy a écrit :

...

I went through Fabio Mancinelli's code and with the help of few inputs from there , I have implemented a basic search functionality

using

...

solr. It also fixes the issue http://jira.xwiki.org/browse/XWIKI-6226.

I

...

...

...

have indexed the page in English, french and Spanish. I have implemented these main functionality:

1) Simple text search using solr. 2) Used *Extended Dismax Parser *to customize search relevancy using boost index.

which fixes well the apostrophe case for sure (and a zillion other ambiguities, and another zillion power user tricks).

...

3) Hit highlighting for English.

This is a good usage of solr.

...

Below is the link to the source code https://github.com/savis/xwiki-platform-search XWiki front end , HTML and Velocity code

https://gist.github.com/2295648

I wanted to send a pull request but I was playing around when renaming my username and deleted the forked repository and I'm having difficulty forking it again. So I have created a new repository and shared the code above.

I think forking is really useful. If not today, it will be needed any day soon. We need to see the diffs. Best would be to create the fork again, then copy your files and let git give us the diff.

...

I have taken few screen shots and have attached the document.

where?

...

I'm working on my application and it should be done today.

What is the planned schedule?

paul _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- best regards, Savitha.s

-- best regards, Savitha.s _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

On Thu, Apr 5, 2012 at 8:32 PM, savitha sundaramurthy <[email protected]> wrote:

Hi Paul,

On Thu, Apr 5, 2012 at 1:52 AM, Paul Libbrecht <[email protected]> wrote:

...

Ok, so this is a little script reporting usage. It looks ok but I need to touch software.

My code is here:

https://github.com/savee1/xwiki-platform/tree/SOLR/xwiki-platform-core/xwiki...

...

I could not build your project because of checkstyle errors. Could you fix that?

Also, I insist on forking for me to see the diffs.

I have forked the code, Sent a pull request to fabio. My commit and the list of diffs for all the files.

Yep I've seen it, though I don't think it's the correct way to proceed. The pull request, in fact, is sent to a branch of a fork in my account. I think that this kind of development should be done in a more "institutional" place like xwiki-contrib. Moreover, since I am not the mentor for this project, I am not entitled to merge pull requests. So I propose to open a fork of xwiki-platform in xwiki-contrib, and that Savitha works on that fork and (re)send pull requests there. WDYT? -Fabio

https://github.com/savee1/xwiki-platform/commit/5bb50fe2ddb12c1ef68fb83e0962...

...

And please remember my request for readme to explain a typical deployment.

I have added the README document too with detailed steps on how to set it up.

...

thanks in advance

paul

PS: what's your regular work schedule?

 I stay in Arizona, its Mountain Standard Time  ( -7 ).  I have classes from morning till evening. I'll be available in IRC in the night and full time during weekends [?]  In summer, except for my thesis work( 20 hours a week) I'm free the other time.

...

Le 5 avr. 2012 à 04:57, savitha sundaramurthy a écrit :

...

Hi Paul,

     Here is the link to the document

https://docs.google.com/document/pub?id=1-2GqvjtzS_a8TRKDNsom9egDZHTT5xIzUf1...

...

Thanks a lot, Savitha

...

On Wed, Apr 4, 2012 at 12:03 PM, Paul Libbrecht <[email protected]>

wrote:

...

...

...

Hi Savitha,

things are moving. This is good.

Do you have a (temporary) server address where this could be tested? (maybe best over IRC?). Or, how could I build your code and inject it on a fresh XWiki?

Le 4 avr. 2012 à 18:46, savitha sundaramurthy a écrit :

...

       I went through Fabio Mancinelli's code and with the help of few inputs from there , I have implemented a basic  search functionality using solr. It also fixes the issue http://jira.xwiki.org/browse/XWIKI-6226.

I

...

...

...

have indexed the page in English, french and Spanish. I have implemented these main functionality:

1) Simple text search using solr. 2) Used *Extended Dismax Parser *to customize search relevancy using boost index.

which fixes well the apostrophe case for sure (and a zillion other ambiguities, and another zillion power user tricks).

...

3) Hit highlighting for English.

This is a good usage of solr.

...

Below is the link to the source code https://github.com/savis/xwiki-platform-search XWiki front end , HTML and Velocity code

https://gist.github.com/2295648

I wanted to send a pull request but I was playing around when renaming my username and deleted the forked repository and I'm having difficulty forking it again. So I have created a new repository and shared the code above.

I think forking is really useful. If not today, it will be needed any day soon. We need to see the diffs. Best would be to create the fork again, then copy your files and let git give us the diff.

...

I have taken few screen shots and have attached the document.

where?

...

I'm working on my application and it should be done today.

What is the planned schedule?

paul _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- best regards, Savitha.s

-- best regards, Savitha.s _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- best regards, Savitha.s

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

On 04/05/2012 02:37 PM, Fabio Mancinelli wrote:

On Thu, Apr 5, 2012 at 8:32 PM, savitha sundaramurthy <[email protected]> wrote:

...

Hi Paul,

On Thu, Apr 5, 2012 at 1:52 AM, Paul Libbrecht<[email protected]> wrote:

...

Ok, so this is a little script reporting usage. It looks ok but I need to touch software.

My code is here:

https://github.com/savee1/xwiki-platform/tree/SOLR/xwiki-platform-core/xwiki...

...

I could not build your project because of checkstyle errors. Could you fix that?

Also, I insist on forking for me to see the diffs.

I have forked the code, Sent a pull request to fabio. My commit and the list of diffs for all the files.

Yep I've seen it, though I don't think it's the correct way to proceed. The pull request, in fact, is sent to a branch of a fork in my account.

I think that this kind of development should be done in a more "institutional" place like xwiki-contrib.

Moreover, since I am not the mentor for this project, I am not entitled to merge pull requests.

So I propose to open a fork of xwiki-platform in xwiki-contrib, and that Savitha works on that fork and (re)send pull requests there.

WDYT?

That won't work, there is a fork already, and github only allows one fork per user (or organization). Let's follow the github philosophy, which is "be social" and work on Savitha's personal fork for the moment. -- Sergiu Dumitriu http://purl.org/net/sergiu/

On 04/08/2012 12:04 PM, Paul Libbrecht wrote:

Le 8 avr. 2012 à 04:07, Sergiu Dumitriu a écrit :

...

...

So I propose to open a fork of xwiki-platform in xwiki-contrib, and that Savitha works on that fork and (re)send pull requests there.

WDYT?

That won't work, there is a fork already, and github only allows one fork per user (or organization).

Let's follow the github philosophy, which is "be social" and work on Savitha's personal fork for the moment.

Fabio,

I insisted on a fork from your repo (with the right branch, that was the trick I and Savitha failed to catch for a while) so that we can evaluate the contribution of Savitha to this work. He found it independently of me.

This is now visible in commit: https://github.com/savee1/xwiki-platform/commit/5bb50fe2ddb12c1ef68fb83e0962...

In all cases, I think the work of Savitha will be on his GitHub repo, at best as a branch from xwiki directly, where a pull request will be able to be formulated. I wouldn't be surprised, however, that several pull requests will be needed in a non-linear way so that things will not be as simple as GitHub wishes us to be. In this case, there will be diff juggling.

Maybe it will be even easier that he makes a new repo, not based on others, so that it becomes something people can deploy when they feel its ready. The big repos at GitHub are not too friendly for this kind of sub-modules unfortunately (this is a great loss to my taste compared to svn).

So... what is the procedure to review his work and give our ranking? Is this only in Melange or is there a way inside? I haven't been able to check, yet, if it runs as desired...

For the moment Melange is the right place for grading proposals, and this should be done before Tuesday when we're supposed to say how many slots we want. -- Sergiu Dumitriu http://purl.org/net/sergiu/

On Thu, Apr 5, 2012 at 2:43 PM, Ludovic Dubost <[email protected]> wrote:

Hi Savitha,

This is quite cool. I'm quite impressed at the speed you learned to do components and manage to do a first basic implementation. It indeed seems to show that SOLR is a good choice.

I've one remark which is the usage of a SOLR Embedded server. I think one of the interesting features of SOLR is also that we can split the search engine which the XWiki engine.

I think that is a heritage of my initial code... Anyway I think it's just a matter of initialization: SolrServer server = new CommonsHttpSolrServer("http://HOST:8983/solr/"); or SolrServer server = new EmbeddedSolrServer(); See: http://wiki.apache.org/solr/Solrj#EmbeddedSolrServer

It would be great to have the choice between both solutions (embedded and not embedded). By default we would bundle the embedded version and for high performance installation it should be possible to separate the SOLR engine and still share most of the logic.

Keep up the good work

Ludovic

2012/4/5 savitha sundaramurthy <[email protected]>

...

Hi Paul,

     Here is the link to the document

https://docs.google.com/document/pub?id=1-2GqvjtzS_a8TRKDNsom9egDZHTT5xIzUf1...

Thanks a lot, Savitha

...

On Wed, Apr 4, 2012 at 12:03 PM, Paul Libbrecht <[email protected]> wrote:

...

Hi Savitha,

things are moving. This is good.

Do you have a (temporary) server address where this could be tested? (maybe best over IRC?). Or, how could I build your code and inject it on a fresh XWiki?

Le 4 avr. 2012 à 18:46, savitha sundaramurthy a écrit :

...

        I went through Fabio Mancinelli's code and with the help of few inputs from there , I have implemented a basic  search functionality using solr. It also fixes the issue http://jira.xwiki.org/browse/XWIKI-6226. I have indexed the page in English, french and Spanish. I have implemented these main functionality:

1) Simple text search using solr. 2) Used *Extended Dismax Parser *to customize search relevancy using boost index.

which fixes well the apostrophe case for sure (and a zillion other ambiguities, and another zillion power user tricks).

...

3) Hit highlighting for English.

This is a good usage of solr.

...

Below is the link to the source code https://github.com/savis/xwiki-platform-search XWiki front end , HTML and Velocity code

https://gist.github.com/2295648

I wanted to send a pull request but I was playing around when renaming my username and deleted the forked repository and I'm having difficulty forking it again. So I have created a new repository and shared the code above.

I think forking is really useful. If not today, it will be needed any day soon. We need to see the diffs. Best would be to create the fork again, then copy your files and let git give us the diff.

...

I have taken few screen shots and have attached the document.

where?

...

I'm working on my application and it should be done today.

What is the planned schedule?

paul _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- best regards, Savitha.s

-- best regards, Savitha.s _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- Ludovic Dubost Founder and CEO Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

Hello Savitha, I took a bit of time to check your patch. Here's a few comments, not really in order but more as a recipient of the work. The readme is ok... I am not sure it follows the xwiki practices. I guess we should discuss how this search should be advertised one day, either as an extension (the readme would be in the wiki then) or as a default component. Shouldn't the versions be 3.5.1-SNAPSHOT? The values in xwiki.cfg all carry the prefix solr, which is good, but I think the prefix search must be kept (so it's be search.solr or something so). Currently, solr-home is set by an environment variable. There must be a way to define the solr home without it... not sure how or when this should be the case but this is a hard issue I think since many deploy their xwiki in a place where the startup variable are not even touchable. The readme says to copy two jars, but they have the same name (one of them should be with search-solr). It is excellent that your readme works with an identified version of Lucene and one of SOLR, it has often been the case that identified versions of solr worked with a privately crafted lucene version. I am not sure all the packages you include are needed, e.g. why is lucene-spatial useful? why is spellchecker useful? is solr-velocity really useful? (isn't that a completely different velocity run method?) why is solr-cell (the document attachments' parser I think) not useful? You have omitted the expansion of solr_config.tar.gz in your readme at the right place. I note that it'd be highly wishable to separate solr-home/conf (typically inside WEB-INF/solr) and solr-home/index (typically inside work/solr) but it may be something hard to achieve. I had a slightly difficult time to make it all work at the start with the view code indicated in the readme. Too many things to consider for a first time (sensible defaults should just work, the fields should be prefilled, also, I think the form should get and not post so that search results are bookmarkable). Also, I think you did not include the highlighting in the current search view code, but that's ok for now. Also, you haven't indicated in the readme that the wiki should be multilingual for all the tests. But all in all it is quite good and it worked for me. So I am happy with this first contribution which shows that it's possible for you to get something workable in short time. There remains an amount of issues which should all be addressed when (and if) the project starts. I have a rather good feeling. paul Le 6 avr. 2012 à 18:47, savitha sundaramurthy a écrit :

Thanks Ludovic and Guillaume.

As Fabio pointed out changing the instantiation of SolrServer(), new EmbeddedServer() , or CommonHttpServer() will do. If any case, there is something else to offer I'll make it into two different components.

Am really looking forward to work with XWiki community.

Thanks , Savitha

On Thu, Apr 5, 2012 at 5:43 AM, Ludovic Dubost <[email protected]> wrote:

...

Hi Savitha,

This is quite cool. I'm quite impressed at the speed you learned to do components and manage to do a first basic implementation. It indeed seems to show that SOLR is a good choice.

I've one remark which is the usage of a SOLR Embedded server. I think one of the interesting features of SOLR is also that we can split the search engine which the XWiki engine. It would be great to have the choice between both solutions (embedded and not embedded). By default we would bundle the embedded version and for high performance installation it should be possible to separate the SOLR engine and still share most of the logic.

Keep up the good work

Ludovic

2012/4/5 savitha sundaramurthy <[email protected]>

...

Hi Paul,

Here is the link to the document

https://docs.google.com/document/pub?id=1-2GqvjtzS_a8TRKDNsom9egDZHTT5xIzUf1...

...

Thanks a lot, Savitha

...

On Wed, Apr 4, 2012 at 12:03 PM, Paul Libbrecht <[email protected]> wrote:

...

Hi Savitha,

things are moving. This is good.

Do you have a (temporary) server address where this could be tested? (maybe best over IRC?). Or, how could I build your code and inject it on a fresh XWiki?

Le 4 avr. 2012 à 18:46, savitha sundaramurthy a écrit :

...

I went through Fabio Mancinelli's code and with the help of

few

...

inputs from there , I have implemented a basic search functionality using solr. It also fixes the issue

http://jira.xwiki.org/browse/XWIKI-6226.

...

I

...

...

...

have indexed the page in English, french and Spanish. I have implemented these main functionality:

1) Simple text search using solr. 2) Used *Extended Dismax Parser *to customize search relevancy using boost index.

which fixes well the apostrophe case for sure (and a zillion other ambiguities, and another zillion power user tricks).

...

3) Hit highlighting for English.

This is a good usage of solr.

...

Below is the link to the source code https://github.com/savis/xwiki-platform-search XWiki front end , HTML and Velocity code

https://gist.github.com/2295648

I wanted to send a pull request but I was playing around when renaming my username and deleted the forked repository and I'm having difficulty forking it again. So I have created a new repository and shared the code above.

I think forking is really useful. If not today, it will be needed any day soon. We need to see the diffs. Best would be to create the fork again, then copy your files and let git give us the diff.

...

I have taken few screen shots and have attached the document.

where?

...

I'm working on my application and it should be done today.

What is the planned schedule?

On Tue, Apr 10, 2012 at 9:19 AM, savitha sundaramurthy <[email protected]> wrote:

Hi Paul,

      Thanks a lot for your comments.

...

I took a bit of time to check your patch.

Here's a few comments, not really in order but more as a recipient of the work.

The readme is ok... I am not sure it follows the xwiki practices. I guess we should discuss how this search should be advertised one day, either as an extension (the readme would be in the wiki then) or as a default component.

   The Readme is not complete. I didn't follow the XWiki standard

practice as I created one to find  the necessary steps to set it   up. I prefer having a complete documentation in dev wiki or extension wiki.

...

Shouldn't the versions be 3.5.1-SNAPSHOT?

   This is little confusing, dunno how the community follows it. As said by Fabio, if we are going to use XWiki Contrib repository its    good to start with 1.0.0-SNAPSHOT (using Continuos Integration ) or 1.0.0. If this is going into platform it should be    4.x-SNAPSHOT. I see the master repository is on 4.x in github.

Depends if it's an extension or supposed to be integrated into platform. But I guess for the GSOC it's better to see it as an extension and change version etc later if we decide to move it. Since an extension has its own versionning it should start at 1.0 so in your case 1.0-SNAPSHOT since no release has been done yet.

...

The values in xwiki.cfg all carry the prefix solr, which is good, but I think the prefix search must be kept (so it's be search.solr or something so).

    I agree with search.solr.*. But am doubtful about xwiki.cfg .

Recently, I found out that there is one more file called    xwiki.properties, looks like its a better place to set up the configuration.

Yes, xwiki.cfg is actually the old place. Also with xwiki.properties you are able to use default configuration source which is a lot nicer to use that the way we get properties from xwiki.cfg. See http://extensions.xwiki.org/xwiki/bin/view/Extension/Configuration+Module

...

Currently, solr-home is set by an environment variable. There must be a way to define the solr home without it... not sure how or when this should be the case but this is a hard issue I think since many deploy their xwiki in a place where the startup variable are not even touchable.

   May be we can include the solr configuration inside WEB-INF of  xwiki enterprise and pick it up from there.

...

The readme says to copy two jars, but they have the same name (one of them should be with search-solr).

   Sorry, its my mistake. Am going to rectify it.

...

It is excellent that your readme works with an identified version of Lucene and one of SOLR, it has often been the case that identified versions of solr worked with a privately crafted lucene version.

...

I am not sure all the packages you include are needed, e.g. why is lucene-spatial useful? why is spellchecker useful? is solr-velocity really useful? (isn't that a completely different velocity run method?) why is solr-cell (the document attachments' parser I think) not useful?

  I haven't spent much time here. The developed component can be considered as  Proof Of Concept with SOLR. It needs to be    refined and standardized.

...

You have omitted the expansion of solr_config.tar.gz in your readme at the right place. I note that it'd be highly wishable to separate solr-home/conf (typically inside WEB-INF/solr) and solr-home/index (typically inside work/solr) but it may be something hard to achieve.

   Yes, I have that in mind. I need to spend more time on how XE is build, how the artifacts and configuration files are picked     up.

...

I had a slightly difficult time to make it all work at the start with the view code indicated in the readme. Too many things to consider for a first time (sensible defaults should just work, the fields should be prefilled, also, I think the form should get and not post so that search results are bookmarkable).

Also, I think you did not include the highlighting in the current search view code, but that's ok for now. Also, you haven't indicated in the readme that the wiki should be multilingual for all the tests.

  Sorry for missing that out .

...

But all in all it is quite good and it worked for me.

So I am happy with this first contribution which shows that it's possible for you to get something workable in short time. There remains an amount of issues which should all be addressed when (and if) the project starts.

I have a rather good feeling.

paul

Thanks once again for your feedback. As the implementation was done in haste, I couldn't follow all the XWiki standards. I was rather concentrating on getting a basic solr search working. As we have something in place now, I would like to refactor the existing code with your comments here and start with it this summer.

Thanks,

   Savitha.

...

Le 6 avr. 2012 à 18:47, savitha sundaramurthy a écrit :

...

Thanks Ludovic and Guillaume.

As Fabio pointed out changing the instantiation of SolrServer(), new EmbeddedServer() , or  CommonHttpServer() will do. If any case, there is something else to offer I'll make it into two different components.

Am really looking forward to work with XWiki community.

Thanks , Savitha

On Thu, Apr 5, 2012 at 5:43 AM, Ludovic Dubost <[email protected]> wrote:

...

Hi Savitha,

This is quite cool. I'm quite impressed at the speed you learned to do components and manage to do a first basic implementation. It indeed seems to show that SOLR is a good choice.

I've one remark which is the usage of a SOLR Embedded server. I think one of the interesting features of SOLR is also that we can split the search engine which the XWiki engine. It would be great to have the choice between both solutions (embedded and not embedded). By default we would bundle the embedded version and for high performance installation it should be possible to separate the SOLR engine and still share most of the logic.

Keep up the good work

Ludovic

2012/4/5 savitha sundaramurthy <[email protected]>

...

Hi Paul,

    Here is the link to the document

https://docs.google.com/document/pub?id=1-2GqvjtzS_a8TRKDNsom9egDZHTT5xIzUf1...

...

...

Thanks a lot, Savitha

...

On Wed, Apr 4, 2012 at 12:03 PM, Paul Libbrecht <[email protected]> wrote:

> Hi Savitha, > > things are moving. This is good. > > Do you have a (temporary) server address where this could be tested? > (maybe best over IRC?). > Or, how could I build your code and inject it on a fresh XWiki? > > Le 4 avr. 2012 à 18:46, savitha sundaramurthy a écrit : >>        I went through Fabio Mancinelli's code and with the help of few >> inputs from there , I have implemented a basic  search functionality > using >> solr. It also fixes the issue

http://jira.xwiki.org/browse/XWIKI-6226.

...

I

...

>> have indexed the page in English, french and Spanish. I have implemented >> these main functionality: >> >> 1) Simple text search using solr. >> 2) Used *Extended Dismax Parser *to customize search relevancy using > boost >> index. > > which fixes well the apostrophe case for sure (and a zillion other > ambiguities, and another zillion power user tricks). > >> 3) Hit highlighting for English. > > This is a good usage of solr. > >> Below is the link to the source code >> https://github.com/savis/xwiki-platform-search >> XWiki front end , HTML and Velocity code >> >> https://gist.github.com/2295648 >> >> I wanted to send a pull request but I was playing around when renaming > my >> username and deleted the forked repository and I'm having difficulty >> forking it again. So I have created a new repository and shared the code >> above. > > I think forking is really useful. If not today, it will be needed any day > soon. > We need to see the diffs. > Best would be to create the fork again, then copy your files and let git > give us the diff. > >> I have taken few screen shots and have attached the document. > > where? > >> I'm working on my application and it should be done today. > > What is the planned schedule? >

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- best regards, Savitha.s _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- Thomas Mortagne