[xwiki-devs] Need help on GSoC project Distributed search
Hi all, Please excuse me if you have received this mail before, there seems to be some problem with my gmail account and I am not sure whether the mail I sent earlier was sent to XWiki devs list or not, so sending it again. please help me. """Support multiple installations of XWiki being indexed on one or multiple XWiki Servers and share the index. It should be possible to share easily the index so that multiple XWiki installations can be shared in one UI.""" ---> I am assuming that one of the several servers, is the main server and has information about other servers like the ip address. -- am I correct ? ---> Can each server maintain it's own index(instead of distributed indexing), but while searching, indexes on multiple servers are used. ---> please provide me some more info regarding this. """Also the search UI should be possible to customize depending on Document types.""" ---> Restricting search to specified document type. If any one has other relevant information, please provide it. -- Sai Krishna.
HI saikrishna, May be you should try the IRC channel ? server : irc.freenode.org channel : #xwiki - Asiri On Tue, Mar 25, 2008 at 12:19 AM, <[email protected]> wrote:
Hi all,
Please excuse me if you have received this mail before, there seems to be some problem with my gmail account and I am not sure whether the mail I sent earlier was sent to XWiki devs list or not, so sending it again.
please help me.
"""Support multiple installations of XWiki being indexed on one or multiple XWiki Servers and share the index. It should be possible to share easily the index so that multiple XWiki installations can be shared in one UI.""" ---> I am assuming that one of the several servers, is the main server and has information about other servers like the ip address. -- am I correct ? ---> Can each server maintain it's own index(instead of distributed indexing), but while searching, indexes on multiple servers are used. ---> please provide me some more info regarding this.
"""Also the search UI should be possible to customize depending on Document types.""" ---> Restricting search to specified document type.
If any one has other relevant information, please provide it.
-- Sai Krishna.
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Asiri, Atlast I got a reply :). Yes I have tried IRC earlier, but I didn't get any reply, also very few people very on #xwiki channel, so thought it would better to mail the mailing list. -Sai Krishna. On Tue, 25 Mar 2008, Asiri Rathnayake wrote:
HI saikrishna,
May be you should try the IRC channel ?
server : irc.freenode.org
channel : #xwiki
- Asiri
On Tue, Mar 25, 2008 at 12:19 AM, <[email protected]> wrote:
Hi all,
Please excuse me if you have received this mail before, there seems to be some problem with my gmail account and I am not sure whether the mail I sent earlier was sent to XWiki devs list or not, so sending it again.
please help me.
"""Support multiple installations of XWiki being indexed on one or multiple XWiki Servers and share the index. It should be possible to share easily the index so that multiple XWiki installations can be shared in one UI.""" ---> I am assuming that one of the several servers, is the main server and has information about other servers like the ip address. -- am I correct ? ---> Can each server maintain it's own index(instead of distributed indexing), but while searching, indexes on multiple servers are used. ---> please provide me some more info regarding this.
"""Also the search UI should be possible to customize depending on Document types.""" ---> Restricting search to specified document type.
If any one has other relevant information, please provide it.
-- Sai Krishna.
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- ******************************************************************************* ******************************************************************************* Success makes life easy but not living easy ** Remember Things can be difficult but not impossible
Hi, Don't be surprised you don't get an immediate reply.. Mailing list are asynchronous and it's on purpose. Also it does not help to add my email in CC one or multiple times, since I'm subscribed to the dev and users mailing list, and since the email is send to both the list and directly to me it still ends up in my mailing list folder ! Concerning the distributed search, Yes, one method I have in mind is to register children servers to the main server and have a way to make the children server search pages in reality call the main server search. With the registration, the main server would get authorization to come and either take the index or get updates so that it can build a global index. It could be possible to have an index on each server, but this would make merging results and counting results more complex. I'm not sure also of the performance. Both methods could be studied and/or implemented. Concerning the document type things, it's not important. We can explain at implementation time. Ludovic Sai Krishna wrote:
Hi Asiri,
Atlast I got a reply :). Yes I have tried IRC earlier, but I didn't get any reply, also very few people very on #xwiki channel, so thought it would better to mail the mailing list.
-Sai Krishna.
On Tue, 25 Mar 2008, Asiri Rathnayake wrote:
HI saikrishna,
May be you should try the IRC channel ?
server : irc.freenode.org
channel : #xwiki
- Asiri
On Tue, Mar 25, 2008 at 12:19 AM, <[email protected]> wrote:
Hi all,
Please excuse me if you have received this mail before, there seems to be some problem with my gmail account and I am not sure whether the mail I sent earlier was sent to XWiki devs list or not, so sending it again.
please help me.
"""Support multiple installations of XWiki being indexed on one or multiple XWiki Servers and share the index. It should be possible to share easily the index so that multiple XWiki installations can be shared in one UI.""" ---> I am assuming that one of the several servers, is the main server and has information about other servers like the ip address. -- am I correct ? ---> Can each server maintain it's own index(instead of distributed indexing), but while searching, indexes on multiple servers are used. ---> please provide me some more info regarding this.
"""Also the search UI should be possible to customize depending on Document types.""" ---> Restricting search to specified document type.
If any one has other relevant information, please provide it.
-- Sai Krishna.
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Ludovic Dubost Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost
Hi Ludovic, Sorry for the multiple mails :(. The reason I have CC a copy to you is, somewhere in GSOC discussions , it is suggested to keep all the discussions only in mailing list and better CC it to the mentor. Any ways... coming to the project, Yes, performance might be an issuse when the indexes are over multiple servers. Having index on single machine is much better but only when index size is small. But the problem is, as the number of servers in the cluster increase and also the data on each XWiki server increases, it would take lot of time to copy the index from each children node to main server and also maintaining it would become a problem. Also if indexes are on different machines, remote indexes can be searched in parallel(using an api of lucene, remote searching can be done with RMI). Parallel search becomes extremly helpful as, the size index keeps on growing. Ofcourse each method has it own pros and cons, but there performance varies as the size of index varies. I am thinking of testing those two methods in terms of response time, indexing some data like english wikipedia which about 2.3 million documents. Is this data enough ??? Another doubt I have in mind is, """have a way to make the children server search pages in reality call the main server search.""" What if main server is down ?? ---If the index is only on one main server then searching is not possible until the server is up. ---Even if the indexes are distributed, searching is not possible as children servers are supposed to call main search server Instead what I am thinking is, the main server maintains information about children servers; and passes that information to each of its children, as and when the information about the children servers gets updated in the main server. Now each server has information about other servers and each server can "act" as main server while searching. Therefore even if one server is down, search can be performed on remaining indexes(a benefit of having multiple indexes). Any comments on this ?? --Sai Krishna. On Tue, 25 Mar 2008, Ludovic Dubost wrote:
Hi,
Don't be surprised you don't get an immediate reply.. Mailing list are asynchronous and it's on purpose. Also it does not help to add my email in CC one or multiple times, since I'm subscribed to the dev and users mailing list, and since the email is send to both the list and directly to me it still ends up in my mailing list folder !
Concerning the distributed search,
Yes, one method I have in mind is to register children servers to the main server and have a way to make the children server search pages in reality call the main server search. With the registration, the main server would get authorization to come and either take the index or get updates so that it can build a global index.
It could be possible to have an index on each server, but this would make merging results and counting results more complex. I'm not sure also of the performance.
Both methods could be studied and/or implemented.
Concerning the document type things, it's not important. We can explain at implementation time.
Ludovic
Sai Krishna wrote:
Hi Asiri,
Atlast I got a reply :). Yes I have tried IRC earlier, but I didn't get any reply, also very few people very on #xwiki channel, so thought it would better to mail the mailing list.
-Sai Krishna.
On Tue, 25 Mar 2008, Asiri Rathnayake wrote:
HI saikrishna,
May be you should try the IRC channel ?
server : irc.freenode.org
channel : #xwiki
- Asiri
On Tue, Mar 25, 2008 at 12:19 AM, <[email protected]> wrote:
Hi all,
Please excuse me if you have received this mail before, there seems to be some problem with my gmail account and I am not sure whether the mail I sent earlier was sent to XWiki devs list or not, so sending it again.
please help me.
"""Support multiple installations of XWiki being indexed on one or multiple XWiki Servers and share the index. It should be possible to share easily the index so that multiple XWiki installations can be shared in one UI.""" ---> I am assuming that one of the several servers, is the main server and has information about other servers like the ip address. -- am I correct ? ---> Can each server maintain it's own index(instead of distributed indexing), but while searching, indexes on multiple servers are used. ---> please provide me some more info regarding this.
"""Also the search UI should be possible to customize depending on Document types.""" ---> Restricting search to specified document type.
If any one has other relevant information, please provide it.
-- Sai Krishna.
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- ******************************************************************************* ******************************************************************************* Success makes life easy but not living easy ** Remember Things can be difficult but not impossible
participants (4)
-
Asiri Rathnayake -
Ludovic Dubost -
Sai Krishna -
saikrishnaï¼ students.iiit.ac.in