Re: [xwiki-users] managing very large amount of data

Hi Yuichiro EBIHARA, Yuichiro a écrit : Nice this sounds like an ambitious project... Are you talking about 15000000 pages ? Each page would have how many additional meta data fields ? Our largest system is currently Curriki.org.. It's more in the 100K of pages but with large pages (they have many XWiki objects). There is also a big volume 20GB. We have many installations running with that size (mostly because of attachments). You absolutely need to implement custom mapping from your structures. Otherwise it will be too big. This is a good question. I'm not a mysql specialist and 15 million rows is a lot. Postgres might be a good candidate too for that volume. You should search on the Internet. Depending on your confidentially needs, maybe an XWiki storage implementation over Google App Engine could be a good candidate. It also depends on how much time/spending you have for this application. Anything that is transparent to the application and that allows good scalability is good There are some things you need to absolutely avoid like showing in one page all the data in a space for example or running a search with rights on a common term. That will clearly not work. and just calling it once could bring down your install. You might want to write a custom rights manager to disable rights altogether if it is an option. I would also suggest that if you have some budget for this project you look into having some support to do it right. Initial choices could be tough to fix with that amount of data. (XWiki SAS provides support http://www.xwiki.com or other experimented developers on this list might be able to help) Ludovic -- Ludovic Dubost Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost