[xwiki-devs] Creating a lot of subwikis
Hi, I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link: http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html I'll be happy to have feedback on that. For example, I would like to know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project. Hope this helps. Sincerely, -- Jean
Hi, Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ? Have fun! Guillaume Fenollar 2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link: http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like to know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi, There is also this issue related to 'Very slow creation time for workspaces or subwikis on MySQL' http://jira.xwiki.org/browse/XWIKI-8245 Not sure it's important but I'd be curious what options you have selected during the creation process (a.k.a you are creating 'old workspaces' (with global users) or wikis with local users). Anyway maybe this kind of tests should be done after we finish the wikis improvements. Thanks, Caty On Fri, Dec 13, 2013 at 11:54 AM, Guillaume Fenollar < [email protected]> wrote:
Hi,
Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ?
Have fun!
Guillaume Fenollar
2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link: http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like to know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi, Thanks for the JIRA report. However, I think there is no link with that since I use the 5.3 which provide a new way to creating subwikis not workspaces anymore (I'm still new in XWiki so correct me if I'm wrong). For the subwikis creation, I did not select particular option for the creation (just use the createWiki function) so I suppose the default configuration applies in this case. However, I will use in the project the following options: - global users - use a template - different kind of visibility depending on the wiki (open, request or invite) I'll probably do further tests later with template loading and a 'true' DB management system and come back with other data. Thanks -- Jean On ven., 2013-12-13 at 12:11 +0200, Ecaterina Moraru (Valica) wrote:
Hi,
There is also this issue related to 'Very slow creation time for workspaces or subwikis on MySQL' http://jira.xwiki.org/browse/XWIKI-8245 Not sure it's important but I'd be curious what options you have selected during the creation process (a.k.a you are creating 'old workspaces' (with global users) or wikis with local users).
Anyway maybe this kind of tests should be done after we finish the wikis improvements.
Thanks, Caty
On Fri, Dec 13, 2013 at 11:54 AM, Guillaume Fenollar < [email protected]> wrote:
Hi,
Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ?
Have fun!
Guillaume Fenollar
2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link: http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like to know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On 13 Dec 2013 at 10:54:38, Guillaume Fenollar ([email protected]) wrote: Hi, Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ? I’ve reproduced the issue. It’s a problem of the context user which gets reset to null. I think it’s a multithreading bug somewhere in our code. Thanks -Vincent Have fun! Guillaume Fenollar 2013/12/12 Jean SIMARD <[email protected]>
Hi, I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link: http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html I'll be happy to have feedback on that. For example, I would like to know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project. Hope this helps. Sincerely, -- Jean
Hi, I didn't look well but I can say one thing. In the first method, when the around-250 subwikis are reached, the log is displaying errors for the next few minutes without stopping. As discussed with Thomas, we may think this is OutOfMemory error due to HSQLDB but it's probably not because of the second method that show that I can create more than 600 subwikis (the second method is more likely to be an OutOfMemory error looking the time evolution, I really should have look more carefully the logs!). I may try later on a MySQL server. Thanks, -- Jean On ven., 2013-12-13 at 10:54 +0100, Guillaume Fenollar wrote:
Hi,
Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ?
Have fun!
Guillaume Fenollar
2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link: http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like to know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
If you wanna test only XWiki, you should consider using MariaDB or even Percona instead of MySQL (so you can lower your chance being limited by database). MariaDB and Percona both uses XtraDB (improved InnoDB by Percona), but Percona should be more "stable" in such stress tests. 2013/12/13 Jean SIMARD <[email protected]>
Hi,
I didn't look well but I can say one thing. In the first method, when the around-250 subwikis are reached, the log is displaying errors for the next few minutes without stopping. As discussed with Thomas, we may think this is OutOfMemory error due to HSQLDB but it's probably not because of the second method that show that I can create more than 600 subwikis (the second method is more likely to be an OutOfMemory error looking the time evolution, I really should have look more carefully the logs!). I may try later on a MySQL server.
Thanks, -- Jean
On ven., 2013-12-13 at 10:54 +0100, Guillaume Fenollar wrote:
Hi,
Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ?
Have fun!
Guillaume Fenollar
2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link: http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like to know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- S pozdravem Lukáš Raška
Hi, I'll probably need to make it work on a MySQL for the final project but I keep MariaDB and Percona (I didn't know this one) in mind for other projects, thanks :-) -- Jean On ven., 2013-12-13 at 12:50 +0100, Lukáš Raška wrote:
If you wanna test only XWiki, you should consider using MariaDB or even Percona instead of MySQL (so you can lower your chance being limited by database).
MariaDB and Percona both uses XtraDB (improved InnoDB by Percona), but Percona should be more "stable" in such stress tests.
2013/12/13 Jean SIMARD <[email protected]>
Hi,
I didn't look well but I can say one thing. In the first method, when the around-250 subwikis are reached, the log is displaying errors for the next few minutes without stopping. As discussed with Thomas, we may think this is OutOfMemory error due to HSQLDB but it's probably not because of the second method that show that I can create more than 600 subwikis (the second method is more likely to be an OutOfMemory error looking the time evolution, I really should have look more carefully the logs!). I may try later on a MySQL server.
Thanks, -- Jean
On ven., 2013-12-13 at 10:54 +0100, Guillaume Fenollar wrote:
Hi,
Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ?
Have fun!
Guillaume Fenollar
2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link: http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like to know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi, Also, keep in mind that, with the code you`re using, you`re creating empty wikis. In practice, when using (copying) a template or a flavour/extension (future feature done with the Extension Manager), your wiki creation times will grow considerably. Basically all you are testing now is the creation of databases and a document (the wiki descriptor). Each empty wiki practically consists of these 2 elements: 1) the database 2) the descriptor. Of course, when creating the database, some basic wiki classes are also created automatically, so you can add them to the count :) Anyway, nice experiment! Thanks for sharing! Would be nice to see more of these on XWiki's features. Cheers, Eduard On Fri, Dec 13, 2013 at 4:19 PM, Jean SIMARD <[email protected]> wrote:
Hi,
I'll probably need to make it work on a MySQL for the final project but I keep MariaDB and Percona (I didn't know this one) in mind for other projects, thanks :-)
-- Jean
On ven., 2013-12-13 at 12:50 +0100, Lukáš Raška wrote:
If you wanna test only XWiki, you should consider using MariaDB or even Percona instead of MySQL (so you can lower your chance being limited by database).
MariaDB and Percona both uses XtraDB (improved InnoDB by Percona), but Percona should be more "stable" in such stress tests.
2013/12/13 Jean SIMARD <[email protected]>
Hi,
I didn't look well but I can say one thing. In the first method, when the around-250 subwikis are reached, the log is displaying errors for the next few minutes without stopping. As discussed with Thomas, we may think this is OutOfMemory error due to HSQLDB but it's probably not because of the second method that show that I can create more than 600 subwikis (the second method is more likely to be an OutOfMemory error looking the time evolution, I really should have look more carefully the logs!). I may try later on a MySQL server.
Thanks, -- Jean
On ven., 2013-12-13 at 10:54 +0100, Guillaume Fenollar wrote:
Hi,
Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ?
Have fun!
Guillaume Fenollar
2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link:
http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like
to
know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
That's right. I was also thinking about the time for templates loading. However, it seems that the evolution of time in my experiments is exponential so it will not make it worse I suppose ^_^ I'll just reduce the maximum number possibly by a hundred which will not really make a difference for me. However, thanks for noticing and thanks for the info about descriptors and database. -- Jean On ven., 2013-12-13 at 16:57 +0200, Eduard Moraru wrote:
Hi,
Also, keep in mind that, with the code you`re using, you`re creating empty wikis. In practice, when using (copying) a template or a flavour/extension (future feature done with the Extension Manager), your wiki creation times will grow considerably. Basically all you are testing now is the creation of databases and a document (the wiki descriptor). Each empty wiki practically consists of these 2 elements: 1) the database 2) the descriptor. Of course, when creating the database, some basic wiki classes are also created automatically, so you can add them to the count :)
Anyway, nice experiment! Thanks for sharing!
Would be nice to see more of these on XWiki's features.
Cheers, Eduard
On Fri, Dec 13, 2013 at 4:19 PM, Jean SIMARD <[email protected]> wrote:
Hi,
I'll probably need to make it work on a MySQL for the final project but I keep MariaDB and Percona (I didn't know this one) in mind for other projects, thanks :-)
-- Jean
On ven., 2013-12-13 at 12:50 +0100, Lukáš Raška wrote:
If you wanna test only XWiki, you should consider using MariaDB or even Percona instead of MySQL (so you can lower your chance being limited by database).
MariaDB and Percona both uses XtraDB (improved InnoDB by Percona), but Percona should be more "stable" in such stress tests.
2013/12/13 Jean SIMARD <[email protected]>
Hi,
I didn't look well but I can say one thing. In the first method, when the around-250 subwikis are reached, the log is displaying errors for the next few minutes without stopping. As discussed with Thomas, we may think this is OutOfMemory error due to HSQLDB but it's probably not because of the second method that show that I can create more than 600 subwikis (the second method is more likely to be an OutOfMemory error looking the time evolution, I really should have look more carefully the logs!). I may try later on a MySQL server.
Thanks, -- Jean
On ven., 2013-12-13 at 10:54 +0100, Guillaume Fenollar wrote:
Hi,
Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ?
Have fun!
Guillaume Fenollar
2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link:
http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like
to
know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
FTR I’m still investigating the problem. The issue is that at some point (which can happen after 5 wiki creation, after 50, after 150, or at any time) the storage is not able to load the xwiki:XWiki.Admin page from the DB. It returns a new document which is then cached empty. Then when the rights checks are performed for the current user (which is Admin) the code gets the Admin document from the cache and since it doesn’t contain any XWikiUsers object, the permission is refused. I’m now trying to investigate in the HSQLDB code why this happens but it’s extremely difficult since it doesn’t always happen. My main suspect is the wrong schema being used but I need to understand if this is the case and more importantly why the schema would be set wrongly suddenly (possibly some multithreading aspect). Thanks -Vincent On 16 Dec 2013 at 08:51:24, Jean SIMARD ([email protected]) wrote: That's right. I was also thinking about the time for templates loading. However, it seems that the evolution of time in my experiments is exponential so it will not make it worse I suppose ^_^ I'll just reduce the maximum number possibly by a hundred which will not really make a difference for me. However, thanks for noticing and thanks for the info about descriptors and database. -- Jean On ven., 2013-12-13 at 16:57 +0200, Eduard Moraru wrote:
Hi,
Also, keep in mind that, with the code you`re using, you`re creating empty wikis. In practice, when using (copying) a template or a flavour/extension (future feature done with the Extension Manager), your wiki creation times will grow considerably. Basically all you are testing now is the creation of databases and a document (the wiki descriptor). Each empty wiki practically consists of these 2 elements: 1) the database 2) the descriptor. Of course, when creating the database, some basic wiki classes are also created automatically, so you can add them to the count :)
Anyway, nice experiment! Thanks for sharing!
Would be nice to see more of these on XWiki's features.
Cheers, Eduard
On Fri, Dec 13, 2013 at 4:19 PM, Jean SIMARD <[email protected]> wrote:
Hi,
I'll probably need to make it work on a MySQL for the final project but I keep MariaDB and Percona (I didn't know this one) in mind for other projects, thanks :-)
-- Jean
On ven., 2013-12-13 at 12:50 +0100, Lukáš Raška wrote:
If you wanna test only XWiki, you should consider using MariaDB or even Percona instead of MySQL (so you can lower your chance being limited by database).
MariaDB and Percona both uses XtraDB (improved InnoDB by Percona), but Percona should be more "stable" in such stress tests.
2013/12/13 Jean SIMARD <[email protected]>
Hi,
I didn't look well but I can say one thing. In the first method, when the around-250 subwikis are reached, the log is displaying errors for the next few minutes without stopping. As discussed with Thomas, we may think this is OutOfMemory error due to HSQLDB but it's probably not because of the second method that show that I can create more than 600 subwikis (the second method is more likely to be an OutOfMemory error looking the time evolution, I really should have look more carefully the logs!). I may try later on a MySQL server.
Thanks, -- Jean
On ven., 2013-12-13 at 10:54 +0100, Guillaume Fenollar wrote:
Hi,
Good initiative. Have you checked the logs? Are you sure the database (HSQLDB here) is not the problem? Perhaps you'll want to run the same tests with a more robust database, like postgreSQL or even MySQL ?
Have fun!
Guillaume Fenollar
2013/12/12 Jean SIMARD <[email protected]>
Hi,
I need to create a lot of subwikis in a XWiki instance (5.3-milestone-2) for a project and was interested in the possibilities and limits of XWiki. I tried to make a basic stress test to evaluate a maximum number of subwikis that XWiki can handle. I wrote a small report and provide files used on this link:
http://hole.tuziwo.info/stress-test-xwiki-for-subwikis-creation.html
I'll be happy to have feedback on that. For example, I would like
to
know if my first method do something bad or if this is a possible bug in XWiki. If this is a bug, I hope this can be improved since I think that I will need more than 1000 subwikis in my project.
Hope this helps. Sincerely, -- Jean
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
participants (6)
-
Ecaterina Moraru (Valica) -
Eduard Moraru -
Guillaume Fenollar -
Jean SIMARD -
Lukáš Raška -
vincent＠massol.net