12 Sep
2014
12 Sep
'14
10:39 a.m.
On Fri, Sep 12, 2014 at 9:43 AM, vincent(a)massol.net <vincent(a)massol.net> wrote:
Hi devs,
I think it’s high time we start to control our performances (page load time, memory
consumption, scalability).
Here’s what I envision:
* At each release (milestone) we get a report of the following (precise details to be
defined together):
** page load times
"page load times" alone is not very clear, should probably be named
"page reloading/refresh time" since loading a page for the very first
time give very different results than reloading it (very variable
stuff put in cache including the page itself if it did not already
ended up in the cache before even being explicitly loaded for some
reason)
*** empty page with “get” action (ie no skin)
*** empty page with “view” action
*** Dashboard.WebHome with “view” action
*** Main.WebHome with "view" action, even if it's mostly the same as
Dashboard.WebHome what we really care about is Main.WebHome because
that's the first page a user see
*** Main.WebHome with "get" action, when we want to test the
performance of a specific page 'get' action is more interesting IMO as
it's not polluted by UI performances variations
*** <others?>
** memory consumption
*** <scenarios to be defined>
*** memory after jetty startup
*** memory after full SOLR index of a 200 (to be defined) wikis farm
(will also catch many possible memory leaks)
Accurate memory test on a simple page load is hard I think since a lot
may be happening in the background et the same time but refreshing a
page 1000 times may be interesting to catch memory leaks.
** scalability
*** max TPS (transaction per second) with a defined page navigation scenario, injected
with lots of users and see how the TPS evolves with # of users
*** <precise scenarios to be defined>
*** access the first page with various number of wikis
*** time spend by SOLR to fully index a 200 (to be defined) wikis farm
*** time spend by SOLR to check that there is nothing to index at
startup in a 200 (to be defined) wikis farm
*** copy a standard subwiki
*** delete a page containing various number of attachments with trash
*** delete a page containing various number of without trash
*** display history of a page having various number of versions
*** load an older version of a page having various number of versions
* Store the results on http://xwiki.org to be able to compare them against past releases
and see how it’s progressing
Note that having comparable speed results when you test 1 year after
is not very easy (it's not easy to keep a perfectly identical
environment) so we might want to store comparison with last "super
stable" release for example instead of raw times (which mean redo the
tests for the previous version before doing those for the new released
version).
Note that Thomas started by installing the jenkins performance plugin with a jmeter job
on ci.xwiki.org but it’s not enough and more importantly it’s not used. What we need is a
report at each release with a quick analysis of how it compares with past releases and
some suggested actions.
The job contain statistics for loading all the pages present in a
standard jetty/hsqldb XE instance 10 times with both 'view' and 'get'
actions.
WDYT?
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Thomas Mortagne