27 Sep
2014
27 Sep
'14
12:18 a.m.
+1 for having some visibility, defining performance metrics and generating
(at least) release-based performance reports for XWiki.
ATM, XWiki`s performance is "black magic"/"taboo" for me, since nobody
is
measuring it (until Thomas started working specifically on it), so we can`t
really talk about it or improve it otherwise.
Thanks,
Eduard
On Fri, Sep 12, 2014 at 11:39 AM, Thomas Mortagne <thomas.mortagne(a)xwiki.com
wrote:
> On Fri, Sep 12, 2014 at 9:43 AM, vincent(a)massol.net <vincent(a)massol.net>
wrote:
> > Hi devs,
> >
> > I think it’s high time we start to control our performances (page load
> time, memory consumption, scalability).
> >
> > Here’s what I envision:
> >
> > * At each release (milestone) we get a report of the following (precise
> details to be defined together):
> >
> > ** page load times
>
> "page load times" alone is not very clear, should probably be named
> "page reloading/refresh time" since loading a page for the very first
> time give very different results than reloading it (very variable
> stuff put in cache including the page itself if it did not already
> ended up in the cache before even being explicitly loaded for some
> reason)
>
> > *** empty page with “get” action (ie no skin)
> > *** empty page with “view” action
> > *** Dashboard.WebHome with “view” action
>
> *** Main.WebHome with "view" action, even if it's mostly the same as
> Dashboard.WebHome what we really care about is Main.WebHome because
> that's the first page a user see
> *** Main.WebHome with "get" action, when we want to test the
> performance of a specific page 'get' action is more interesting IMO as
> it's not polluted by UI performances variations
>
> > *** <others?>
> >
> > ** memory consumption
> > *** <scenarios to be defined>
>
> *** memory after jetty startup
> *** memory after full SOLR index of a 200 (to be defined) wikis farm
> (will also catch many possible memory leaks)
>
> Accurate memory test on a simple page load is hard I think since a lot
> may be happening in the background et the same time but refreshing a
> page 1000 times may be interesting to catch memory leaks.
>
> >
> > ** scalability
> > *** max TPS (transaction per second) with a defined page navigation
> scenario, injected with lots of users and see how the TPS evolves with # of
> users
> > *** <precise scenarios to be defined>
>
> *** access the first page with various number of wikis
> *** time spend by SOLR to fully index a 200 (to be defined) wikis farm
> *** time spend by SOLR to check that there is nothing to index at
> startup in a 200 (to be defined) wikis farm
> *** copy a standard subwiki
> *** delete a page containing various number of attachments with trash
> *** delete a page containing various number of without trash
> *** display history of a page having various number of versions
> *** load an older version of a page having various number of versions
>
> >
> > * Store the results on http://xwiki.org to be able to compare them
> against past releases and see how it’s progressing
>
> Note that having comparable speed results when you test 1 year after
> is not very easy (it's not easy to keep a perfectly identical
> environment) so we might want to store comparison with last "super
> stable" release for example instead of raw times (which mean redo the
> tests for the previous version before doing those for the new released
> version).
>
> >
> > Note that Thomas started by installing the jenkins performance plugin
> with a jmeter job on ci.xwiki.org but it’s not enough and more
> importantly it’s not used. What we need is a report at each release with a
> quick analysis of how it compares with past releases and some suggested
> actions.
>
> The job contain statistics for loading all the pages present in a
> standard jetty/hsqldb XE instance 10 times with both 'view' and
'get'
> actions.
>
> >
> > WDYT?
> >
> > Thanks
> > -Vincent
> > _______________________________________________
> > devs mailing list
> > devs(a)xwiki.org
> > http://lists.xwiki.org/mailman/listinfo/devs
>
>
>
> --
> Thomas Mortagne
> _______________________________________________
> devs mailing list
> devs(a)xwiki.org
> http://lists.xwiki.org/mailman/listinfo/devs
>