4 Nov
2014
4 Nov
'14
7:34 a.m.
Hello all,
After your feedback I made this draft with QA Stats:
http://dev.xwiki.org/xwiki/bin/view/Drafts/QA+Stats
I await your reply and feedback regarding the draft and stats.
Thank you.
On Thu, Oct 2, 2014 at 3:29 PM, vincent(a)massol.net <vincent(a)massol.net>
wrote:
Hi Andrea,
On 2 Oct 2014 at 10:01:17, Andreea Popescu (andreea.popescu(a)xwiki.com
(mailto:andreea.popescu@xwiki.com)) wrote:
--
Andreea Zenovia Popescu
QA Engineer @XWiki SAS
Hello everyone,
If you’re wondering what kept the QA team busy during the last few days,
What QA team, there’s no such team/role in the XWiki project ATM :)
I think yo meant the XWiki SAS QA Team. Since I’m also from XWiki SAS I
can explain a bit more about this team:
- it’s globally in charge of ensuring the quality of the XWiki releases
from the point of view of XWiki SAS clients (for example some XWiki SAS
clients are using Oracle and this team takes special care to ensure there’s
no regression on Oracle, same for browser versions, and globally for
anything that could impact XWiki SAS clients)
- XWiki SAS is delegating Andrea full time (and Manuel 1/3rd of this time)
to help the xwiki.org project by doing manual testing of XWiki releases.
They are maintaining the http://test.xwiki.org wiki and filling the
manual testing part of the Release notes.
the answer is a lot of testing and a
brainstorming effort in order to
devise a series of statistics for the product, which would be featured
with
the product version 6.2.
cool
We believe that the statistics ought to be
implemented for the following
reasons:
a) They would allow for better communication between us and the community
You’re not community? :) Personally I’ve always considered you community
like anyone else contributing something to the xwiki project!
b) Each version would be more easily evaluated
this way
c) The product’s evolution would be more easily observable by tracking
the
changes in these statistics from one version to
another
d) A more complete overview on the long-term evolution of the product
would be possible, with an opportunity to analyze a series of items in
detail
Yes what’s important are not the raw values, but the trends between
versions.
I started working on this here:
http://dev.xwiki.org/xwiki/bin/view/Community/ProjectHealth
Thomas also recently started working on performance stats here:
http://test.xwiki.org/xwiki/bin/view/Performances/Jetty+HSQLDB+single+wiki
See also this mail thread on the topic of having regular perf reports at
each release:
http://markmail.org/message/dprnorr37ox7tvfu
IMO this needs to be included in the page you’re going to create for each
release.
See below for more on that.
We consider some of the above-mentioned items to
be quite important. Here
is a list of the most important ones:
- No. of downloads
- No. of active installs
- No. of tests executed / added
Not very easy to compute but I know how to do it if you need help on that.
- Jira issues fixed - by resolution, by priority,
by type
- Jira issues opened - by priority, by type
- Closed vs open tickets
- Stats for important tags: e.g. ie10, mobile, flamingo
- Stats for major features: e.g. flamingo, extension manager, solr
- Top overall issues reporters
- Top non-XWiki SAS issues reporters
- Extensions quality: issues reported (top 10 extensions)
- Extensions quality: issues closed (top 10 extensions)
- l10n translations: existent, missing
- Performance stats
Yes!
Therefore, your opinion on the following issues
would be of great use to
us:
a) The list offered a number of items that we deem important for our
measurements and assessments. We would like to know whether you agree
with
the list and we would like to have your opinion
regarding other items
that
we would consider inserting in the list?
Just to stress it again, what’s important is to get the figure for the
past releases and compare so that we get trends over several releases and
see where we’re going.
- Global TPC is also interesting to compute and see the evolution. Sorin
started doing this in the past (he did it once only unfortunately ;)). I
can also explain how to compute this.
Now you don’t have to start with 100 metrics. Just 3-4 is enough. It’s
more important to send the reports regularly and have the trends for each
metrics than to have too many metrics.
Also, what’s important is to analyze them. There’s no point in having the
metrics and not doing anything about them. So we’ll need to think about
that too but we can decide after.
b) Where would you like us to publish the
statistics and the conclusions
that would result from their analysis? We were thinking about the
following:
- On a ‘Project Health’ page
- On a page designated to each product version (each version would have
its own page with statistics)
- In the test.xwiki.org
- Within a blogpost
- A mix of the above
Here’s my POV:
1)
* I’d really like you to continue the page I started at
http://dev.xwiki.org/xwiki/bin/view/Community/ProjectHealth but to morph
it
* I see a ProjectHealth space on dev.xwiki.org
* On the home page of the ProjectHealth space, show all the trends across
the years or across the versions + have a Livetable listing specific
reports for each version analyzed
* The raw data should be stored in the pages or in xobjects (they needs to
be stored on xwiki.org that’s the important part)
2) Once a report page is ready for a version, ask for feedback about it on
the devs list
3) After the feedback has been incorporated or after a few days without
answer, add a blog post on xwiki.org pointing to the page
4) Tweet about it with the xwiki.org account
We await your reply and feedback regarding the
viability and usefulness
of
our proposal.
Great, let’s do it and we’ll tune the details as we progress!
Thanks
-Vincent
Thank you for your time
--
Andreea Zenovia Popescu