4 Sep
2018
4 Sep
'18
4:03 p.m.
On 29 Aug 2018, at 11:20, Vincent Massol
<vincent(a)massol.net> wrote:
[snip]
Objectives/KPIs/Metrics for STAMP
===========================
The STAMP project has defined 9 KPIs that all partners (and thus XWiki) need to work on:
1) K01: Increase test coverage
* Global increase by reducing by 40% the non-covered code. For XWiki since we’re at about
70%, this means reaching about 80% before the end of STAMP (ie. before end of 2019)
* Increase the coverage contributions of each tool developed by STAMP.
Strategy:
* Primary goal:
** Increase coverage by executing Descartes and improving our tests. This is
http://markmail.org/message/ejmdkf3hx7drkj52
** Don’t do anything with DSpot. I’ll do that part. Note that the goal is to write a
Jenkins pipeline to automatically execute DSpot from time to time and commit the generated
tests in a separate test source and have our build execute both src/test/java and this new
test source.
Contrary to what was proposed initially, it would be nice to run DSpot too.
FTR a good command line to use for DSpot is:
java -jar <path>/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar --path-to-properties
dspot.properties --verbose --generate-new-test-class --with-comment
The --generate-new-test-class tells DSpot to generate in its output dir only the new tests
added and not include existing tests.
The --with-comment tells DSpot to keep the comments and thus the license header too
I did a session today and committed the results in
https://github.com/STAMP-project/dspot-usecases-output/commit/113726c0aac3a…
What I did:
* For each module tested with DSpot create a folder in
https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki
* For cases where DSpot could generate some tests, commit them and modify the pom.xml so
that they are executed
* Note: tests need to have their license headers adjusted so that they don’t fail the
build
* Computed coverage + mutation scores before and after and reported in the README.md in
each folder
Thanks
-Vincent
** Don’t do anything with TestContainers FTM since I
need to finish a first working version. I may need help in the future to implement docker
images for more configurations (on Oracle, in a cluster, with LibreOffice, with an
external SOLR server, etc).
** For EvoCrash: We’ll count contributions of EvoCrash to coverage in K08.
* Secondary goal:
** Increase our global TPC as mentioned above by fixing the modules in red.
2) K02: Reduce flaky tests.
* Objective: reduce the number of flaky tests by 20%
Strategy:
* Record flaky tests in jira
* Fix the max number of them
3) K03: Better test quality
* Objective: increase mutation score by 20%
Strategy:
* Same strategy as K01.
[snip]
Thanks
-Vincent