On 29 Aug 2018, at 11:20, Vincent Massol <[email protected]> wrote:
[snip]
Objectives/KPIs/Metrics for STAMP ===========================
The STAMP project has defined 9 KPIs that all partners (and thus XWiki) need to work on:
1) K01: Increase test coverage * Global increase by reducing by 40% the non-covered code. For XWiki since we’re at about 70%, this means reaching about 80% before the end of STAMP (ie. before end of 2019) * Increase the coverage contributions of each tool developed by STAMP.
Strategy: * Primary goal: ** Increase coverage by executing Descartes and improving our tests. This is http://markmail.org/message/ejmdkf3hx7drkj52 ** Don’t do anything with DSpot. I’ll do that part. Note that the goal is to write a Jenkins pipeline to automatically execute DSpot from time to time and commit the generated tests in a separate test source and have our build execute both src/test/java and this new test source.
Contrary to what was proposed initially, it would be nice to run DSpot too. FTR a good command line to use for DSpot is: java -jar <path>/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar --path-to-properties dspot.properties --verbose --generate-new-test-class --with-comment The --generate-new-test-class tells DSpot to generate in its output dir only the new tests added and not include existing tests. The --with-comment tells DSpot to keep the comments and thus the license header too I did a session today and committed the results in https://github.com/STAMP-project/dspot-usecases-output/commit/113726c0aac3af... What I did: * For each module tested with DSpot create a folder in https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki * For cases where DSpot could generate some tests, commit them and modify the pom.xml so that they are executed * Note: tests need to have their license headers adjusted so that they don’t fail the build * Computed coverage + mutation scores before and after and reported in the README.md in each folder Thanks -Vincent
** Don’t do anything with TestContainers FTM since I need to finish a first working version. I may need help in the future to implement docker images for more configurations (on Oracle, in a cluster, with LibreOffice, with an external SOLR server, etc). ** For EvoCrash: We’ll count contributions of EvoCrash to coverage in K08. * Secondary goal: ** Increase our global TPC as mentioned above by fixing the modules in red.
2) K02: Reduce flaky tests. * Objective: reduce the number of flaky tests by 20%
Strategy: * Record flaky tests in jira * Fix the max number of them
3) K03: Better test quality * Objective: increase mutation score by 20%
Strategy: * Same strategy as K01.
[snip] Thanks -Vincent