25 Aug
2018
25 Aug
'18
8:41 p.m.
Hi devs,
On 30 Jul 2018, at 12:39, Vincent Massol
<vincent(a)massol.net> wrote:
Hi devs,
It would be great if you could help improve our unit tests using Descartes. This is
needed for the STAMP research project (https://www.stamp-project.eu/view/main/) and will
benefit XWiki by having 2 effects:
* increasing the test coverage
* improving the tests themselves (increasing their mutation score)
Since 10.7 is 50% testing and 50% BFD, it would be great if you could spend all or a
substantial part of your testing time working on this.
Just thought about something. In order to know if Descartes (and other STAMP tools) are
interesting in the longer run, there’s one more parameter that I think we should measure
during our test sessions described below: the time it took to improve by N% the coverage
and P% the mutation score.
Because, improving the TPC of a module is not an enough data by itself IMO. Imagine that
we have 1 free hour to work on testing. We could choose to do:
* Option 1: write a new test for some code that has not been tested and that currently has
0% coverage and bring it to, say, 20%
* Option 2: use Descartes to help improve the quality of an existing tests and improve the
test coverage by, say, 2%
If option 1 takes 2 hours and option 2 takes 5 minutes then option 2 might be more
interesting. If OTH option 1 takes 1 hour and option 2 takes 15mn then option 1 is more
interesting by far.
Yes, in order to compare we would need to know also the time it takes to write new tests.
However I think that some order of magnitude for using Descartes could already provide
interesting info to us.
Thus, if you agree if would be nice to also record this in the test data we record at
https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki in each
session.
I propose the following strategy:
* You find a module you want to work on.
* In that module you run: mvn clean install -Pquality -Dxwiki.pitest.skip=false
* Then you check target/pit-reports/<date>/issues/index.html and verify if there
are "pseudo tested" methods listed (when we have finished fixing all of those we
can move to “partially tested methods”).
* If there are some, then please record the current jacoco threshold and the current
mutation score.
* You can get the jacoco threshold by running "mvn clean install -Pquality
-Dxwiki.pitest.skip=false -Dxwiki.pitest.mutationThreshold=100” (or by checking
target/pit-reports/<date>/index.html, I haven’t checked yet if they are the same).
* You can get the current mutation score by checking
target/pit-reports/<date>/index.html
* Also record the current time
* Then fix the test so that Descartes doesn’t report
any pseudo tested or partially tested methods
* Record the time again
* Update the jacoco threshold and the mutation scores
in the pom.xml
* Send a PR on
https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki using the
format already defined there.
* And include the time in the PR.
WDYT?
Thanks
-Vincent
PS: I’ll try to roughly estimate the times it took me to do the test sessions already in
https://github.com/STAMP-project/descartes-usecases-output/tree/master/xwiki
WDYT? Doable?
Thanks
-Vincent