17 Oct
2018
17 Oct
'18
3:54 p.m.
Hi,
On 17 Oct 2018, at 11:20, Vincent Massol
<vincent(a)massol.net> wrote:
Hi,
[snip]
So actually, after speaking with Benjamin, I’ve realized a few things:
* By default DSpot runs with the PIT selector (PitMutantScoreSelector) which is configured
to use the default PIT mutations. This is why we need to run with the PIT selector but
configured to use the Descartes mutation, and this is done by specifying --descartes.
* Now this will optimize the generation of new tests for their increased mutation score.
Right now we got 0% all the time on our tests (see
https://docs.google.com/spreadsheets/d/1LULpGpsJirmFyvHNstLGv-Gv5DVBdpLTM2h…)
and it’s because we didn’t use --descartes. We need to try again or run on new modules
with --descartes and see what it gives us. It’s possible it’ll generate even less tests…
* For the coverage part, there are 2 other selectors that can be used with DSpot to
generate tests that all increase the coverage:
** "--test-criterion JacocoCoverageSelector": uses jacoco and keep tests that
increase the instruction coverage
** "--test-criterion CloverCoverageSelector”: uses openclover and keep tests that
increase the branch coverage
So we need to test with the various selectors and see what we get.
If we want to get the best values, we should use --descartes for K03 and either jacoco or
clover selector for K01. Now we need to see what tests we get.
Thanks
-Vincent
Process to run DSpot:
1) Pick a module. Measure coverage and mutation score (or take the value there already if
they’re in the pom.xml). Same as for Descartes testing.
2) Run DSpot on the module, see
https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot for explanations
One important detail that I had missed. We need to run Dspot with “—descartes” on the
command line so that it uses Descartes for computing the mutation score for mutations and
only keep tests that increase the mutation score as reported by Descartes. 3) If DSpot has generated tests, add them to
XWiki’s source code in src/test/dspot and add the following to the pom of that module:
<build>
<plugins>
<!-- Add test source root for executing DSpot-generated tests -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
Example:
https://github.com/xwiki/xwiki-commons/tree/244ee07976c691c335b7f54c48e6308…
Note: The generated tests sometimes need to be modified a bit to pass. Personally I’ve
only committed tests that were passing and I reported issues for those that were not
passing.
4) File the various reports:
a) https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki both for
success and failures
b)
https://docs.google.com/spreadsheets/d/1LULpGpsJirmFyvHNstLGv-Gv5DVBdpLTM2h…
c) for failures, file a github issue at https://github.com/STAMP-project/dspot/issues and
link to the place on
https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki where we put the
failing result.
Note: The reason we need to report failures too is because DSpot fails a lot so we need
to show what we have tested
Thanks
-Vincent
[snip]
Thanks
-Vincent