17 Oct
2018
17 Oct
'18
6:33 p.m.
On 17 Oct 2018, at 15:54, Vincent Massol
<vincent(a)massol.net> wrote:
Hi,
I’ve retested on xwiki-commons-component-default:
1) With —descartes: failure, see https://github.com/STAMP-project/dspot/issues/584
2) With jacoco selector: failure, see https://github.com/STAMP-project/dspot/issues/586.
I’ve manually fixed the tests and remove those that didn’t pass. I got only +0.18% jacoco
coverage increase and -2% descartes mutation score… That’s the problem, we would need a
selector that optimizes for both. I’ve created
https://github.com/STAMP-project/dspot/issues/587
3) With clover selector: no tests generated! Opened
https://github.com/STAMP-project/dspot/issues/588
So my recommendation is to wait for https://github.com/STAMP-project/dspot/issues/584 to
be fixed and then to use —descartes for our measures FTM.
Thanks
-Vincent
PS: Command lines used for reference:
- java -jar
/Users/vmassol/dev/dspot/dspot/target/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar
--path-to-properties dspot.properties --descartes --verbose --generate-new-test-class
--with-comment
- java -jar
/Users/vmassol/dev/dspot/dspot/target/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar
--path-to-properties dspot.properties --test-criterion JacocoCoverageSelector --verbose
--generate-new-test-class --with-comment
- java -jar
/Users/vmassol/dev/dspot/dspot/target/dspot-1.1.1-SNAPSHOT-jar-with-dependencies.jar
--path-to-properties dspot.properties --test-criterion CloverCoverageSelector --verbose
--generate-new-test-class --with-comment
On 17 Oct 2018, at 11:20, Vincent Massol
<vincent(a)massol.net> wrote:
Hi,
[snip]
So actually, after speaking with Benjamin, I’ve realized a few things:
* By default DSpot runs with the PIT selector (PitMutantScoreSelector) which is
configured to use the default PIT mutations. This is why we need to run with the PIT
selector but configured to use the Descartes mutation, and this is done by specifying
--descartes.
* Now this will optimize the generation of new tests for their increased mutation score.
Right now we got 0% all the time on our tests (see
https://docs.google.com/spreadsheets/d/1LULpGpsJirmFyvHNstLGv-Gv5DVBdpLTM2h…)
and it’s because we didn’t use --descartes. We need to try again or run on new modules
with --descartes and see what it gives us. It’s possible it’ll generate even less tests…
* For the coverage part, there are 2 other selectors that can be used with DSpot to
generate tests that all increase the coverage:
** "--test-criterion JacocoCoverageSelector": uses jacoco and keep tests that
increase the instruction coverage
** "--test-criterion CloverCoverageSelector”: uses openclover and keep tests that
increase the branch coverage
So we need to test with the various selectors and see what we get. Process to run DSpot:
1) Pick a module. Measure coverage and mutation score (or take the value there already if
they’re in the pom.xml). Same as for Descartes testing.
2) Run DSpot on the module, see
https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot for explanations
One important detail that I had missed. We need to run Dspot with “—descartes” on the
command line so that it uses Descartes for computing the mutation score for mutations and
only keep tests that increase the mutation score as reported by Descartes.
If we want to get the best values, we should use --descartes for K03 and either jacoco or
clover selector for K01. Now we need to see what tests we get.
Thanks
-Vincent
>
>> 3) If DSpot has generated tests, add them to XWiki’s source code in
src/test/dspot and add the following to the pom of that module:
>>
>> <build>
>> <plugins>
>> <!-- Add test source root for executing DSpot-generated tests -->
>> <plugin>
>> <groupId>org.codehaus.mojo</groupId>
>> <artifactId>build-helper-maven-plugin</artifactId>
>> </plugin>
>> </plugins>
>> </build>
>>
>> Example:
https://github.com/xwiki/xwiki-commons/tree/244ee07976c691c335b7f54c48e6308…
>>
>> Note: The generated tests sometimes need to be modified a bit to pass. Personally
I’ve only committed tests that were passing and I reported issues for those that were not
passing.
>>
>> 4) File the various reports:
>> a) https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki both
for success and failures
>> b)
https://docs.google.com/spreadsheets/d/1LULpGpsJirmFyvHNstLGv-Gv5DVBdpLTM2h…
>> c) for failures, file a github issue at
https://github.com/STAMP-project/dspot/issues and link to the place on
https://github.com/STAMP-project/dspot-usecases-output/tree/master/xwiki where we put the
failing result.
>>
>> Note: The reason we need to report failures too is because DSpot fails a lot so
we need to show what we have tested
>>
>> Thanks
>> -Vincent
>>
>
> [snip]
>
> Thanks
> -Vincent