Re: [xwiki-devs] [Proposal] Improve our Test Coverage strategy

On 14 May 2018, at 13:30, Marius Dumitru Florea <

On Mon, May 14, 2018 at 2:17 PM, Vincent Massol &lt;vincent(a)massol.net&gt;

> > >> On 14 May 2018, at 12:28, Marius Dumitru Florea < > mariusdumitru.florea(a)xwiki.com&gt; wrote: >> >> On Thu, May 10, 2018 at 10:03 PM, Vincent Massol &lt;vincent(a)massol.net&gt; > wrote: >> >>> Hi devs, >>> >>> Our current Test coverage strategy is to fail the build when new code >>> added results in a coverage lower than the threshold for the module, > using >>> jacoco. >>> >>> This has 2 limitations causing our global TPC to go down from time to > time >>> (see https://markmail.org/message/hqumkdiz7jm76ya6 ). >>> >>> Thus I’d like to propose the following addition to our strategy: >>> >>> * We already have a jenkins pipeline to automatically compute the full > TPC >>> using Clover. See http://dev.xwiki.org/xwiki/ > bin/view/Community/Testing# >>> HUsingClover2BJenkins >>> * Make it run more often (it’s currently executed once per month, see >>> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to >>> execute. Thus we could run it once per week or even once per day

> the >>> night. >>> * Add some groovy logic in the pipeline to perform an analysis after

Requiring 70% TPC for new packages is a nice goal but I find it hard to achieve in practice.

>> >>> ** Find packages and/or files having a TPC lower than the previous TPC >>> ** Find removed packages that had a TPC higher than the average

>>> TPC >>> >> >> I would add: >> >> ** find the packages that have the TPC higher than what is declared in > the >> pom (because we don't always update the TPC value declared in the pom > when >> we refactor the code or when we add new tests) > > Yes I agree. We should do that but not in the pipeline for the global > coverage. We should have another pipeline for this and update the

> files in it. > >> >>> * Save a report in the directory for the Clover report at >>> http://ci.xwiki.org/view/Tools/job/Clover/ >>> * For all failures, send an email to notifications(a)xwiki.org with > details >>> and a link to the saved report >>> * Ideally, and if we can do it, call the github API to find the

> of >>> commits for those packages and add them in the report. Examples of

> we >>> could use: >>> ** https://api.github.com/repos/xwiki/xwiki-platform/commits? >>> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path >>> parameter that could be used to filter but I don’t think it’ll work >>> ** https://github.com/xwiki/xwiki-platform/compare/master@ >>> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors

>>> to be extracted from the HTML which is a bit fragile) >>> * Add a step in the Release process to ensure that the global TPC has > not >>> been lowered. This would be a way to ensure we pay attention to that

>>> fix it when we lower it. We would need to tune this to find something > that >>> helps keep the TPC increasing while not putting too much pressure at

>>> same time on the release date. It doesn’t have to be in the release > process >>> but we need some checkpoint to make sure we look at it and that all

>>> fix the tests when they lower the global TPC, or at the very least

> an >>> analysis is done in case where it’s hard to keep the TPC (for example, >>> removing existing code that had a lot of tests will lower the TPC ;)). >>> >>> WDYT? >>> >>> As a dev, would you be willing to pay attention to not lower the

Sure.

> > Thanks > -Vincent > >>> >>> Thanks >>> -Vincent