Re: [xwiki-devs] [Proposal] Improve our Test Coverage strategy

On 14 May 2018, at 13:48, Marius Dumitru Florea <mariusdumitru.florea(a)xwiki.com> wrote:

*should* but I don't think it happens in practice, for various reasons, mainly due to limited time. Did you check the TPC of the last 10 new packages? Is it above 70%?

> * For some modules, it makes less sense or it’s harder to have unit tests > and integration tests are better. We’re not talking about 70% of unit test > coverage but 70%+ coverage of overall testing (unit, integration, > functional). > > Now we’ll need to put in place to see it in action and verify if there are > cases where this can be hard to achieve. But I’d prefer that we consider > those cases as exceptional and handle them in an ad-hoc manner. > >> >>>> >>>>> ** Find packages and/or files having a TPC lower than the previous TPC >>>>> ** Find removed packages that had a TPC higher than the average > computed >>>>> TPC >>>>> >>>> >>>> I would add: >>>> >>>> ** find the packages that have the TPC higher than what is declared in >>> the >>>> pom (because we don't always update the TPC value declared in the pom >>> when >>>> we refactor the code or when we add new tests) >>> >>> Yes I agree. We should do that but not in the pipeline for the global >>> coverage. We should have another pipeline for this and update the > pom.xml >>> files in it. >>> >>>> >>>>> * Save a report in the directory for the Clover report at >>>>> http://ci.xwiki.org/view/Tools/job/Clover/ >>>>> * For all failures, send an email to notifications(a)xwiki.org with >>> details >>>>> and a link to the saved report >>>>> * Ideally, and if we can do it, call the github API to find the > authors >>> of >>>>> commits for those packages and add them in the report. Examples of > APIs >>> we >>>>> could use: >>>>> ** https://api.github.com/repos/xwiki/xwiki-platform/commits? >>>>> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path >>>>> parameter that could be used to filter but I don’t think it’ll work >>>>> ** https://github.com/xwiki/xwiki-platform/compare/master@ >>>>> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors > need >>>>> to be extracted from the HTML which is a bit fragile) >>>>> * Add a step in the Release process to ensure that the global TPC has >>> not >>>>> been lowered. This would be a way to ensure we pay attention to that > and >>>>> fix it when we lower it. We would need to tune this to find something >>> that >>>>> helps keep the TPC increasing while not putting too much pressure at > the >>>>> same time on the release date. It doesn’t have to be in the release >>> process >>>>> but we need some checkpoint to make sure we look at it and that all > devs >>>>> fix the tests when they lower the global TPC, or at the very least > that >>> an >>>>> analysis is done in case where it’s hard to keep the TPC (for example, >>>>> removing existing code that had a lot of tests will lower the TPC ;)). >>>>> >>>>> WDYT? >>>>> >>>>> As a dev, would you be willing to pay attention to not lower the > global >>>>> TPC and work to fix it when it happens? >>> >>> @Marius: what about this question? Would you be ok with it? :) >>> >> >> Sure. > > ok cool > > Thanks > -Vincent > >> >> >>> >>> Thanks >>> -Vincent >>> >>>>> >>>>> Thanks >>>>> -Vincent