[xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 2)

Hi devs, We started discussing a first global test coverage strategy in https://markmail.org/message/grphwta63pp5p4l7 I’d like to propose some updates and tuning now that we have a Clover Jenkins pipeline working (brainstormed with Simon): Here’s the new proposal: * We run the Clover Jenkins pipeline every night (between 11PM-8AM) * The pipeline sends an email whenever the new report has modules lowering the global TPC score compared to the baseline report (negative contribution per module) * The baseline report is the report generated just after each XS release. This means that we keep the same baseline during a XS release ** Technically it means that the pipeline will update the latest.txt file (which contains the clover report timestamp) when it notices a version change * We add a step in the Release Plan Template to have the report passing before we can release. * The RM is in charge of a release from day 1 to the release day (already the case), and is also in charge of making sure that the global coverage job failures get addressed before the release day so that we’re ready on the release day. Options: * Make it easier and only fail the pipeline job when the global TPC value is lower than the baseline (vs failing whenever a module has negative contribution) WDYT? Thanks -Vincent