[xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 3)

Hi devs, After TAKE 2 (see http://markmail.org/message/owtyhkmrz4tcbymn ), and after analyzing several modules (I analyzed about 4-5 in total), I think we should improve a bit the strategy to make it usable and applicable. Lessons learnt ============ * It takes a lot of time to analyze a single global tpc drop (every time it takes me around 2 hours) * In general the results of the analysis are not that great. There are often “good enough” reasons for the drop. It’s often a lack of unit tests and code that is exercised by functional tests but the path has changed for various reasons. * I find the ratio of time to spend on the analysis vs result to be too low. * In the end what’s important is that our global TPC continues to grow New Strategy =========== * We run the Clover Jenkins pipeline every night (between 11PM-8AM) * The pipeline sends an email whenever the new report has its global TPC going down when compared with the baseline (vs one or more modules had their TPC lowered in TAKE 2) * The baseline report is the report generated just after each XS release. This means that we keep the same baseline during a XS release ** Technically it means that the pipeline will update the latest.txt file (which contains the clover report timestamp) when it notices a version change * We add a step in the Release Plan Template to have the report passing before we can release. * The RM is in charge of a release from day 1 to the release day (already the case), and is also in charge of making sure that the global coverage job failures get addressed before the release day so that we’re ready on the release day. * Implementation detail: don’t send a failure email when there are failing tests in the build, to avoid false positives WDYT? Thanks -Vincent