4 Jan
2019
4 Jan
'19
3:14 p.m.
Hi devs,
FYI I've modified the clover job to do the following today:
* run daily around midnight on a4
* generate 2 diff reports: one against 20171220 (that's the one we're currently
getting) and one against 20190101
* the first diff report doesn’t send an email on failure while the second one does.
I also changed yesterday:
* Only mark in RED (ie failures) modules where the global diff TPC is negative (before it
was in RED when the global contribution was < 0 but that wasn’t accounting for removed
modules for example).
Consequences:
* When we get report failures from the CI we need to fix it since it means we’ve regressed
since 20190101
* It would still be good that we continue to analyze the diff report since 20171220 to
fully understand it and make sure we haven’t regressed in quality since then, or at least
fix the big regressions.
TODO:
* Need to find a way to indicate false positives, i.e. modules in RED for which we
consider it’s ok to have a negative diff TPC, and thus know what’s left to do for the
release.
* Once we better understand and fix modules we’ll get to the second step which is to
automate the baseline value (manual ATM) and to have the report passing as a constraint
for releasing and thus indicated in the Release Plan.
Let me know if you have questions.
Thanks
-Vincent
On 3 Oct 2018, at 10:56, Vincent Massol
<vincent(a)massol.net> wrote:
Hi devs,
We started discussing a first global test coverage strategy in
https://markmail.org/message/grphwta63pp5p4l7
I’d like to propose some updates and tuning now that we have a Clover Jenkins pipeline
working (brainstormed with Simon):
Here’s the new proposal:
* We run the Clover Jenkins pipeline every night (between 11PM-8AM)
* The pipeline sends an email whenever the new report has modules lowering the global TPC
score compared to the baseline report (negative contribution per module)
* The baseline report is the report generated just after each XS release. This means that
we keep the same baseline during a XS release
** Technically it means that the pipeline will update the latest.txt file (which contains
the clover report timestamp) when it notices a version change
* We add a step in the Release Plan Template to have the report passing before we can
release.
* The RM is in charge of a release from day 1 to the release day (already the case), and
is also in charge of making sure that the global coverage job failures get addressed
before the release day so that we’re ready on the release day.
Options:
* Make it easier and only fail the pipeline job when the global TPC value is lower than
the baseline (vs failing whenever a module has negative contribution)
WDYT?
Thanks
-Vincent