11 May
2018
11 May
'18
9:25 p.m.
On 11 May 2018, at 15:38, Thomas Mortagne
<thomas.mortagne(a)xwiki.com> wrote:
On Thu, May 10, 2018 at 9:03 PM, Vincent Massol <vincent(a)massol.net> wrote:
Yes I thought about that too. It actually works but you need to handle past reports and
not discard them, which I agree can be a problem especially if you want to have a failing
job (see below).
I thought about having a file that we generate in the pipeline script and put in
http://maven.xwiki.org/site/clover/ which will contain the directory name for the last
good Clover report (i.e the last report having a global TPC higher than the previous one).
It’s not perfect either but maybe good enough to start with. The problem is not reacting
fast enough and accumulating too many changes and keeping a failing TPC check forever.
Hi devs,
Our current Test coverage strategy is to fail the build when new code added results in a
coverage lower than the threshold for the module, using jacoco.
This has 2 limitations causing our global TPC to go down from time to time (see
https://markmail.org/message/hqumkdiz7jm76ya6 ).
Thus I’d like to propose the following addition to our strategy:
* We already have a jenkins pipeline to automatically compute the full TPC using Clover.
See http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HUsingClover2BJenkins
* Make it run more often (it’s currently executed once per month, see
http://ci.xwiki.org/view/Tools/job/Clover/). Takes abll out 5-6 hours to execute. Thus we
could run it once per week or even once per day during the night.
* Add some groovy logic in the pipeline to perform an analysis after the Clover report
has been generated. Perform 2 checks by comparing the previous report with the one that
just executed:
** Find new packages introduced that have a TPC < the average computed TPC
** Find packages and/or files having a TPC lower than the previous TPC
** Find removed packages that had a TPC higher than the average computed TPC
"comparing the previous report" won't work if we run this every days,
it means you just have to wait 24h and the current report will be all
green :)
It should be compared to a specific report (or even several) like
comparing to the current LTS branch report and to the report produced
after the previous master release. * Save a report in the directory for the Clover
report at http://ci.xwiki.org/view/Tools/job/Clover/
* For all failures, send an email to notifications(a)xwiki.org with details and a link to
the saved report
Will we also see a build in error in the ci.xwiki.org home page ? Not
sure a notifications(a)xwiki.org mail only would be enough :) * Ideally, and if we can do it, call the github
API to find the authors of commits for those packages and add them in the report. Examples
of APIs we could use:
**
https://api.github.com/repos/xwiki/xwiki-platform/commits?since=2018-05-07T…
(there’s a path parameter that could be used to filter but I don’t think it’ll work
**
https://github.com/xwiki/xwiki-platform/compare/master@%7B2018-05-07%7D...m…
(the committers/authors need to be extracted from the HTML which is a bit fragile)
* Add a step in th e Release process to ensure that the global TPC has not been lowered.
This would be a way to ensure we pay attention to that and fix it when we lower it. We
would need to tune this to find something that helps keep the TPC increasing while not
putting too much pressure at the same time on the release date. It doesn’t have to be in
the release process but we need some checkpoint to make sure we look at it and that all
devs fix the tests when they lower the global TPC, or at the very least that an analysis
is done in case where it’s hard to keep the TPC (for example, removing existing code that
had a lot of tests will lower the TPC ;)).
WDYT?
As a dev, would you be willing to pay attention to not lower the global TPC and work to
fix it when it happens?
Thanks
-Vincent
Sounds OK.
--
Thomas Mortagne