14 May
2018
14 May
'18
1:41 p.m.
On 14 May 2018, at 13:30, Marius Dumitru Florea
<mariusdumitru.florea(a)xwiki.com> wrote:
On Mon, May 14, 2018 at 2:17 PM, Vincent Massol <vincent(a)massol.net> wrote:
TPC
What is the average computed TPC currently?
It’s about 70%, see for example:
http://maven.xwiki.org/site/clover/20180511/clover-
commons+rendering+platform-20180511-0147/dashboard.html
Requiring 70% TPC for new packages is a nice goal but I find it hard to
achieve in practice.
Two points on this:
* We should have above 80-90%+ in practice for new modules, not 70%. If we have 70% or
less we can be sure you did a bad job on quality and that you’re impacting the overall
quality of XWiki.
* For some modules, it makes less sense or it’s harder to have unit tests and integration
tests are better. We’re not talking about 70% of unit test coverage but 70%+ coverage of
overall testing (unit, integration, functional).
Now we’ll need to put in place to see it in action and verify if there are cases where
this can be hard to achieve. But I’d prefer that we consider those cases as exceptional
and handle them in an ad-hoc manner.
On 14 May 2018, at 12:28, Marius Dumitru Florea
<
mariusdumitru.florea(a)xwiki.com> wrote:
On Thu, May 10, 2018 at 10:03 PM, Vincent Massol <vincent(a)massol.net>
wrote:
> Hi devs,
>
> Our current Test coverage strategy is to fail the build when new code
> added results in a coverage lower than the threshold for the module,
using
> jacoco.
>
> This has 2 limitations causing our global TPC to go down from time to
time
> (see
https://markmail.org/message/hqumkdiz7jm76ya6 ).
>
> Thus I’d like to propose the following addition to our strategy:
>
> * We already have a jenkins pipeline to automatically compute the full
TPC
> using Clover. See http://dev.xwiki.org/xwiki/
bin/view/Community/Testing#
> HUsingClover2BJenkins
> * Make it run more often (it’s currently executed once per month, see
> http://ci.xwiki.org/view/Tools/job/Clover/). Takes about 5-6 hours to
> execute. Thus we could run it once per week or even once per day during
the
night.
* Add some groovy logic in the pipeline to perform an analysis after the
Clover report has been generated. Perform 2 checks by comparing the
previous report with the one that just executed:
> ** Find new packages introduced that have a TPC < the average computed ** Find packages and/or files having a TPC lower
than the previous TPC
** Find removed packages that had a TPC higher than the average computed
TPC
I would add:
** find the packages that have the TPC higher than what is declared in pom (because we don't always update the TPC
value declared in the pom
when
we refactor the code or when we add new tests)
Yes I agree. We should do that but not in the pipeline for the global
coverage. We should have another pipeline for this and update the pom.xml
files in it.
> * Save a report in the directory for the Clover report at
> http://ci.xwiki.org/view/Tools/job/Clover/
> * For all failures, send an email to notifications(a)xwiki.org with
details
> and a link to the saved report
> * Ideally, and if we can do it, call the github API to find the authors
of
> commits for those packages and add them in
the report. Examples of APIs
we
> could use:
> ** https://api.github.com/repos/xwiki/xwiki-platform/commits?
> since=2018-05-07T00:00:00Z&until=2018-05-10T00:00:00Z (there’s a path
> parameter that could be used to filter but I don’t think it’ll work
> ** https://github.com/xwiki/xwiki-platform/compare/master@
> %7B2018-05-07%7D...master@%7B2018-05-10%7D (the committers/authors need
> to be extracted from the HTML which is a bit fragile)
> * Add a step in the Release process to ensure that the global TPC has
not
> been lowered. This would be a way to ensure
we pay attention to that and
> fix it when we lower it. We would need to tune this to find something
that
> helps keep the TPC increasing while not
putting too much pressure at the
> same time on the release date. It doesn’t have to be in the release
process
> but we need some checkpoint to make sure we
look at it and that all devs
> fix the tests when they lower the global TPC, or at the very least that
an
> analysis is done in case where it’s hard to
keep the TPC (for example,
> removing existing code that had a lot of tests will lower the TPC ;)).
>
> WDYT?
>
> As a dev, would you be willing to pay attention to not lower the global
> TPC and work to fix it when it happens?
@Marius: what about this question? Would you be ok with it? :)
>
> Thanks
> -Vincent
>
>>>
>>> Thanks
>>> -Vincent