3 Jul
2012
3 Jul
'12
10:18 a.m.
On Jul 3, 2012, at 10:11 AM, Vincent Massol wrote:
On Jul 2, 2012, at 6:58 PM, Vincent Massol wrote:
>
> On Jul 2, 2012, at 6:53 PM, Vincent Massol wrote:
>
>> Hi devs,
>>
>> I've finally spent the time to measure our TPC (Test Percentage Coverage)
using Clover.
>>
>> The reports are very interesting and show where we need to spend our efforts in
testing (be them automated or manual).
>>
>> Here are the raw top level results:
>> ===========================
>>
>> All results: http://maven.xwiki.org/site/clover/20120701/
>>
>> * Only Commons (this means unit tests and integration tests in the commons
repository only): 62.4%
>> * Only Rendering: 75.5%
>> * Only Platform: 36.8%
>> * Commons+Rendering (ie Rendering tests exercise also Commons code): 72.1%
>> * Commons+Rendering+Platform: 47.4%
>> * Commons+Rendering+Platform+Enterprise:
>
> The value is 63.1%
>
>>
>> So our code is roughly tested at X% (note that 100% TPC doesn't mean the code
is working, it means each line of code has been tested in some way but it doesn't mean
all combinations have been tested!).
>
> s/X/63.1%/
>
> I also forgot to mention that several functional tests failed, almost in each
functional test module…
>
> Thanks
> -Vincent
>
>> A good value is between 70-80%.
>>
>> We're actually better than I thought (I had 50% or lower in mind) :) However
there are huge disparity with code having 90+% and code having 0% coverage (402 classes
have 0% coverage!).
We have 946 classes with a TPC < 60%, with a total of 3006 classes. So that's 31%
of our code not tested well enough.
Thanks
-Vincent
> Quick high
level analysis:
> ====================
>
> * Commons is missing quite a lot of unit tests
> * Wikimodel in Rendering is dragging the TPC down, I think we were above 80% before
we included WikiModel)
> * Platform has a very low level of unit tests which means it's easy to break
stuff in platform and it's hard to verify that what we add in platform works well in
general (means going through longer stabilization phases than needed)
> * Our functional tests are quite effective (even though they take a very very long
time to execute) since they make the TPC go from 47% to 63% for
Commons+Rendering+Platform. The bad TPC for Platform is probably improved by the
functional tests.
>
> Next steps
> =========
>
> * IMO all developers need to look in details at the results and familiarize
themselves with the Clover reports
> * Each dev should look at stuff he has coded and ensure that they have a good level
of unit tests
> * All new code should get at the very minimum between 70-80% unit test coverage. For
example the new diff module in commons only has only 43.4% coverage which is far from
enough
> * Spend time to automate the generation of these reports. We need a dedicated agent
for this since it "contaminates" the local repository with clovered jars. The
overall generation time is around 4-5 hours.
> * Dig more around the reports to identify functional areas of XWiki missing automated
tests. These areas must be covered by manual testing at each release.
> * Decide if we want a more rigorous strategy for new code, and how to get alerts for
modules missing coverage.
List of XWiki modules that are not tested or badly tested:
* Portlet
* Some Actions: Download revision, Delete version, Reset version, Tex Action (mathematic
formula rendering), View Attach Rev, Redirect, Prop Enable, Prop Disable, etc
* GWT API
* Feed plugin
* Chart plugin (we need to remove it IMO)
* Rights Manager plugin
* Database Migrations are not tested at all by automated tests
* User directory plugin (do we still need it?)
* Stats
* Repository and Extension missing some tests but Thomas/Marius are adding them
* Monitor plugin/api
* Workspaces
* Autotag plugin
* WikiManager plugin
* Global Search plugin (didn't even know about it!)
* ApplicationManager plugin
* Graphviz plugin
* SVG plugin
* Usertools plugin
* Openoffice importer
* The oldcore model is not that well tested (around 50-60% which is low for something so
important)
* Activity stream is not well tested
* Rendering 1.0 is not that well tested (but we don't care ;))
* Mail plugin (probably need to be retired and possibly replaced by the work done by
Jeremie)
Note that it's hard to find "features" from the TPC since feature are
cross-modules.
IMO a lot of these modules need to be moved out of the platform and moved to contrib and
not be bundled by default.
Thanks
-Vincent
WDYT?
Let's share your analysis in this thread so that we can brainstorm about what to do
with these reports
Thanks
-Vincent