[xwiki-devs] Clover TPC values for Commons/Rendering/Platform/Enterprise
Hi devs, I've finally spent the time to measure our TPC (Test Percentage Coverage) using Clover. The reports are very interesting and show where we need to spend our efforts in testing (be them automated or manual). Here are the raw top level results: =========================== All results: http://maven.xwiki.org/site/clover/20120701/ * Only Commons (this means unit tests and integration tests in the commons repository only): 62.4% * Only Rendering: 75.5% * Only Platform: 36.8% * Commons+Rendering (ie Rendering tests exercise also Commons code): 72.1% * Commons+Rendering+Platform: 47.4% * Commons+Rendering+Platform+Enterprise: So our code is roughly tested at X% (note that 100% TPC doesn't mean the code is working, it means each line of code has been tested in some way but it doesn't mean all combinations have been tested!). A good value is between 70-80%. We're actually better than I thought (I had 50% or lower in mind) :) However there are huge disparity with code having 90+% and code having 0% coverage (402 classes have 0% coverage!). Quick high level analysis: ==================== * Commons is missing quite a lot of unit tests * Wikimodel in Rendering is dragging the TPC down, I think we were above 80% before we included WikiModel) * Platform has a very low level of unit tests which means it's easy to break stuff in platform and it's hard to verify that what we add in platform works well in general (means going through longer stabilization phases than needed) * Our functional tests are quite effective (even though they take a very very long time to execute) since they make the TPC go from 47% to 63% for Commons+Rendering+Platform. The bad TPC for Platform is probably improved by the functional tests. Next steps ========= * IMO all developers need to look in details at the results and familiarize themselves with the Clover reports * Each dev should look at stuff he has coded and ensure that they have a good level of unit tests * All new code should get at the very minimum between 70-80% unit test coverage. For example the new diff module in commons only has only 43.4% coverage which is far from enough * Spend time to automate the generation of these reports. We need a dedicated agent for this since it "contaminates" the local repository with clovered jars. The overall generation time is around 4-5 hours. * Dig more around the reports to identify functional areas of XWiki missing automated tests. These areas must be covered by manual testing at each release. * Decide if we want a more rigorous strategy for new code, and how to get alerts for modules missing coverage. WDYT? Let's share your analysis in this thread so that we can brainstorm about what to do with these reports Thanks -Vincent
On Jul 2, 2012, at 6:53 PM, Vincent Massol wrote:
Hi devs,
I've finally spent the time to measure our TPC (Test Percentage Coverage) using Clover.
The reports are very interesting and show where we need to spend our efforts in testing (be them automated or manual).
Here are the raw top level results: ===========================
All results: http://maven.xwiki.org/site/clover/20120701/
* Only Commons (this means unit tests and integration tests in the commons repository only): 62.4% * Only Rendering: 75.5% * Only Platform: 36.8% * Commons+Rendering (ie Rendering tests exercise also Commons code): 72.1% * Commons+Rendering+Platform: 47.4% * Commons+Rendering+Platform+Enterprise:
The value is 63.1%
So our code is roughly tested at X% (note that 100% TPC doesn't mean the code is working, it means each line of code has been tested in some way but it doesn't mean all combinations have been tested!).
s/X/63.1%/ I also forgot to mention that several functional tests failed, almost in each functional test module… Thanks -Vincent
A good value is between 70-80%.
We're actually better than I thought (I had 50% or lower in mind) :) However there are huge disparity with code having 90+% and code having 0% coverage (402 classes have 0% coverage!).
Quick high level analysis: ====================
* Commons is missing quite a lot of unit tests * Wikimodel in Rendering is dragging the TPC down, I think we were above 80% before we included WikiModel) * Platform has a very low level of unit tests which means it's easy to break stuff in platform and it's hard to verify that what we add in platform works well in general (means going through longer stabilization phases than needed) * Our functional tests are quite effective (even though they take a very very long time to execute) since they make the TPC go from 47% to 63% for Commons+Rendering+Platform. The bad TPC for Platform is probably improved by the functional tests.
Next steps =========
* IMO all developers need to look in details at the results and familiarize themselves with the Clover reports * Each dev should look at stuff he has coded and ensure that they have a good level of unit tests * All new code should get at the very minimum between 70-80% unit test coverage. For example the new diff module in commons only has only 43.4% coverage which is far from enough * Spend time to automate the generation of these reports. We need a dedicated agent for this since it "contaminates" the local repository with clovered jars. The overall generation time is around 4-5 hours. * Dig more around the reports to identify functional areas of XWiki missing automated tests. These areas must be covered by manual testing at each release. * Decide if we want a more rigorous strategy for new code, and how to get alerts for modules missing coverage.
WDYT?
Let's share your analysis in this thread so that we can brainstorm about what to do with these reports
Thanks -Vincent
On Jul 2, 2012, at 6:58 PM, Vincent Massol wrote:
On Jul 2, 2012, at 6:53 PM, Vincent Massol wrote:
Hi devs,
I've finally spent the time to measure our TPC (Test Percentage Coverage) using Clover.
The reports are very interesting and show where we need to spend our efforts in testing (be them automated or manual).
Here are the raw top level results: ===========================
All results: http://maven.xwiki.org/site/clover/20120701/
* Only Commons (this means unit tests and integration tests in the commons repository only): 62.4% * Only Rendering: 75.5% * Only Platform: 36.8% * Commons+Rendering (ie Rendering tests exercise also Commons code): 72.1% * Commons+Rendering+Platform: 47.4% * Commons+Rendering+Platform+Enterprise:
The value is 63.1%
So our code is roughly tested at X% (note that 100% TPC doesn't mean the code is working, it means each line of code has been tested in some way but it doesn't mean all combinations have been tested!).
s/X/63.1%/
I also forgot to mention that several functional tests failed, almost in each functional test module…
Thanks -Vincent
A good value is between 70-80%.
We're actually better than I thought (I had 50% or lower in mind) :) However there are huge disparity with code having 90+% and code having 0% coverage (402 classes have 0% coverage!).
Quick high level analysis: ====================
* Commons is missing quite a lot of unit tests * Wikimodel in Rendering is dragging the TPC down, I think we were above 80% before we included WikiModel) * Platform has a very low level of unit tests which means it's easy to break stuff in platform and it's hard to verify that what we add in platform works well in general (means going through longer stabilization phases than needed) * Our functional tests are quite effective (even though they take a very very long time to execute) since they make the TPC go from 47% to 63% for Commons+Rendering+Platform. The bad TPC for Platform is probably improved by the functional tests.
Next steps =========
* IMO all developers need to look in details at the results and familiarize themselves with the Clover reports * Each dev should look at stuff he has coded and ensure that they have a good level of unit tests * All new code should get at the very minimum between 70-80% unit test coverage. For example the new diff module in commons only has only 43.4% coverage which is far from enough * Spend time to automate the generation of these reports. We need a dedicated agent for this since it "contaminates" the local repository with clovered jars. The overall generation time is around 4-5 hours. * Dig more around the reports to identify functional areas of XWiki missing automated tests. These areas must be covered by manual testing at each release. * Decide if we want a more rigorous strategy for new code, and how to get alerts for modules missing coverage.
List of XWiki modules that are not tested or badly tested: * Portlet * Some Actions: Download revision, Delete version, Reset version, Tex Action (mathematic formula rendering), View Attach Rev, Redirect, Prop Enable, Prop Disable, etc * GWT API * Feed plugin * Chart plugin (we need to remove it IMO) * Rights Manager plugin * Database Migrations are not tested at all by automated tests * User directory plugin (do we still need it?) * Stats * Repository and Extension missing some tests but Thomas/Marius are adding them * Monitor plugin/api * Workspaces * Autotag plugin * WikiManager plugin * Global Search plugin (didn't even know about it!) * ApplicationManager plugin * Graphviz plugin * SVG plugin * Usertools plugin * Openoffice importer * The oldcore model is not that well tested (around 50-60% which is low for something so important) * Activity stream is not well tested * Rendering 1.0 is not that well tested (but we don't care ;)) * Mail plugin (probably need to be retired and possibly replaced by the work done by Jeremie) Note that it's hard to find "features" from the TPC since feature are cross-modules. IMO a lot of these modules need to be moved out of the platform and moved to contrib and not be bundled by default. Thanks -Vincent
WDYT?
Let's share your analysis in this thread so that we can brainstorm about what to do with these reports
Thanks -Vincent
On Jul 3, 2012, at 10:11 AM, Vincent Massol wrote:
On Jul 2, 2012, at 6:58 PM, Vincent Massol wrote:
On Jul 2, 2012, at 6:53 PM, Vincent Massol wrote:
Hi devs,
I've finally spent the time to measure our TPC (Test Percentage Coverage) using Clover.
The reports are very interesting and show where we need to spend our efforts in testing (be them automated or manual).
Here are the raw top level results: ===========================
All results: http://maven.xwiki.org/site/clover/20120701/
* Only Commons (this means unit tests and integration tests in the commons repository only): 62.4% * Only Rendering: 75.5% * Only Platform: 36.8% * Commons+Rendering (ie Rendering tests exercise also Commons code): 72.1% * Commons+Rendering+Platform: 47.4% * Commons+Rendering+Platform+Enterprise:
The value is 63.1%
So our code is roughly tested at X% (note that 100% TPC doesn't mean the code is working, it means each line of code has been tested in some way but it doesn't mean all combinations have been tested!).
s/X/63.1%/
I also forgot to mention that several functional tests failed, almost in each functional test module…
Thanks -Vincent
A good value is between 70-80%.
We're actually better than I thought (I had 50% or lower in mind) :) However there are huge disparity with code having 90+% and code having 0% coverage (402 classes have 0% coverage!).
We have 946 classes with a TPC < 60%, with a total of 3006 classes. So that's 31% of our code not tested well enough. Thanks -Vincent
Quick high level analysis: ====================
* Commons is missing quite a lot of unit tests * Wikimodel in Rendering is dragging the TPC down, I think we were above 80% before we included WikiModel) * Platform has a very low level of unit tests which means it's easy to break stuff in platform and it's hard to verify that what we add in platform works well in general (means going through longer stabilization phases than needed) * Our functional tests are quite effective (even though they take a very very long time to execute) since they make the TPC go from 47% to 63% for Commons+Rendering+Platform. The bad TPC for Platform is probably improved by the functional tests.
Next steps =========
* IMO all developers need to look in details at the results and familiarize themselves with the Clover reports * Each dev should look at stuff he has coded and ensure that they have a good level of unit tests * All new code should get at the very minimum between 70-80% unit test coverage. For example the new diff module in commons only has only 43.4% coverage which is far from enough * Spend time to automate the generation of these reports. We need a dedicated agent for this since it "contaminates" the local repository with clovered jars. The overall generation time is around 4-5 hours. * Dig more around the reports to identify functional areas of XWiki missing automated tests. These areas must be covered by manual testing at each release. * Decide if we want a more rigorous strategy for new code, and how to get alerts for modules missing coverage.
List of XWiki modules that are not tested or badly tested:
* Portlet * Some Actions: Download revision, Delete version, Reset version, Tex Action (mathematic formula rendering), View Attach Rev, Redirect, Prop Enable, Prop Disable, etc * GWT API * Feed plugin * Chart plugin (we need to remove it IMO) * Rights Manager plugin * Database Migrations are not tested at all by automated tests * User directory plugin (do we still need it?) * Stats * Repository and Extension missing some tests but Thomas/Marius are adding them * Monitor plugin/api * Workspaces * Autotag plugin * WikiManager plugin * Global Search plugin (didn't even know about it!) * ApplicationManager plugin * Graphviz plugin * SVG plugin * Usertools plugin * Openoffice importer * The oldcore model is not that well tested (around 50-60% which is low for something so important) * Activity stream is not well tested * Rendering 1.0 is not that well tested (but we don't care ;)) * Mail plugin (probably need to be retired and possibly replaced by the work done by Jeremie)
Note that it's hard to find "features" from the TPC since feature are cross-modules.
IMO a lot of these modules need to be moved out of the platform and moved to contrib and not be bundled by default.
Thanks -Vincent
WDYT?
Let's share your analysis in this thread so that we can brainstorm about what to do with these reports
Thanks -Vincent
participants (1)
-
Vincent Massol