20 Aug
2012
20 Aug
'12
5:17 p.m.
Hi devs,
Each java module has a current TPC level (Test Percentage Coverage), be it 5% of 90%.
As part of improving XWiki's quality, the idea is that future commits in this module
shouldn't lower the module's quality.
One easy way to ensure this is to measure the current TPC level and make the build fail
whenever the build is under this TPC threshold.
This means that if you commit code that has less tests average than what currently exists
then the build will fail.
Here's one way to implement it:
<!-- Fail the build if the test coverage is below a given value.
Note: Since this takes a bit of time to execute we only run this when the
integration-tests profile is active -->
<profiles>
<profile>
<id>integration-tests</id>
<build>
<plugins>
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
<configuration>
<targetPercentage>86.9%</targetPercentage>
</configuration>
<executions>
<execution>
<id>clover-check</id>
<phase>verify</phase>
<goals>
<goal>instrument-test</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
So this proposal is about the following strategy:
* Allow committers to modify module's pom.xml to fail the build when then TPC is under
the current threshold
* Since oldcore is really difficult to test, maybe we should not set build failing for
oldcore. OTOH oldcore is an important module. WDYT?
* If for some reason a committer wants to lower the TPC threshold for a module he needs to
get the agreement from the other committers (vote). There might be valid reasons like
fixing an important bug quickly for a release and writing a test is just too complex so
the writing of the test is to be delayed for after the release, etc.
* I'd like to consider this as an experiment and see how it goes
Note: This is about unit test TPC which is different from the TPC that includes functional
tests.
WDYT?
Here's my +1 to try this out and see how it goes.
Thanks
-Vincent
20 Aug
20 Aug
9:28 p.m.
New subject: [xwiki-devs] [Proposal] Failing a module's build if under a given TPC threshold
On 08/20/2012 11:17 AM, Vincent Massol wrote:
Hi devs,
Each java module has a current TPC level (Test Percentage Coverage), be it 5% of 90%.
As part of improving XWiki's quality, the idea is that future commits in this module
shouldn't lower the module's quality.
One easy way to ensure this is to measure the current TPC level and make the build fail
whenever the build is under this TPC threshold.
This means that if you commit code that has less tests average than what currently exists
then the build will fail.
Here's one way to implement it:
<!-- Fail the build if the test coverage is below a given value.
Note: Since this takes a bit of time to execute we only run this when the
integration-tests profile is active -->
<profiles>
<profile>
<id>integration-tests</id>
<build>
<plugins>
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
<configuration>
<targetPercentage>86.9%</targetPercentage>
</configuration>
<executions>
<execution>
<id>clover-check</id>
<phase>verify</phase>
<goals>
<goal>instrument-test</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
So this proposal is about the following strategy:
* Allow committers to modify module's pom.xml to fail the build when then TPC is
under the current threshold
* Since oldcore is really difficult to test, maybe we should not set build failing for
oldcore. OTOH oldcore is an important module. WDYT?
* If for some reason a committer wants to lower the TPC threshold for a module he needs
to get the agreement from the other committers (vote). There might be valid reasons like
fixing an important bug quickly for a release and writing a test is just too complex so
the writing of the test is to be delayed for after the release, etc.
* I'd like to consider this as an experiment and see how it goes
Note: This is about unit test TPC which is different from the TPC that includes
functional tests.
WDYT?
Here's my +1 to try this out and see how it goes.
+1.
I'm not sure that a [vote] is required, since that is usually something
that is too formal. By the rules, a vote should last 3 days, it should
have at least 3 +1, and no -1. But IMO a +1 means strong support, the
voter understands and agrees with the matter, and lowering the TPC is
not something on which people have a strong opinion. And the example
that you give requires a quick decision, the 3 days required to pass a
vote should be enough to write the missing tests.
So, I'd rather trust developers not to lower the threshold without a
valid reason, and we can always signal a change that we don't agree with
afterwards.
--
Sergiu Dumitriu
http://purl.org/net/sergiu/
11:08 p.m.
New subject: [xwiki-devs] [Proposal] Failing a module's build if under a given TPC threshold
+1 with the same concern as Sergiu.
Caleb
On 08/20/2012 03:28 PM, Sergiu Dumitriu wrote:
On 08/20/2012 11:17 AM, Vincent Massol wrote:
Hi devs,
Each java module has a current TPC level (Test Percentage Coverage), be it 5% of 90%.
As part of improving XWiki's quality, the idea is that future commits in this module
shouldn't lower the module's quality.
One easy way to ensure this is to measure the current TPC level and make the build fail
whenever the build is under this TPC threshold.
This means that if you commit code that has less tests average than what currently exists
then the build will fail.
Here's one way to implement it:
<!-- Fail the build if the test coverage is below a given value.
Note: Since this takes a bit of time to execute we only run this when the
integration-tests profile is active -->
<profiles>
<profile>
<id>integration-tests</id>
<build>
<plugins>
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
<configuration>
<targetPercentage>86.9%</targetPercentage>
</configuration>
<executions>
<execution>
<id>clover-check</id>
<phase>verify</phase>
<goals>
<goal>instrument-test</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
So this proposal is about the following strategy:
* Allow committers to modify module's pom.xml to fail the build when then TPC is
under the current threshold
* Since oldcore is really difficult to test, maybe we should not set build failing for
oldcore. OTOH oldcore is an important module. WDYT?
* If for some reason a committer wants to lower the TPC threshold for a module he needs
to get the agreement from the other committers (vote). There might be valid reasons like
fixing an important bug quickly for a release and writing a test is just too complex so
the writing of the test is to be delayed for after the release, etc.
* I'd like to consider this as an experiment and see how it goes
Note: This is about unit test TPC which is different from the TPC that includes
functional tests.
WDYT?
Here's my +1 to try this out and see how it goes.
+1.
I'm not sure that a [vote] is required, since that is usually something that is too
formal. By the rules, a vote should last 3 days, it should have at least 3 +1, and no -1.
But IMO a +1 means strong support, the voter understands and agrees with the matter, and
lowering the TPC is not something on which people have a strong opinion. And the example
that you give requires a quick decision, the 3 days required to pass a vote should be
enough to write the missing tests.
So, I'd rather trust developers not to lower the threshold without a valid reason,
and we can always signal a change that we don't agree with afterwards.
21 Aug
21 Aug
1:34 a.m.
New subject: [xwiki-devs] [Proposal] Failing a module's build if under a given TPC threshold
+1 with no vote
On Mon, Aug 20, 2012 at 11:08 PM, Caleb James DeLisle <
calebdelisle(a)lavabit.com> wrote:
+1 with the same concern as Sergiu.
Caleb
On 08/20/2012 03:28 PM, Sergiu Dumitriu wrote:
>
As part of improving XWiki's quality, the idea is that future commits in this
module shouldn't lower the module's quality.
>
One easy way to ensure this is to measure the current TPC level and make the build
fail whenever the build is under this TPC threshold.
>
This means that if you commit code that has less tests average than what currently
exists then the build will fail.
>
Here's one way to implement it:
>
> <!-- Fail the build if the test coverage is below a given value.
> Note: Since this takes a bit of time to execute we only run this when
the integration-tests profile is active -->
that
is too formal. By the rules, a vote should last 3 days, it should have
at least 3 +1, and no -1. But IMO a +1 means strong support, the voter
understands and agrees with the matter, and lowering the TPC is not
something on which people have a strong opinion. And the example that you
give requires a quick decision, the 3 days required to pass a vote should
be enough to write the missing tests.
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
On 08/20/2012 11:17 AM, Vincent Massol wrote:
> Hi devs,
>
> Each java module has a current TPC level (Test Percentage Coverage), be
it 5%
of 90%.
> <profiles>
> <profile>
> <id>integration-tests</id>
> <build>
> <plugins>
> <plugin>
> <groupId>com.atlassian.maven.plugins</groupId>
> <artifactId>maven-clover2-plugin</artifactId>
> <configuration>
> <targetPercentage>86.9%</targetPercentage>
> </configuration>
> <executions>
> <execution>
> <id>clover-check</id>
> <phase>verify</phase>
> <goals>
> <goal>instrument-test</goal>
> <goal>check</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
> </plugins>
> </build>
> </profile>
> </profiles>
>
> So this proposal is about the following strategy:
> * Allow committers to modify module's pom.xml to fail the build when
then
TPC is under the current threshold
> * Since oldcore is really difficult to test,
maybe we should not set
build failing for oldcore. OTOH oldcore is an important
module. WDYT?
> * If for some reason a committer wants to
lower the TPC threshold for a
module he needs to get the agreement from the other
committers (vote).
There might be valid reasons like fixing an important bug quickly for a
release and writing a test is just too complex so the writing of the test
is to be delayed for after the release, etc.
> * I'd like to consider this as an
experiment and see how it goes
>
> Note: This is about unit test TPC which is different from the TPC that
includes functional tests.
WDYT?
Here's my +1 to try this out and see how it goes.
+1.
I'm not sure that a [vote] is required, since that is usually something
So, I'd rather trust developers not to lower the threshold without a
valid
reason, and we can always signal a change that we don't agree with
afterwards.
8:47 a.m.
New subject: [xwiki-devs] [Proposal] Failing a module's build if under a given TPC threshold
On Tue, Aug 21, 2012 at 2:34 AM, Denis Gervalle <dgl(a)softec.lu> wrote:
+1 with no vote
Same, +1.
Thanks,
Marius
On Mon, Aug 20, 2012 at 11:08 PM, Caleb James DeLisle <
calebdelisle(a)lavabit.com> wrote:
+1 with the same concern as Sergiu.
Caleb
On 08/20/2012 03:28 PM, Sergiu Dumitriu wrote:
>
As part of improving XWiki's quality, the idea is that future commits in this
module shouldn't lower the module's quality.
>
One easy way to ensure this is to measure the current TPC level and make the build
fail whenever the build is under this TPC threshold.
>
This means that if you commit code that has less tests average than what currently
exists then the build will fail.
>
Here's one way to implement it:
>
> <!-- Fail the build if the test coverage is below a given value.
> Note: Since this takes a bit of time to execute we only run this when
the integration-tests profile is active -->
that
is too formal. By the rules, a vote should last 3 days, it should have
at least 3 +1, and no -1. But IMO a +1 means strong support, the voter
understands and agrees with the matter, and lowering the TPC is not
something on which people have a strong opinion. And the example that you
give requires a quick decision, the 3 days required to pass a vote should
be enough to write the missing tests.
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs On 08/20/2012 11:17 AM, Vincent Massol wrote:
> Hi devs,
>
> Each java module has a current TPC level (Test Percentage Coverage), be
it 5%
of 90%.
> <profiles>
> <profile>
> <id>integration-tests</id>
> <build>
> <plugins>
> <plugin>
> <groupId>com.atlassian.maven.plugins</groupId>
> <artifactId>maven-clover2-plugin</artifactId>
> <configuration>
> <targetPercentage>86.9%</targetPercentage>
> </configuration>
> <executions>
> <execution>
> <id>clover-check</id>
> <phase>verify</phase>
> <goals>
> <goal>instrument-test</goal>
> <goal>check</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
> </plugins>
> </build>
> </profile>
> </profiles>
>
> So this proposal is about the following strategy:
> * Allow committers to modify module's pom.xml to fail the build when
then
TPC is under the current threshold
> * Since oldcore is really difficult to test,
maybe we should not set
build failing for oldcore. OTOH oldcore is an important
module. WDYT?
> * If for some reason a committer wants to
lower the TPC threshold for a
module he needs to get the agreement from the other
committers (vote).
There might be valid reasons like fixing an important bug quickly for a
release and writing a test is just too complex so the writing of the test
is to be delayed for after the release, etc.
> * I'd like to consider this as an
experiment and see how it goes
>
> Note: This is about unit test TPC which is different from the TPC that
includes functional tests.
WDYT?
Here's my +1 to try this out and see how it goes.
+1.
I'm not sure that a [vote] is required, since that is usually something
So, I'd rather trust developers not to lower the threshold without a
valid
reason, and we can always signal a change that we don't agree with
afterwards.
9:52 a.m.
New subject: [xwiki-devs] [Proposal] Failing a module's build if under a given TPC threshold
On Aug 20, 2012, at 9:28 PM, Sergiu Dumitriu wrote:
On 08/20/2012 11:17 AM, Vincent Massol wrote:
ok, makes sense.
Thanks
-Vincent
Hi devs,
Each java module has a current TPC level (Test Percentage Coverage), be it 5% of 90%.
As part of improving XWiki's quality, the idea is that future commits in this module
shouldn't lower the module's quality.
One easy way to ensure this is to measure the current TPC level and make the build fail
whenever the build is under this TPC threshold.
This means that if you commit code that has less tests average than what currently exists
then the build will fail.
Here's one way to implement it:
<!-- Fail the build if the test coverage is below a given value.
Note: Since this takes a bit of time to execute we only run this when the
integration-tests profile is active -->
<profiles>
<profile>
<id>integration-tests</id>
<build>
<plugins>
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
<configuration>
<targetPercentage>86.9%</targetPercentage>
</configuration>
<executions>
<execution>
<id>clover-check</id>
<phase>verify</phase>
<goals>
<goal>instrument-test</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
So this proposal is about the following strategy:
* Allow committers to modify module's pom.xml to fail the build when then TPC is
under the current threshold
* Since oldcore is really difficult to test, maybe we should not set build failing for
oldcore. OTOH oldcore is an important module. WDYT?
* If for some reason a committer wants to lower the TPC threshold for a module he needs
to get the agreement from the other committers (vote). There might be valid reasons like
fixing an important bug quickly for a release and writing a test is just too complex so
the writing of the test is to be delayed for after the release, etc.
* I'd like to consider this as an experiment and see how it goes
Note: This is about unit test TPC which is different from the TPC that includes
functional tests.
WDYT?
Here's my +1 to try this out and see how it goes.
+1.
I'm not sure that a [vote] is required, since that is usually something that is too
formal. By the rules, a vote should last 3 days, it should have at least 3 +1, and no -1.
But IMO a +1 means strong support, the voter understands and agrees with the matter, and
lowering the TPC is not something on which people have a strong opinion. And the example
that you give requires a quick decision, the 3 days required to pass a vote should be
enough to write the missing tests.
So, I'd rather trust developers not to lower the threshold without a valid reason,
and we can always signal a change that we don't agree with afterwards.
24 Aug
24 Aug
11:31 a.m.
New subject: [xwiki-devs] [Proposal] Failing a module's build if under a given TPC threshold
On Aug 21, 2012, at 9:52 AM, Vincent Massol <vincent(a)massol.net> wrote:
On Aug 20, 2012, at 9:28 PM, Sergiu Dumitriu wrote:
Actually I've experimented a valid use case this morning.
When you add some code you have 3 choices for testing it: unit test, integration tests or
functional tests.
Unit tests and integration tests will count for the TPC threshold but not functional tests
ATM (we don't measure it in the build).
Thus when I add my code the build failed since my TPC % decreased. I could have written a
functional tests and then I'd have had to lower the TPC threshold value in the
pom.xml.
This is valid.
Thanks
-Vincent
On 08/20/2012 11:17 AM, Vincent Massol wrote:
ok, makes sense. Hi devs,
Each java module has a current TPC level (Test Percentage Coverage), be it 5% of 90%.
As part of improving XWiki's quality, the idea is that future commits in this module
shouldn't lower the module's quality.
One easy way to ensure this is to measure the current TPC level and make the build fail
whenever the build is under this TPC threshold.
This means that if you commit code that has less tests average than what currently exists
then the build will fail.
Here's one way to implement it:
<!-- Fail the build if the test coverage is below a given value.
Note: Since this takes a bit of time to execute we only run this when the
integration-tests profile is active -->
<profiles>
<profile>
<id>integration-tests</id>
<build>
<plugins>
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
<configuration>
<targetPercentage>86.9%</targetPercentage>
</configuration>
<executions>
<execution>
<id>clover-check</id>
<phase>verify</phase>
<goals>
<goal>instrument-test</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
So this proposal is about the following strategy:
* Allow committers to modify module's pom.xml to fail the build when then TPC is
under the current threshold
* Since oldcore is really difficult to test, maybe we should not set build failing for
oldcore. OTOH oldcore is an important module. WDYT?
* If for some reason a committer wants to lower the TPC threshold for a module he needs
to get the agreement from the other committers (vote). There might be valid reasons like
fixing an important bug quickly for a release and writing a test is just too complex so
the writing of the test is to be delayed for after the release, etc.
* I'd like to consider this as an experiment and see how it goes
Note: This is about unit test TPC which is different from the TPC that includes
functional tests.
WDYT?
Here's my +1 to try this out and see how it goes.
+1.
I'm not sure that a [vote] is required, since that is usually something that is too
formal. By the rules, a vote should last 3 days, it should have at least 3 +1, and no -1.
But IMO a +1 means strong support, the voter understands and agrees with the matter, and
lowering the TPC is not something on which people have a strong opinion. And the example
that you give requires a quick decision, the 3 days required to pass a vote should be
enough to write the missing tests.
So, I'd rather trust developers not to lower the threshold without a valid reason,
and we can always signal a change that we don't agree with afterwards.
12:03 p.m.
New subject: [xwiki-devs] [Proposal] Failing a module's build if under a given TPC threshold
On Aug 24, 2012, at 11:31 AM, Vincent Massol <vincent(a)massol.net> wrote:
On Aug 21, 2012, at 9:52 AM, Vincent Massol <vincent(a)massol.net> wrote:
One issue we have is that we're going to see a lot more Jenkins build failure since
the TPC computation is done when the integration-tests profile active and developers
usually don't run with it for speed reasons. So it means there can be a long time
between a commit and realizing it's breaking the build and fixing it…
The only solution I could think for this would be something like:
* Use another profile, say the existing "clover" one (or some new
"clovercheck" or "quality" or "tpc") and only check TPC in
that profile
* This means that all Jenkins build will work fine and won't disrupt the normal coding
cycles day to day
* Add a Jenkins job that runs once per night and that builds the whole code with the
Clover profile
This means it'll be even longer before you realize that the code you've added has
reduced the TPC but it has some advantages:
1) jenkins build will be faster for day to day dev which is very important IMO. Checking
TPC is actually quite long since it involves building the project twice
2) it won't prevent devs from working day to day
3) less jenkins build failure emails on the list
WDYT?
Thanks
-Vincent
On Aug 20, 2012, at 9:28 PM, Sergiu Dumitriu wrote:
Actually I've experimented a valid use case this morning.
When you add some code you have 3 choices for testing it: unit test, integration tests or
functional tests.
Unit tests and integration tests will count for the TPC threshold but not functional
tests ATM (we don't measure it in the build).
Thus when I add my code the build failed since my TPC % decreased. I could have written a
functional tests and then I'd have had to lower the TPC threshold value in the
pom.xml.
This is valid. On 08/20/2012 11:17 AM, Vincent Massol wrote:
ok, makes sense. Hi devs,
Each java module has a current TPC level (Test Percentage Coverage), be it 5% of 90%.
As part of improving XWiki's quality, the idea is that future commits in this module
shouldn't lower the module's quality.
One easy way to ensure this is to measure the current TPC level and make the build fail
whenever the build is under this TPC threshold.
This means that if you commit code that has less tests average than what currently exists
then the build will fail.
Here's one way to implement it:
<!-- Fail the build if the test coverage is below a given value.
Note: Since this takes a bit of time to execute we only run this when the
integration-tests profile is active -->
<profiles>
<profile>
<id>integration-tests</id>
<build>
<plugins>
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>maven-clover2-plugin</artifactId>
<configuration>
<targetPercentage>86.9%</targetPercentage>
</configuration>
<executions>
<execution>
<id>clover-check</id>
<phase>verify</phase>
<goals>
<goal>instrument-test</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
So this proposal is about the following strategy:
* Allow committers to modify module's pom.xml to fail the build when then TPC is
under the current threshold
* Since oldcore is really difficult to test, maybe we should not set build failing for
oldcore. OTOH oldcore is an important module. WDYT?
* If for some reason a committer wants to lower the TPC threshold for a module he needs
to get the agreement from the other committers (vote). There might be valid reasons like
fixing an important bug quickly for a release and writing a test is just too complex so
the writing of the test is to be delayed for after the release, etc.
* I'd like to consider this as an experiment and see how it goes
Note: This is about unit test TPC which is different from the TPC that includes
functional tests.
WDYT?
Here's my +1 to try this out and see how it goes.
+1.
I'm not sure that a [vote] is required, since that is usually something that is too
formal. By the rules, a vote should last 3 days, it should have at least 3 +1, and no -1.
But IMO a +1 means strong support, the voter understands and agrees with the matter, and
lowering the TPC is not something on which people have a strong opinion. And the example
that you give requires a quick decision, the 3 days required to pass a vote should be
enough to write the missing tests.
So, I'd rather trust developers not to lower the threshold without a valid reason,
and we can always signal a change that we don't agree with afterwards.
4410
days inactive
4414
days old
7 comments
5 participants
participants (5)
-
Caleb James DeLisle -
Denis Gervalle -
Marius Dumitru Florea -
Sergiu Dumitriu -
Vincent Massol