31 May
2011
31 May
'11
4:54 p.m.
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
2) When tests are failing, everyone should stop what they're doing and help stabilize
again. We synchronize on IRC.
3) Flickering tests can be marked as @Ignore and a jira issue created to stabilize the
build.
4) Release Manager creates a release branch 1 week before the release to let everyone
stabilize the build
On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), which means
not committing anything more till we have all functional tests passing.
Thanks
-Vincent
31 May
31 May
5:09 p.m.
New subject: [xwiki-devs] [VOTE] Important: Strategy to fix failing tests and stability
+1 on all counts.
In addition to more build engines we should investigate the build speed and how we can
improve it.
Caleb
On 05/31/2011 10:54 AM, Vincent Massol wrote:
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
2) When tests are failing, everyone should stop what they're doing and help stabilize
again. We synchronize on IRC.
3) Flickering tests can be marked as @Ignore and a jira issue created to stabilize the
build.
4) Release Manager creates a release branch 1 week before the release to let everyone
stabilize the build
On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), which
means not committing anything more till we have all functional tests passing.
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
7:22 p.m.
New subject: [xwiki-devs] [VOTE] Important: Strategy to fix failing tests and stability
+1 sounds good
On Tue, May 31, 2011 at 16:54, Vincent Massol <vincent(a)massol.net> wrote:
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
2) When tests are failing, everyone should stop what they're doing and help stabilize
again. We synchronize on IRC.
3) Flickering tests can be marked as @Ignore and a jira issue created to stabilize the
build.
4) Release Manager creates a release branch 1 week before the release to let everyone
stabilize the build
Is 4) really needed if everyone is supposed to help stabilize the
release anyway ?
On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), which
means not committing anything more till we have all functional tests passing.
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Thomas Mortagne
7:41 p.m.
New subject: [xwiki-devs] [VOTE] Important: Strategy to fix failing tests and stability
On May 31, 2011, at 7:22 PM, Thomas Mortagne wrote:
+1 sounds good
On Tue, May 31, 2011 at 16:54, Vincent Massol <vincent(a)massol.net> wrote:
I thought about it but put it as an extra safety measure but indeed if 1) works we
don't need it.
Thanks
-Vincent
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
2) When tests are failing, everyone should stop what they're doing and help stabilize
again. We synchronize on IRC.
3) Flickering tests can be marked as @Ignore and a jira issue created to stabilize the
build.
4) Release Manager creates a release branch 1 week before the release to let everyone
stabilize the build
Is 4) really needed if everyone is supposed to help stabilize the
release anyway ? > On a long term we need to work on improving our CI
so that functional tests are built faster. One idea is: more agents and functional tests
spread on several agents.
>
> Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), which
means not committing anything more till we have all functional tests passing.
>
> Thanks
> -Vincent
1 Jun
1 Jun
6:30 p.m.
On 05/31/2011 05:54 PM, Vincent Massol wrote:
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
2) When tests are failing, everyone should stop what they're doing and help stabilize
again. We synchronize on IRC.
3) Flickering tests can be marked as @Ignore and a
jira issue created to stabilize the build.
Does this apply to Selenium 1 tests?
4) Release Manager creates a release branch 1 week
before the release to let everyone stabilize the build
I think (1) and (2) are hard to achieve because tests take too long to
run. CI is at least 2h behind the latest commit. What are we supposed to
do while CI runs the tests? I think we should fix the real problem now:
add more agents and review the tests to see why they take so long to
run. I'm willing to spend time to make the WYSIWYG tests run faster.
So +0.
Thanks,
Marius
On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), which
means not committing anything more till we have all functional tests passing.
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
11:35 p.m.
New subject: [xwiki-devs] [VOTE] Important: Strategy to fix failing tests and stability
On Jun 1, 2011, at 6:30 PM, Marius Dumitru Florea wrote:
On 05/31/2011 05:54 PM, Vincent Massol wrote:
BTW on this topic XWiki SAS has purchased some new build server today so that we can have
several new agents on it (it should be able to run 4 agents). I'll let you know once
it's set up.
Thanks
-Vincent
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
2) When tests are failing, everyone should stop what they're doing and help stabilize
again. We synchronize on IRC.
3) Flickering tests can be marked as @Ignore and
a jira issue created to stabilize the build.
Does this apply to Selenium 1 tests?
4) Release Manager creates a release branch 1
week before the release to let everyone stabilize the build
I think (1) and (2) are hard to achieve because tests take too long to
run. CI is at least 2h behind the latest commit. What are we supposed to
do while CI runs the tests? I think we should fix the real problem now:
add more agents and review the tests to see why they take so long to
run. I'm willing to spend time to make the WYSIWYG tests run faster.
So +0.
Thanks,
Marius
>
> On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
>
> Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), which
means not committing anything more till we have all functional tests passing.
>
> Thanks
> -Vincent
3 Jun
3 Jun
3:34 p.m.
New subject: [xwiki-devs] [VOTE] Important: Strategy to fix failing tests and stability
On 06/02/2011 12:35 AM, Vincent Massol wrote:
On Jun 1, 2011, at 6:30 PM, Marius Dumitru Florea wrote:
On 05/31/2011 05:54 PM, Vincent Massol wrote:
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
2) When tests are failing, everyone should stop what they're doing and help stabilize
again. We synchronize on IRC.
3) Flickering tests can be marked as @Ignore and
a jira issue created to stabilize the build.
Does this apply to Selenium 1 tests?
4) Release Manager creates a release branch 1
week before the release to let everyone stabilize the build
I think (1) and (2) are hard to achieve because tests take too long to
run. CI is at least 2h behind the latest commit. What are we supposed to
do while CI runs the tests? I think we should fix the real problem now:
add more agents and review the tests to see why they take so long to
run. I'm willing to spend time to make the WYSIWYG tests run faster. BTW on this topic XWiki SAS has purchased some new
build server today so that we can have several new agents on it (it should be able to run
4 agents). I'll let you know once it's set up.
That's great news! Should we set up a Selenium Grid (
http://selenium-grid.seleniumhq.org/ )?
Thanks,
Marius
Thanks
-Vincent
So +0.
Thanks,
Marius
>
> On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
>
> Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), which
means not committing anything more till we have all functional tests passing.
>
> Thanks
> -Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs 
6 Jun
6 Jun
11:20 a.m.
On 05/31/2011 04:54 PM, Vincent Massol wrote:
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
-0. Running all tests takes a lot of time, we can't wait 2 hours after
each change to check that it works before committing.
-1 for forcing everyone to stop their work when somebody else broke some
tests.
+1 for a different strategy: Whoever commits something should check the
next build, and if something fails, he should stop what he's doing and
work on fixing the problem. The build should not stay broken for more
than 6 working hours.
2) When tests are failing, everyone should stop what
they're doing and help stabilize again. We synchronize on IRC.
+1, but for this to work we have to reach a stable state by default,
meaning that a failed build should be a rare exception.
3) Flickering tests can be marked as @Ignore and a
jira issue created to stabilize the build.
+1
4) Release Manager creates a release branch 1 week
before the release to let everyone stabilize the build
+0, the term should not be strict. If all tests are passing, then
there's no need for such a long stabilization period. If releases should
happen on Monday, then the branch could be created Friday morning.
On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
+1.
Here's my +1 to apply this now for master
(3.2-SNAPSHOT leading to 3.2M1), which means not committing anything more till we have all
functional tests passing.
Thanks
-Vincent
--
Sergiu Dumitriu
http://purl.org/net/sergiu/
11:30 a.m.
New subject: [xwiki-devs] [VOTE] Important: Strategy to fix failing tests and stability
On Jun 6, 2011, at 11:20 AM, Șergiu Dumitriu wrote:
On 05/31/2011 04:54 PM, Vincent Massol wrote:
I didn't mean to wait. I meant to check ci.xwiki.org and verify that the last build
was succesful.
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
-0. Running all tests takes a lot of time, we can't wait 2 hours after
each change to check that it works before committing. -1 for forcing everyone to stop their work when
somebody else broke some
tests.
+1 for a different strategy: Whoever commits something should check the
next build, and if something fails, he should stop what he's doing and
work on fixing the problem. The build should not stay broken for more
than 6 working hours.
It's not just the next build, it's all other builds since tests can flicker. Also
it can take time for a CI error to bubble up (may require several builds).
It's not fun to fix tests and even less fun to fix other's tests. Especially when
other people are having fun working on the next stuff...
We also need to stabilize and make tests passing the norm.
So right now nobody should commit anything till all tests are passing and consistently.
This is not the case.
Thanks
-Vincent
2) When tests are failing, everyone should stop
what they're doing and help stabilize again. We synchronize on IRC.
+1, but for this to work we have to reach a stable state by default,
meaning that a failed build should be a rare exception.
3) Flickering tests can be marked as @Ignore and
a jira issue created to stabilize the build.
+1
4) Release Manager creates a release branch 1
week before the release to let everyone stabilize the build
+0, the term should not be strict. If all tests are passing, then
there's no need for such a long stabilization period. If releases should
happen on Monday, then the branch could be created Friday morning.
On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
+1.
Here's my +1 to apply this now for master
(3.2-SNAPSHOT leading to 3.2M1), which means not committing anything more till we have all
functional tests passing.
Thanks
-Vincent
--
Sergiu Dumitriu
http://purl.org/net/sergiu/
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
4851
days inactive
4857
days old
8 comments
5 participants
participants (5)
-
Caleb James DeLisle -
Marius Dumitru Florea -
Thomas Mortagne -
Vincent Massol -
Șergiu Dumitriu