3 Jun
2011
3 Jun
'11
3:34 p.m.
On 06/02/2011 12:35 AM, Vincent Massol wrote:
On Jun 1, 2011, at 6:30 PM, Marius Dumitru Florea wrote:
On 05/31/2011 05:54 PM, Vincent Massol wrote:
Hi devs,
Here's the situation:
* We've been having a hard time releasing on time and the main issue is test
stability. We lag by at least a week and we even release with failing tests, causing
regressions.
* It's not the role of the release manager to fix tests before releasing
* It's not normal that some people spend time fixing issues caused by others and that
others continue to work on the next thing they are working on. Everyone needs to help.
Here's what I propose as a drastic and temporary measure till we get better:
1) It's forbidden to commit anything till all tests are passing (unit AND functional
tests), unless what is committed is about fixing tests. On the exceptional case when a
committer absolutely needs to commit even though tests are failing he needs to ask the
permission explicitly.
2) When tests are failing, everyone should stop what they're doing and help stabilize
again. We synchronize on IRC.
3) Flickering tests can be marked as @Ignore and
a jira issue created to stabilize the build.
Does this apply to Selenium 1 tests?
4) Release Manager creates a release branch 1
week before the release to let everyone stabilize the build
I think (1) and (2) are hard to achieve because tests take too long to
run. CI is at least 2h behind the latest commit. What are we supposed to
do while CI runs the tests? I think we should fix the real problem now:
add more agents and review the tests to see why they take so long to
run. I'm willing to spend time to make the WYSIWYG tests run faster. BTW on this topic XWiki SAS has purchased some new
build server today so that we can have several new agents on it (it should be able to run
4 agents). I'll let you know once it's set up.
That's great news! Should we set up a Selenium Grid (
http://selenium-grid.seleniumhq.org/ )?
Thanks,
Marius
Thanks
-Vincent
So +0.
Thanks,
Marius
>
> On a long term we need to work on improving our CI so that functional tests are built
faster. One idea is: more agents and functional tests spread on several agents.
>
> Here's my +1 to apply this now for master (3.2-SNAPSHOT leading to 3.2M1), which
means not committing anything more till we have all functional tests passing.
>
> Thanks
> -Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs