[xwiki-devs] [Proposal] Stopping the 4.4M1 release till tests are all passing - Commando mode - xwiki-devs@xwiki.org

Marius Dumitru Florea

13 Dec 13 Dec

5:54 p.m.

New subject: [xwiki-devs] [Proposal] Stopping the 4.4M1 release till tests are all passing - Commando mode

Big +1, obviously. Let's do it! Marius On Thu, Dec 13, 2012 at 6:42 PM, Vincent Massol <vincent(a)massol.net> wrote:

Hi devs, We have too many test failures on http://ci.xwiki.org/view/Functional%20Tests/ and too many emails sent by Jenkins on the list. It has become a nightmare and it's impossible to perform a release anymore with a good confidence it's going to work. This is all the more bad that we're ending the 4.x cycle. Thus I propose to do the following: * Don't release 4.4M1 till all tests are passing with no more flickers (say the tests should all pass during 10 full builds for example) * Create a Commando unit in charge of solving the flickers. Since I've already discussed this with Marius I propose that Marius and myself be the first 2 members. If anyone else would like to help please reply to this mail and join us. * This commando unit gives itself 1 full week to solve the flickers (ie till the 21st of December). We'll decide what to do next if we fail to achieve our goal after that deadline. * We start by creating a branch for 4.4M1 so that we isolate ourselves from the rest of the devs who continue to work for 4.4RC1 (reminder: only important bug fixes should go in 4.4RC1) * When we have fixed all flickers on the 4.4M1 branch we merge the changes to both master and the stable-4.3 branch * At the end of next week we also propose a strategy so that this mess doesn't happen again in the future WDYT? Thanks -Vincent Note: We need to release 4.3.1 ASAP so this strategy above will not apply to 4.3.1. For 4.3.1 Edy will need to figure out if all the failing tests are real issues or test issues. I think Edy could do this by a combination of running them locally and doing some manual tests where they also fail locally. Edy WDYT? _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs

Reply

Sergiu Dumitriu

6:05 p.m.

New subject: [xwiki-devs] [Proposal] Stopping the 4.4M1 release till tests are all passing - Commando mode

On 12/13/2012 12:00 PM, Eduard Moraru wrote:

...

Hi, On Thu, Dec 13, 2012 at 6:42 PM, Vincent Massol <vincent(a)massol.net> wrote:

Hi devs, We have too many test failures on http://ci.xwiki.org/view/Functional%20Tests/ and too many emails sent by Jenkins on the list. It has become a nightmare and it's impossible to perform a release anymore with a good confidence it's going to work. This is all the more bad that we're ending the 4.x cycle. Thus I propose to do the following: * Don't release 4.4M1 till all tests are passing with no more flickers (say the tests should all pass during 10 full builds for example) * Create a Commando unit in charge of solving the flickers. Since I've already discussed this with Marius I propose that Marius and myself be the first 2 members. If anyone else would like to help please reply to this mail and join us. * This commando unit gives itself 1 full week to solve the flickers (ie till the 21st of December). We'll decide what to do next if we fail to achieve our goal after that deadline. * We start by creating a branch for 4.4M1 so that we isolate ourselves from the rest of the devs who continue to work for 4.4RC1 (reminder: only important bug fixes should go in 4.4RC1) * When we have fixed all flickers on the 4.4M1 branch we merge the changes to both master and the stable-4.3 branch * At the end of next week we also propose a strategy so that this mess doesn't happen again in the future WDYT?

+1 and, as I`ve already mentioned it to Vincent, maybe this gives us the opportunity to improve our release process/scripts to be able to release from a "release branch" instead of blocking "master" for the whole period of the release.

The release script already does that. I mean, when starting the release it creates a local release branch, but that branch gets discarded at the end. Do you want to create the release branch beforehand, so that devs can stabilize the build in that release branch? I'm against this, since it contradicts another important principle that we're supposed to follow: the build should always work! There is no good reason why there would be a need for stabilization. That's why we have continuous integration.

...

Thanks -Vincent Note: We need to release 4.3.1 ASAP so this strategy above will not apply to 4.3.1. For 4.3.1 Edy will need to figure out if all the failing tests are real issues or test issues. I think Edy could do this by a combination of running them locally and doing some manual tests where they also fail locally. Edy WDYT?

Since I`ve already agreed to being RM for 4.3.1, I guess that this was already part of the work, additional to the help from the owners of the problematic modules on which I`m counting on :) Thanks, Eduard

_______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs

-- Sergiu Dumitriu http://purl.org/net/sergiu

Reply

Vincent Massol

6:34 p.m.

New subject: [xwiki-devs] [Proposal] Stopping the 4.4M1 release till tests are all passing - Commando mode

Hi Denis, On Dec 13, 2012, at 6:06 PM, Denis Gervalle <dgl(a)softec.lu> wrote:

...

On Thu, Dec 13, 2012 at 5:42 PM, Vincent Massol <vincent(a)massol.net> wrote:

Hi devs, We have too many test failures on http://ci.xwiki.org/view/Functional%20Tests/ and too many emails sent by Jenkins on the list. It has become a nightmare and it's impossible to perform a release anymore with a good confidence it's going to work. This is all the more bad that we're ending the 4.x cycle. Thus I propose to do the following: * Don't release 4.4M1 till all tests are passing with no more flickers (say the tests should all pass during 10 full builds for example) * Create a Commando unit in charge of solving the flickers. Since I've already discussed this with Marius I propose that Marius and myself be the first 2 members. If anyone else would like to help please reply to this mail and join us. * This commando unit gives itself 1 full week to solve the flickers (ie till the 21st of December). We'll decide what to do next if we fail to achieve our goal after that deadline. * We start by creating a branch for 4.4M1 so that we isolate ourselves from the rest of the devs who continue to work for 4.4RC1 (reminder: only important bug fixes should go in 4.4RC1) * When we have fixed all flickers on the 4.4M1 branch we merge the changes to both master and the stable-4.3 branch * At the end of next week we also propose a strategy so that this mess doesn't happen again in the future WDYT? Thanks -Vincent Note: We need to release 4.3.1 ASAP so this strategy above will not apply to 4.3.1. For 4.3.1 Edy will need to figure out if all the failing tests are real issues or test issues. I think Edy could do this by a combination of running them locally and doing some manual tests where they also fail locally. Edy WDYT?

There are still a blocker issue for 4.3.1, shouldn't we also have that solve first ? There is also a lot of other issue marked to be fix, I suppose Edy will triage, and if some are important, these should be fixed first. I am just afraid, because most of these issue are assigned to Marius, and IMO it is more important to focus on the 4.3.1 release than on the 4.4M1 one.

At this stage it's more important to release 4.3.1 ASAP and we can do a 4.3.2 after for less important bugs if we want but I don't think it'll be needed because we'll be released 4.4 which is a stabilization/bug fix release and same for 4.5 so we'll recommend users to upgrade to 4.4 and then 4.5 and not 4.3.1 or 4.3.2... If you know of some issues that you consider blocking for 4.3.1 please raise them in a different thread and we can decide if they warrant postponing 4.3.1 more. Thanks -Vincent PS: I suggest you be the next release manager so that you can really decide if it's more important to fix the build or fix bugs… ;) There are some committers that don't help much in fixing the build so it's quite easy if you've not been doing this to consider that the build isn't important.. The most important is to have a stable build. That comes first. Even before bugs because it's a prerequisite for fixing bugs.

Reply

Arvind Gupta

6:43 p.m.

New subject: [xwiki-devs] [Proposal] Stopping the 4.4M1 release till tests are all passing - Commando mode

Denis you can consider me as a resource, would like to contribute in 4.3.1 as well as other tasks. -arvind On Thu, Dec 13, 2012 at 11:04 PM, Vincent Massol <vincent(a)massol.net> wrote:

...

Hi Denis, On Dec 13, 2012, at 6:06 PM, Denis Gervalle <dgl(a)softec.lu> wrote:

On Thu, Dec 13, 2012 at 5:42 PM, Vincent Massol <vincent(a)massol.net>

wrote:

> Hi devs, > > We have too many test failures on > http://ci.xwiki.org/view/Functional%20Tests/ and too many emails sent

by

> Jenkins on the list. > > It has become a nightmare and it's impossible to perform a release

anymore

> with a good confidence it's going to work. > > This is all the more bad that we're ending the 4.x cycle. > > Thus I propose to do the following: > * Don't release 4.4M1 till all tests are passing with no more flickers > (say the tests should all pass during 10 full builds for example) > * Create a Commando unit in charge of solving the flickers. Since I've > already discussed this with Marius I propose that Marius and myself be

the

> first 2 members. If anyone else would like to help please reply to this > mail and join us. > * This commando unit gives itself 1 full week to solve the flickers (ie > till the 21st of December). We'll decide what to do next if we fail to > achieve our goal after that deadline. > * We start by creating a branch for 4.4M1 so that we isolate ourselves > from the rest of the devs who continue to work for 4.4RC1 (reminder:

only

> important bug fixes should go in 4.4RC1) > * When we have fixed all flickers on the 4.4M1 branch we merge the

changes

> to both master and the stable-4.3 branch > * At the end of next week we also propose a strategy so that this mess > doesn't happen again in the future > > WDYT? > > Thanks > -Vincent > > Note: We need to release 4.3.1 ASAP so this strategy above will not

apply

> to 4.3.1. For 4.3.1 Edy will need to figure out if all the failing tests > are real issues or test issues. I think Edy could do this by a

combination

of running them locally and doing some manual tests where they also fail locally. Edy WDYT?

There are still a blocker issue for 4.3.1, shouldn't we also have that solve first ? There is also a lot of other issue marked to be fix, I suppose Edy will triage, and if some are important, these should be fixed first. I am just afraid, because most of these issue are assigned to Marius, and IMO it is more important to focus on the 4.3.1 release than

on

the 4.4M1 one.

At this stage it's more important to release 4.3.1 ASAP and we can do a 4.3.2 after for less important bugs if we want but I don't think it'll be needed because we'll be released 4.4 which is a stabilization/bug fix release and same for 4.5 so we'll recommend users to upgrade to 4.4 and then 4.5 and not 4.3.1 or 4.3.2... If you know of some issues that you consider blocking for 4.3.1 please raise them in a different thread and we can decide if they warrant postponing 4.3.1 more. Thanks -Vincent PS: I suggest you be the next release manager so that you can really decide if it's more important to fix the build or fix bugs… ;) There are some committers that don't help much in fixing the build so it's quite easy if you've not been doing this to consider that the build isn't important.. The most important is to have a stable build. That comes first. Even before bugs because it's a prerequisite for fixing bugs. _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs

Reply

Sergiu Dumitriu

13 Dec 13 Dec

6:11 p.m.

New subject: [xwiki-devs] [Proposal] Stopping the 4.4M1 release till tests are all passing - Commando mode

On 12/13/2012 11:42 AM, Vincent Massol wrote:

...

Hi devs, We have too many test failures on http://ci.xwiki.org/view/Functional%20Tests/ and too many emails sent by Jenkins on the list. It has become a nightmare and it's impossible to perform a release anymore with a good confidence it's going to work. This is all the more bad that we're ending the 4.x cycle. Thus I propose to do the following: * Don't release 4.4M1 till all tests are passing with no more flickers (say the tests should all pass during 10 full builds for example) * Create a Commando unit in charge of solving the flickers. Since I've already discussed this with Marius I propose that Marius and myself be the first 2 members. If anyone else would like to help please reply to this mail and join us. * This commando unit gives itself 1 full week to solve the flickers (ie till the 21st of December). We'll decide what to do next if we fail to achieve our goal after that deadline. * We start by creating a branch for 4.4M1 so that we isolate ourselves from the rest of the devs who continue to work for 4.4RC1 (reminder: only important bug fixes should go in 4.4RC1) * When we have fixed all flickers on the 4.4M1 branch we merge the changes to both master and the stable-4.3 branch * At the end of next week we also propose a strategy so that this mess doesn't happen again in the future WDYT?

I've looked at the build failures earlier, and most of them seem to be false alarms. A few failed with a SocketTimedOut exception, which means that either the machine (agent) is flaky, or that XWiki is getting too slow (performance issue). Some tests failed with a missing body, which again suggests a network problem (empty response?). Since most agents share the same physical hardware, maybe these failures just mean that we're putting too much load on the CPU/RAM/disk, and we should reduce the number of agents.

...

Thanks -Vincent Note: We need to release 4.3.1 ASAP so this strategy above will not apply to 4.3.1. For 4.3.1 Edy will need to figure out if all the failing tests are real issues or test issues. I think Edy could do this by a combination of running them locally and doing some manual tests where they also fail locally. Edy WDYT?

-- Sergiu Dumitriu http://purl.org/net/sergiu

Reply