[xwiki-devs] [VOTE] Skip tests while building a release
Hi devs, Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build). Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release. Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well. Here's my +1. Thanks, Eduard
On 12/19/2012 12:29 PM, Eduard Moraru wrote:
Hi devs,
Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build).
Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release.
Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well.
Here's my +1.
+1. Initially, only unit tests were present in the CRP projects (Commons, Rendering and Platform), so the release script needed to skip tests (integration) just for XE and XEM. And the original reason for skipping these tests was also the fact that tests were flickering. -- Sergiu Dumitriu http://purl.org/net/sergiu
+1, makes sense. On Wed, Dec 19, 2012 at 6:52 PM, Sergiu Dumitriu <[email protected]> wrote:
On 12/19/2012 12:29 PM, Eduard Moraru wrote:
Hi devs,
Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build).
Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release.
Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well.
Here's my +1.
+1. Initially, only unit tests were present in the CRP projects (Commons, Rendering and Platform), so the release script needed to skip tests (integration) just for XE and XEM. And the original reason for skipping these tests was also the fact that tests were flickering.
-- Sergiu Dumitriu http://purl.org/net/sergiu _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
-1 for unit tests (they shouldn't be flickering) +1 for functional tests Thanks, Marius On Wed, Dec 19, 2012 at 7:29 PM, Eduard Moraru <[email protected]> wrote:
Hi devs,
Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build).
Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release.
Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well.
Here's my +1.
Thanks, Eduard _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Marius, but why run even the Unit tests? The build is already tested continuously (both unit and functional), why waste more time during the release? Thanks, Eduard On Thu, Dec 20, 2012 at 12:18 AM, Marius Dumitru Florea < [email protected]> wrote:
-1 for unit tests (they shouldn't be flickering) +1 for functional tests
Thanks, Marius
On Wed, Dec 19, 2012 at 7:29 PM, Eduard Moraru <[email protected]> wrote:
Hi devs,
Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build).
Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release.
Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well.
Here's my +1.
Thanks, Eduard _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Thu, Dec 20, 2012 at 1:23 AM, Eduard Moraru <[email protected]> wrote:
Marius, but why run even the Unit tests? The build is already tested continuously (both unit and functional), why waste more time during the release?
Just to be safe. Mistakes can always occur and the time gained by skipping the unit tests (btw, do you have an estimate of the time we gain by skipping unit tests?) is less important IMO than the insecurity we introduce by skipping the unit tests. Let's see what others think. Thanks, Marius
Thanks, Eduard
On Thu, Dec 20, 2012 at 12:18 AM, Marius Dumitru Florea < [email protected]> wrote:
-1 for unit tests (they shouldn't be flickering) +1 for functional tests
Thanks, Marius
On Wed, Dec 19, 2012 at 7:29 PM, Eduard Moraru <[email protected]> wrote:
Hi devs,
Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build).
Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release.
Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well.
Here's my +1.
Thanks, Eduard _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On 12/19/2012 06:48 PM, Marius Dumitru Florea wrote:
On Thu, Dec 20, 2012 at 1:23 AM, Eduard Moraru <[email protected]> wrote:
Marius, but why run even the Unit tests? The build is already tested continuously (both unit and functional), why waste more time during the release?
Just to be safe. Mistakes can always occur and the time gained by skipping the unit tests (btw, do you have an estimate of the time we gain by skipping unit tests?) is less important IMO than the insecurity we introduce by skipping the unit tests. Let's see what others think.
I'm -ε for skipping unit tests, however I strongly believe that we should switch to staged releases from automatic CI builds, where the release is just a build promotion, and in this case all tests should be enabled since the build is automatic and happens before the release process starts. Until then, I think that safety is more important, given that unit tests actually take a lot less than integration tests.
Thanks, Marius
Thanks, Eduard
On Thu, Dec 20, 2012 at 12:18 AM, Marius Dumitru Florea < [email protected]> wrote:
-1 for unit tests (they shouldn't be flickering) +1 for functional tests
Thanks, Marius
On Wed, Dec 19, 2012 at 7:29 PM, Eduard Moraru <[email protected]> wrote:
Hi devs,
Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build).
Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release.
Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well.
Here's my +1.
Thanks, Eduard
-- Sergiu Dumitriu http://purl.org/net/sergiu/
Hi Edy, On Dec 19, 2012, at 6:29 PM, Eduard Moraru <[email protected]> wrote:
Hi devs,
Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build).
Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release.
Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well.
The build that is released should always have had all tests passed on it so Im' not a bit fan of this option. A better way is to do build promotion, i.e. release from jenkins. There are various jenkins plugin for this. The way I view it is like this: * once a day we run the release job (starting after RC is released usually), which will build the whole stack (from commons to functional tests) using the target version and we push it to a staging repo on nexus * this release job can also be triggered manually * we start to only release like this and we thus always release with a perfectly passing build * as soon as there is a promoted build during sorin and manuel could do some smoke testing on it and we can advertise the build on our lists and give it 2-3 days before we move from staging to target repo. I don't think this is hard to set up and I could work on this after I finish with false positives emails. Thanks -Vincent
On Dec 20, 2012, at 8:30 AM, Vincent Massol <[email protected]> wrote:
Hi Edy,
On Dec 19, 2012, at 6:29 PM, Eduard Moraru <[email protected]> wrote:
Hi devs,
Well, the subject pretty much sums it up. I am proposing to skip any testing while building a release, since that is the job of the CI infrastructure and, it happened more than once for the build of the release to fail because of a flickering test. This is really annoying for a release manager that has to start over the entire build (since right now it's a real pain to alter the scripts to continue the build).
Besides this, tests (specially the functional ones added into platform) uselessly delay the build, thus, the release.
Note: We are already skipping tests for enterprise and manager. This vote is actually about skipping tests for commons, rendering and platform as well.
The build that is released should always have had all tests passed on it so Im' not a bit fan of this option.
That said, a few weeks ago, I modified our release script to skip tests for platform :)
A better way is to do build promotion, i.e. release from jenkins. There are various jenkins plugin for this.
The way I view it is like this: * once a day we run the release job (starting after RC is released usually), which will build the whole stack (from commons to functional tests) using the target version and we push it to a staging repo on nexus * this release job can also be triggered manually * we start to only release like this and we thus always release with a perfectly passing build * as soon as there is a promoted build during sorin and manuel could do some smoke testing on it and we can advertise the build on our lists and give it 2-3 days before we move from staging to target repo.
I don't think this is hard to set up and I could work on this after I finish with false positives emails.
I really think we should work on this. I'll try to find some time; it would save us a lot of time. We should probably use https://wiki.jenkins-ci.org/display/JENKINS/Promoted+Builds+Plugin Thanks -Vincent
participants (5)
-
Denis Gervalle -
Eduard Moraru -
Marius Dumitru Florea -
Sergiu Dumitriu -
Vincent Massol