[xwiki-devs] [Proposal] Jobs for the multiple configuration testing
Hi devs, I’m still not sure but FTM I was thinking of having 2 pipeline jobs: 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable. 2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine. Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile. WDYT? Thanks -Vincent
On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
That's interesting indeed. +1
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Can't those configs be executed in parallel in different agents ?
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
-- Thomas Mortagne
On 07/11/2018 08:44, Thomas Mortagne wrote:
On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
That's interesting indeed. +1
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Can't those configs be executed in parallel in different agents ?
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
+1 for both jobs
Thanks -Vincent
-- Simon Urli Software Engineer at XWiki SAS [email protected] More about us at http://www.xwiki.com
Hi Thomas,
On 7 Nov 2018, at 08:44, Thomas Mortagne <[email protected]> wrote:
On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
That's interesting indeed. +1
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Can't those configs be executed in parallel in different agents ?
Yes this is fully parallelizable. However we have only one agent with docker ATM. Once we migrate more agents then we can have them in parallel. Thanks -Vincent
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
-- Thomas Mortagne
On Wed, Nov 7, 2018 at 9:20 AM Vincent Massol <[email protected]> wrote:
Hi Thomas,
On 7 Nov 2018, at 08:44, Thomas Mortagne <[email protected]> wrote:
On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
That's interesting indeed. +1
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Can't those configs be executed in parallel in different agents ?
Yes this is fully parallelizable. However we have only one agent with docker ATM. Once we migrate more agents then we can have them in parallel.
Indeed forgot about that. Anyway +1 for both
Thanks -Vincent
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
-- Thomas Mortagne
-- Thomas Mortagne
Hello, I'm +1 to try having both jobs. Thanks, Adel On Wed, Nov 7, 2018 at 9:20 AM Vincent Massol <[email protected]> wrote:
Hi Thomas,
On 7 Nov 2018, at 08:44, Thomas Mortagne <[email protected]> wrote:
On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
That's interesting indeed. +1
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Can't those configs be executed in parallel in different agents ?
Yes this is fully parallelizable. However we have only one agent with docker ATM. Once we migrate more agents then we can have them in parallel.
Thanks -Vincent
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
-- Thomas Mortagne
+1 for both but we need to push for more powerful CI agents. ATM the build is faster on my laptop. Thanks, Marius On Mon, Nov 5, 2018 at 6:22 PM Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
Hi devs, So we now have 2 dockers jobs: - one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ - another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all... All are passing FTM :) Thanks -Vincent
On 5 Nov 2018, at 17:22, Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
On 16 Nov 2018, at 22:03, Vincent Massol <[email protected]> wrote:
Hi devs,
So we now have 2 dockers jobs: - one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ - another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all...
And to see which configs are tested and passing or not: https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all... Thanks -Vincent
All are passing FTM :)
Thanks -Vincent
On 5 Nov 2018, at 17:22, Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
Ok I’ve created a 3rd job for unsupported configs: https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_uns... So we now have: * Job 1: run all tests on latest versions of supported configs * Job 2: run smoke tests (ie a few tests only) on all supported configs * Job 3: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc) Definitions are all here: https://github.com/xwiki/xwiki-jenkins-pipeline/blob/06b679bf511261dbd0125a3... Thanks -Vincent
On 18 Nov 2018, at 16:57, Vincent Massol <[email protected]> wrote:
On 16 Nov 2018, at 22:03, Vincent Massol <[email protected]> wrote:
Hi devs,
So we now have 2 dockers jobs: - one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ - another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all...
And to see which configs are tested and passing or not: https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all...
Thanks -Vincent
All are passing FTM :)
Thanks -Vincent
On 5 Nov 2018, at 17:22, Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
Self reminder: Just realized that I now need to modify the clover pipeline so that we now run docker-based tests too! This means the clover job can only run on agent4. This also means that I’ll need to setup FF on agent4 for non-docker-based functional tests. Thanks -Vincent
On 18 Nov 2018, at 17:49, Vincent Massol <[email protected]> wrote:
Ok I’ve created a 3rd job for unsupported configs: https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_uns...
So we now have: * Job 1: run all tests on latest versions of supported configs * Job 2: run smoke tests (ie a few tests only) on all supported configs * Job 3: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc)
Definitions are all here: https://github.com/xwiki/xwiki-jenkins-pipeline/blob/06b679bf511261dbd0125a3...
Thanks -Vincent
On 18 Nov 2018, at 16:57, Vincent Massol <[email protected]> wrote:
On 16 Nov 2018, at 22:03, Vincent Massol <[email protected]> wrote:
Hi devs,
So we now have 2 dockers jobs: - one that executes daily and runs the latest version of the supported configs. It runs all docker tests (right now that’s MenuIT but I’ll start converting some other tests and we need to convert them all). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ - another one that executes weekly and that runs all supported configs but only on smoke tests (right now that MenuIT only). See https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all...
And to see which configs are tested and passing or not: https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all...
Thanks -Vincent
All are passing FTM :)
Thanks -Vincent
On 5 Nov 2018, at 17:22, Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the maximum number of configurations, in order to flesh out configs that don’t start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat 9.x docker image uses java9+). The job would not send a mail on failure but it would update a report page (could even update a page on xwiki.org directly or if too complex update some page on maven.xwiki.org somewhere). This job would run not very often but say once per week. Note that one config takes 3-4 minutes to run, so 50 configs would take 3 hours which is acceptable.
2) Job 2: Execute all functional tests on a subset of supported configs. For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take a long time to execute. We’ll start with 3-4 configs and will go to about 10 configs when we add more. The tests will take roughly 2 hours to execute per config I think. So a total of 20 hours when we have 10 configs. If we run those once per week it should be fine.
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as part of the platform JenkinsFile.
WDYT?
Thanks -Vincent
participants (5)
-
Adel Atallah -
Marius Dumitru Florea -
Simon Urli -
Thomas Mortagne -
Vincent Massol