[xwiki-devs] [Brainstorming] Colocated Functional Test (CFT)
Hi devs, I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE. For example for the linkchecker module we have the following directories: xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests) The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too Here are some findings about this experiment: A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module Solutions/ideas: * One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only * Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it). I think the first proposal is the best one and brings the best of the 2 worlds. WDYT? Thanks -Vincent
On Sun, Jan 15, 2012 at 12:27 PM, Vincent Massol <[email protected]> wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup:
** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
+1 for this. Thanks, Marius
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
WDYT?
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Sun, Jan 15, 2012 at 11:27 AM, Vincent Massol <[email protected]> wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
It would be also be easy for an application developer to copy and existing test module (or ideally to use an archetype) and write tests for his application. I had to copy the entire distribution to achieve this in the past.
[snip]
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
Indeed the first proposal sounds like the best. Considering the small number of XE instance it is reasonable to run on an average machine the second one sounds overkill. JV.
Hi, As anyone who went trough the hassle of creating manually a whole distribution for the sake of having functional tests for an application, I can only be enthusiastic about this proposition :) Big +1 Another topic would be how do we make it easier/faster to write tests ? I'm playing with the thought that it would be easier to write functional tests in groovy (maybe not the page objects, but the tests themselves). Or maybe something like geb (http://www.gebish.org/) all the way would make sense in that direction. Another thing I would like to dig some day would be the possibility to write tests in the wiki application itself and then run them either from the browser or from maven. That might not be easy to achieve though, since it requires starting another browser with a clean profile... from the browser itself. It could be done with the help of a RC daemon on the local machine, then the browser sends commands to that daemon, which in turn runs commands WebDriver. Something in the spirit of jasmine maven integration, where you start a jasmine daemon with mvn jasmine:bdd (http://searls.github.com/jasmine-maven-plugin/) Food for thought. Jerome On Sun, Jan 15, 2012 at 11:27 AM, Vincent Massol <[email protected]> wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
WDYT?
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Jérôme Velociter Winesquare http://www.winesquare.net/
Hi Jerome, On Jan 17, 2012, at 3:48 PM, Jerome Velociter wrote:
Hi,
As anyone who went trough the hassle of creating manually a whole distribution for the sake of having functional tests for an application, I can only be enthusiastic about this proposition :)
Big +1
Another topic would be how do we make it easier/faster to write tests ? I'm playing with the thought that it would be easier to write functional tests in groovy (maybe not the page objects, but the tests themselves). Or maybe something like geb (http://www.gebish.org/) all the way would make sense in that direction. Another thing I would like to dig some day would be the possibility to write tests in the wiki application itself and then run them either from the browser or from maven. That might not be easy to achieve though, since it requires starting another browser with a clean profile... from the browser itself. It could be done with the help of a RC daemon on the local machine, then the browser sends commands to that daemon, which in turn runs commands WebDriver. Something in the spirit of jasmine maven integration, where you start a jasmine daemon with mvn jasmine:bdd (http://searls.github.com/jasmine-maven-plugin/) Food for thought.
I'd like that too. I even started an experiment back in 2007 (time flies!): http://extensions.xwiki.org/xwiki/bin/view/Extension/Selenium+Application The only problem is that "Make it fast to write test" is not really compatible with "Write the tests inside the wiki". The main problem is that or wiki is not yet a full-fledged IDE and we don't have syntax coloring but more importantly auto-completion, ability to easily search in code and follow methods calls, find usages, etc So right now writing a test in the IDE is much faster than in the browser. Note that with our Page Objects strategy it's really easy to write a test in java using our "DSL". Unfortunately we often have to add new PO since often new tests are about new features… ;) Note also that I've tried in the past to use recording tool and so far I've not found one that worked well. They always had issues. It's easy to understand. Imagine for example the following scenario: you got to a page and then click on the attachment tab at the bottom. The recorded script will do just that… But when executed it'll fail very often. Why? Because the load of the attachment tab is asynchronous and you need to wait for it to be loaded before being able to assert its content. So recording helps a bit but the problem is that usually the generated script is hard to read and you need to fully understand it in order to be able to fix the script so that it works. All in all it in lots of cases (if not all) it takes longer than to write the test from the beginning… I'm all for doing experiments though :) We just need to address the issues mentioned above. Thanks -Vincent PS: Note that we have a license for Greenpepper which does exactly that: writing tests in XWiki ;) http://www.greenpeppersoftware.com/confluence/display/GPW/Home/
Jerome
On Sun, Jan 15, 2012 at 11:27 AM, Vincent Massol <[email protected]> wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
WDYT?
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Jérôme Velociter Winesquare http://www.winesquare.net/ _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Jan 15, 2012, at 11:27 AM, Vincent Massol wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
Actually there's another one: * Run functional tests in platform * Configure jenkins so that we have one jenkins job per platform top module This would allow several things: * Faster feedback to user: for example if a commit is done in a module that happens late in the build order the whole platform wouldn't need to be rebuilt before noticing the problem) - Note: Jenkins supports a "Local subdirectory for repo" feature that would make this possible I think) * Use the full power of our agents by distributing the jobs * Since functional tests run at the same time as unit tests if the module build works we can be sure it's good Thus even though packaging and starting XE takes a lot of time (I've timed it to roughly 1.5 minutes on my machine) the balancing of jobs on multiple agents might overcome this - it would still be great to be able to improve the XE start time. Note that creating jenkins job for so many modules might seem daunting but this can be easily automated with http://ci.xwiki.org/api/ or even better http://evgeny-goldin.com/wiki/Maven-jenkins-plugin I'm still hesitating between the previous idea and this one but thought I would share my current thoughts with you in case you have some idea/preference… :) Thanks -Vincent
WDYT?
Thanks -Vincent
On Jan 25, 2012, at 5:02 PM, Vincent Massol wrote:
On Jan 15, 2012, at 11:27 AM, Vincent Massol wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
Actually there's another one:
* Run functional tests in platform * Configure jenkins so that we have one jenkins job per platform top module
This would allow several things: * Faster feedback to user: for example if a commit is done in a module that happens late in the build order the whole platform wouldn't need to be rebuilt before noticing the problem) - Note: Jenkins supports a "Local subdirectory for repo" feature that would make this possible I think)
After researching this a bit I don't think it's possible… This "Local subdirectory for repo" is something else: "Specify a local directory (relative to the workspace root) where the Git repository will be checked out. If left empty, the workspace root itself will be used." We could check out the whole Git repo but 1) it'll take space (a lot of space) but more importantly 2) GitHub will trigger a build on all jobs for that repo which isn't good. So I think we'll have to drop the idea of having a single job per module. See also http://stackoverflow.com/questions/8922857/jenkins-how-to-build-multiple-top... Thanks -Vincent
* Use the full power of our agents by distributing the jobs * Since functional tests run at the same time as unit tests if the module build works we can be sure it's good
Thus even though packaging and starting XE takes a lot of time (I've timed it to roughly 1.5 minutes on my machine) the balancing of jobs on multiple agents might overcome this - it would still be great to be able to improve the XE start time.
Note that creating jenkins job for so many modules might seem daunting but this can be easily automated with http://ci.xwiki.org/api/ or even better http://evgeny-goldin.com/wiki/Maven-jenkins-plugin
I'm still hesitating between the previous idea and this one but thought I would share my current thoughts with you in case you have some idea/preference… :)
Thanks -Vincent
WDYT?
Thanks -Vincent
On Feb 15, 2012, at 4:10 PM, Vincent Massol wrote:
On Jan 25, 2012, at 5:02 PM, Vincent Massol wrote:
On Jan 15, 2012, at 11:27 AM, Vincent Massol wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
Actually there's another one:
* Run functional tests in platform * Configure jenkins so that we have one jenkins job per platform top module
This would allow several things: * Faster feedback to user: for example if a commit is done in a module that happens late in the build order the whole platform wouldn't need to be rebuilt before noticing the problem) - Note: Jenkins supports a "Local subdirectory for repo" feature that would make this possible I think)
After researching this a bit I don't think it's possible…
This "Local subdirectory for repo" is something else: "Specify a local directory (relative to the workspace root) where the Git repository will be checked out. If left empty, the workspace root itself will be used."
We could check out the whole Git repo but 1) it'll take space (a lot of space) but more importantly 2) GitHub will trigger a build on all jobs for that repo which isn't good.
So I think we'll have to drop the idea of having a single job per module.
See also http://stackoverflow.com/questions/8922857/jenkins-how-to-build-multiple-top...
hmm actually this is what we already do for functional tests but we don't trigger than on Git Hub notifications, we trigger them on xwiki-enterprise (after it's been built). I guess we could do the same, i.e.: * Only trigger the top level module in a given repo using a GitHub push * Only trigger the other modules on "Build whenever a SNAPSHOT dependency is built" The problem is that it would still use a lot of space. For example my xwiki-platform/ dir takes 1.2GB of disk space (with target/ dirs) So with roughly 230+ modules that's 276 GB just for xwiki-platform… Combined with commons, rendering and enterprise that's probably around 500GB easily… In conclusion it doesn't seem like a good idea… Thanks -Vincent
Thanks -Vincent
* Use the full power of our agents by distributing the jobs * Since functional tests run at the same time as unit tests if the module build works we can be sure it's good
Thus even though packaging and starting XE takes a lot of time (I've timed it to roughly 1.5 minutes on my machine) the balancing of jobs on multiple agents might overcome this - it would still be great to be able to improve the XE start time.
Note that creating jenkins job for so many modules might seem daunting but this can be easily automated with http://ci.xwiki.org/api/ or even better http://evgeny-goldin.com/wiki/Maven-jenkins-plugin
I'm still hesitating between the previous idea and this one but thought I would share my current thoughts with you in case you have some idea/preference… :)
Thanks -Vincent
WDYT?
Thanks -Vincent
On Feb 15, 2012, at 4:58 PM, Vincent Massol <[email protected]> wrote:
On Feb 15, 2012, at 4:10 PM, Vincent Massol wrote:
On Jan 25, 2012, at 5:02 PM, Vincent Massol wrote:
On Jan 15, 2012, at 11:27 AM, Vincent Massol wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
Actually there's another one:
* Run functional tests in platform * Configure jenkins so that we have one jenkins job per platform top module
This would allow several things: * Faster feedback to user: for example if a commit is done in a module that happens late in the build order the whole platform wouldn't need to be rebuilt before noticing the problem) - Note: Jenkins supports a "Local subdirectory for repo" feature that would make this possible I think)
After researching this a bit I don't think it's possible…
This "Local subdirectory for repo" is something else: "Specify a local directory (relative to the workspace root) where the Git repository will be checked out. If left empty, the workspace root itself will be used."
We could check out the whole Git repo but 1) it'll take space (a lot of space) but more importantly 2) GitHub will trigger a build on all jobs for that repo which isn't good.
Regarding space, I've tested it and I've also tested the "shallow clone" option of Jenkins. So for platform: * with shallow clone: 281MB * without shallow clone (ie full clone): 346MB We have 91 modules in platform so that's 91*281=25571MB == 25GB Even with the full clone that's 31GB We need to add the target/ dir size, so let's say 1GB more in total, to 26GB or 32GB. That's doable! :) So I guess it's ok and that's not our biggest issue. There are 2 more issues: * Automatic creation of jenkins job. That's doable and almost done: http://jira.xwiki.org/jira/browse/XCOMMONS-87 * Triggers. This is the hardest since we'll need to create a manual ordering of jobs as otherwise they'll all start building at the same time when there's a github event coming in… It's not hard to do but it's maintenance intensive. Of course this ordering would be defined in the automated jenkins job creations but it still needs to be maintained... In conclusion it's doable and we need to work on http://jira.xwiki.org/jira/browse/XCOMMONS-87 first.
So I think we'll have to drop the idea of having a single job per module.
See also http://stackoverflow.com/questions/8922857/jenkins-how-to-build-multiple-top...
hmm actually this is what we already do for functional tests but we don't trigger than on Git Hub notifications, we trigger them on xwiki-enterprise (after it's been built).
I guess we could do the same, i.e.:
* Only trigger the top level module in a given repo using a GitHub push * Only trigger the other modules on "Build whenever a SNAPSHOT dependency is built"
The problem is that it would still use a lot of space. For example my xwiki-platform/ dir takes 1.2GB of disk space (with target/ dirs) So with roughly 230+ modules that's 276 GB just for xwiki-platform…
This computation is wrong, see above. Thanks -Vincent
Combined with commons, rendering and enterprise that's probably around 500GB easily…
In conclusion it doesn't seem like a good idea…
Thanks -Vincent
Thanks -Vincent
* Use the full power of our agents by distributing the jobs * Since functional tests run at the same time as unit tests if the module build works we can be sure it's good
Thus even though packaging and starting XE takes a lot of time (I've timed it to roughly 1.5 minutes on my machine) the balancing of jobs on multiple agents might overcome this - it would still be great to be able to improve the XE start time.
Note that creating jenkins job for so many modules might seem daunting but this can be easily automated with http://ci.xwiki.org/api/ or even better http://evgeny-goldin.com/wiki/Maven-jenkins-plugin
I'm still hesitating between the previous idea and this one but thought I would share my current thoughts with you in case you have some idea/preference… :)
Thanks -Vincent
WDYT?
Thanks -Vincent
On Nov 22, 2012, at 11:28 AM, Vincent Massol <[email protected]> wrote:
On Feb 15, 2012, at 4:58 PM, Vincent Massol <[email protected]> wrote:
On Feb 15, 2012, at 4:10 PM, Vincent Massol wrote:
On Jan 25, 2012, at 5:02 PM, Vincent Massol wrote:
On Jan 15, 2012, at 11:27 AM, Vincent Massol wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
Actually there's another one:
* Run functional tests in platform * Configure jenkins so that we have one jenkins job per platform top module
This would allow several things: * Faster feedback to user: for example if a commit is done in a module that happens late in the build order the whole platform wouldn't need to be rebuilt before noticing the problem) - Note: Jenkins supports a "Local subdirectory for repo" feature that would make this possible I think)
After researching this a bit I don't think it's possible…
This "Local subdirectory for repo" is something else: "Specify a local directory (relative to the workspace root) where the Git repository will be checked out. If left empty, the workspace root itself will be used."
We could check out the whole Git repo but 1) it'll take space (a lot of space) but more importantly 2) GitHub will trigger a build on all jobs for that repo which isn't good.
Regarding space, I've tested it and I've also tested the "shallow clone" option of Jenkins.
So for platform: * with shallow clone: 281MB * without shallow clone (ie full clone): 346MB
We have 91 modules in platform so that's 91*281=25571MB == 25GB Even with the full clone that's 31GB
We need to add the target/ dir size, so let's say 1GB more in total, to 26GB or 32GB. That's doable! :)
So I guess it's ok and that's not our biggest issue.
There are 2 more issues: * Automatic creation of jenkins job. That's doable and almost done: http://jira.xwiki.org/jira/browse/XCOMMONS-87 * Triggers. This is the hardest since we'll need to create a manual ordering of jobs as otherwise they'll all start building at the same time when there's a github event coming in… It's not hard to do but it's maintenance intensive. Of course this ordering would be defined in the automated jenkins job creations but it still needs to be maintained...
In conclusion it's doable and we need to work on http://jira.xwiki.org/jira/browse/XCOMMONS-87 first.
Actually maybe a better thing to do (much simpler) would simply be to use parallel builds for platform and instead ensure that our functional tests there use different ports so that they can run in //... That's provided the agents are powerful enough to support a large # of threads. Would be interesting to compute the ideal # of threads based on our dependency graph :) Thanks -Vincent
So I think we'll have to drop the idea of having a single job per module.
See also http://stackoverflow.com/questions/8922857/jenkins-how-to-build-multiple-top...
hmm actually this is what we already do for functional tests but we don't trigger than on Git Hub notifications, we trigger them on xwiki-enterprise (after it's been built).
I guess we could do the same, i.e.:
* Only trigger the top level module in a given repo using a GitHub push * Only trigger the other modules on "Build whenever a SNAPSHOT dependency is built"
The problem is that it would still use a lot of space. For example my xwiki-platform/ dir takes 1.2GB of disk space (with target/ dirs) So with roughly 230+ modules that's 276 GB just for xwiki-platform…
This computation is wrong, see above.
Thanks -Vincent
Combined with commons, rendering and enterprise that's probably around 500GB easily…
In conclusion it doesn't seem like a good idea…
Thanks -Vincent
Thanks -Vincent
* Use the full power of our agents by distributing the jobs * Since functional tests run at the same time as unit tests if the module build works we can be sure it's good
Thus even though packaging and starting XE takes a lot of time (I've timed it to roughly 1.5 minutes on my machine) the balancing of jobs on multiple agents might overcome this - it would still be great to be able to improve the XE start time.
Note that creating jenkins job for so many modules might seem daunting but this can be easily automated with http://ci.xwiki.org/api/ or even better http://evgeny-goldin.com/wiki/Maven-jenkins-plugin
I'm still hesitating between the previous idea and this one but thought I would share my current thoughts with you in case you have some idea/preference… :)
Thanks -Vincent
WDYT?
Thanks -Vincent
On Nov 22, 2012, at 11:41 AM, Vincent Massol <[email protected]> wrote:
On Nov 22, 2012, at 11:28 AM, Vincent Massol <[email protected]> wrote:
On Feb 15, 2012, at 4:58 PM, Vincent Massol <[email protected]> wrote:
On Feb 15, 2012, at 4:10 PM, Vincent Massol wrote:
On Jan 25, 2012, at 5:02 PM, Vincent Massol wrote:
On Jan 15, 2012, at 11:27 AM, Vincent Massol wrote:
Hi devs,
I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE.
For example for the linkchecker module we have the following directories:
xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests)
The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too
Here are some findings about this experiment:
A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module
Solutions/ideas:
* One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only
* Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it).
I think the first proposal is the best one and brings the best of the 2 worlds.
Actually there's another one:
* Run functional tests in platform * Configure jenkins so that we have one jenkins job per platform top module
This would allow several things: * Faster feedback to user: for example if a commit is done in a module that happens late in the build order the whole platform wouldn't need to be rebuilt before noticing the problem) - Note: Jenkins supports a "Local subdirectory for repo" feature that would make this possible I think)
After researching this a bit I don't think it's possible…
This "Local subdirectory for repo" is something else: "Specify a local directory (relative to the workspace root) where the Git repository will be checked out. If left empty, the workspace root itself will be used."
We could check out the whole Git repo but 1) it'll take space (a lot of space) but more importantly 2) GitHub will trigger a build on all jobs for that repo which isn't good.
Regarding space, I've tested it and I've also tested the "shallow clone" option of Jenkins.
So for platform: * with shallow clone: 281MB * without shallow clone (ie full clone): 346MB
We have 91 modules in platform so that's 91*281=25571MB == 25GB Even with the full clone that's 31GB
We need to add the target/ dir size, so let's say 1GB more in total, to 26GB or 32GB. That's doable! :)
So I guess it's ok and that's not our biggest issue.
There are 2 more issues: * Automatic creation of jenkins job. That's doable and almost done: http://jira.xwiki.org/jira/browse/XCOMMONS-87 * Triggers. This is the hardest since we'll need to create a manual ordering of jobs as otherwise they'll all start building at the same time when there's a github event coming in… It's not hard to do but it's maintenance intensive. Of course this ordering would be defined in the automated jenkins job creations but it still needs to be maintained...
In conclusion it's doable and we need to work on http://jira.xwiki.org/jira/browse/XCOMMONS-87 first.
Actually maybe a better thing to do (much simpler) would simply be to use parallel builds for platform and instead ensure that our functional tests there use different ports so that they can run in //...
That's provided the agents are powerful enough to support a large # of threads. Would be interesting to compute the ideal # of threads based on our dependency graph :)
Yet another solution is to build platform without the -Pintegration-tests profile with parallel builds and then create one job per functional test in platform. -Vincent
Thanks -Vincent
So I think we'll have to drop the idea of having a single job per module.
See also http://stackoverflow.com/questions/8922857/jenkins-how-to-build-multiple-top...
hmm actually this is what we already do for functional tests but we don't trigger than on Git Hub notifications, we trigger them on xwiki-enterprise (after it's been built).
I guess we could do the same, i.e.:
* Only trigger the top level module in a given repo using a GitHub push * Only trigger the other modules on "Build whenever a SNAPSHOT dependency is built"
The problem is that it would still use a lot of space. For example my xwiki-platform/ dir takes 1.2GB of disk space (with target/ dirs) So with roughly 230+ modules that's 276 GB just for xwiki-platform…
This computation is wrong, see above.
Thanks -Vincent
Combined with commons, rendering and enterprise that's probably around 500GB easily…
In conclusion it doesn't seem like a good idea…
Thanks -Vincent
Thanks -Vincent
* Use the full power of our agents by distributing the jobs * Since functional tests run at the same time as unit tests if the module build works we can be sure it's good
Thus even though packaging and starting XE takes a lot of time (I've timed it to roughly 1.5 minutes on my machine) the balancing of jobs on multiple agents might overcome this - it would still be great to be able to improve the XE start time.
Note that creating jenkins job for so many modules might seem daunting but this can be easily automated with http://ci.xwiki.org/api/ or even better http://evgeny-goldin.com/wiki/Maven-jenkins-plugin
I'm still hesitating between the previous idea and this one but thought I would share my current thoughts with you in case you have some idea/preference… :)
Thanks -Vincent
WDYT?
Thanks -Vincent
participants (4)
-
Jean-Vincent Drean -
Jerome Velociter -
Marius Dumitru Florea -
Vincent Massol