[xwiki-devs] [Brainstorming] Colocated Functional Test (CFT)

Hi devs, I've started an experiment to have colocated functional tests (CFT), which means having the functional tests located where the functional domain sources are located instead of in XE. For example for the linkchecker module we have the following directories: xwiki-platform-linkchecker/ |_ xwiki-platform-linkchecker-refresher (JAR) |_ xwiki-platform-linkchecker-ui (XAR) |_ xwiki-platform-linkchecker-tests (functional tests) The rationale for this was: * Have everything about a functional domain self-contained (source and all tests) * Making it easy to run only tests for a given functional domain * Move page objects to the functional domain too Here are some findings about this experiment: A - It takes about 30 seconds to generate the adhoc packaging and start XWiki. This would be done for each module having functional tests compared to only once if all tests were executed in XE B- The package mojo created to generate a full packaging is quite nice and I plan to reuse it in lots of other places in our build (distributions, database, places where we need XWiki configuration files) C- We will not be able to run platform builds in Maven multithreaded mode since it would mean that several XWiki instance could be started at the same time on the same port D- The colocated functional test module Solutions/ideas: * One idea to overcome A and C would to have the following setup: ** Keep functional test modules colocated but have them generate a test JAR ** Still allow running functional tests from the colocated module (this makes it easy to verify no regression was introduced when making changes to a given domain) ** Have functional tests in XE depend on the colocated functional test module JARs and configure Jenkins to run all functional tests from XE only * Another solution to overcome C is to auto-discover the port to use in our XWiki startup script (and save it in a file so that the stop script can use it). I think the first proposal is the best one and brings the best of the 2 worlds. WDYT? Thanks -Vincent