[xwiki-devs] [Brainstorming] Next gen functional testing for XWiki

Hi devs, I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project. Needs ===== * Be able to test xwiki on multiple environments Context ====== * Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin. Proposal ======= * The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki. Pros ==== * Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance Implementation ============ Something like: --> XWikiSeleniumExtension extends SeleniumExtension @ExtendWith(XWikiSeleniumExtension.class) public class Test @Test public void xxx(XWikiWebDriver driver) { … } And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver). The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images). The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir Migration ======= Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests. WDYT? I’m planning to continue my investigation/development of this. So please let me know if you have feedback. Thanks -Vincent