Re: [xwiki-devs] [Brainstorming] Next gen functional testing for XWiki

Hi, It will be very useful to be able to run functional tests on all supported browsers. However I wonder if it will also be as useful for different servlet containers and databases (at least when the database schema doesn't change). I tried to run the tests on my machine but I got an error: https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you might have an idea why I'm getting this). Also it's a bit disturbing to wait for the docker images to be downloaded without knowing it (we should at least have a message saying "Downloading images..." or even better, the docker output). Thanks, Adel On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <vincent(a)massol.net> wrote: > Important note: > * We’ll only be able to test Firefox and Chrome browsers since there’s no Docker containers for windows/mac having IE11/Edge and Safari (they’re not supported ATM even on windows machines). > * We could work around this by having some agents running on Windows and using the installed IE11/Edge browser (i.e. not running in a Docker container). However it’s too complex FTM and I’d instead move this to later and instead focus on running XWiki on various DBs. > * So this means that we’ll still need manual tests for IE11/Edge/Safari for the moment... > > Thanks > -Vincent > >> On 30 Jun 2018, at 14:59, Vincent Massol <vincent(a)massol.net> wrote: >> >> Status as of today: >> >> * I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines. >> >> * I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting >> >> Next steps: >> >> * Commit the work one (in the coming days) >> * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. >> * Integrate ability to choose the DB to use for the tests. >> >> Feedback most welcome. >> >> Thanks >> -Vincent >> >>> On 27 May 2018, at 21:44, Vincent Massol <vincent(a)massol.net> wrote: >>> >>> Hi devs, >>> >>> I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project. >>> >>> Needs >>> ===== >>> >>> * Be able to test xwiki on multiple environments >>> >>> Context >>> ====== >>> >>> * Right now we test only in 1 env (Jetty+HSQLDB) >>> * I've started some docker images in xwiki-contrib >>> * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. >>> * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin. >>> >>> Proposal >>> ======= >>> >>> * The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). >>> * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java >>> * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ >>> ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ >>> * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki. >>> >>> Pros >>> ==== >>> >>> * Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. >>> * Very simple to test any combination of DB/Servlet Container/Browser. >>> * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). >>> * Using JUnit5 and thus the latest features >>> * Moving to the latest Selenium version too >>> * Also supports manually executing tests in a given running xwiki instance >>> >>> Implementation >>> ============ >>> >>> Something like: >>> >>> --> XWikiSeleniumExtension extends SeleniumExtension >>> >>> @ExtendWith(XWikiSeleniumExtension.class) >>> public class Test >>> >>> @Test >>> public void xxx(XWikiWebDriver driver) >>> { >>> … >>> } >>> >>> And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver). >>> >>> The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images). >>> >>> The packaging will be configurable. Some ideas of options: >>> * use an already running xwiki instance >>> * docker created from full XS zip from URL >>> * docker created from XS zip from maven artifact >>> * docker created from computed based on pom in current dir >>> >>> Migration >>> ======= >>> >>> Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests. >>> >>> WDYT? >>> >>> I’m planning to continue my investigation/development of this. So please let me know if you have feedback. >>> >>> Thanks >>> -Vincent >>> >>> >> >