[xwiki-devs] [Brainstorming] Next gen functional testing for XWiki
Hi devs, I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project. Needs ===== * Be able to test xwiki on multiple environments Context ====== * Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin. Proposal ======= * The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki. Pros ==== * Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance Implementation ============ Something like: --> XWikiSeleniumExtension extends SeleniumExtension @ExtendWith(XWikiSeleniumExtension.class) public class Test @Test public void xxx(XWikiWebDriver driver) { … } And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver). The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images). The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir Migration ======= Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests. WDYT? I’m planning to continue my investigation/development of this. So please let me know if you have feedback. Thanks -Vincent
Sounds good. On Sun, May 27, 2018 at 9:44 PM, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
-- Thomas Mortagne
Status as of today: * I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines. * I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting Next steps: * Commit the work one (in the coming days) * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests. Feedback most welcome. Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote:
Status as of today:
* I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines.
* I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting
Next steps:
* Commit the work one (in the coming days)
Done and documented at https://dev.xwiki.org/xwiki/bin/view/Community/Testing#HSelenium3-basedFrame... Would be great if you could try it. To summarize there are 2 advantages right now over the selenium2 test framework we have: * You can run the tests in FF or Chrome * The version of FF or Chrome that you installed on your local computer doesn’t matter and won’t be used. So no more juggling with different FF versions. * You get to use the latest released Selenium3 version and the latest Chrome/FF versions which are compatible with the selenium version, this testing what we’re supposed to test. Thanks -Vincent
* I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests.
Feedback most welcome.
Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
Important note: * We’ll only be able to test Firefox and Chrome browsers since there’s no Docker containers for windows/mac having IE11/Edge and Safari (they’re not supported ATM even on windows machines). * We could work around this by having some agents running on Windows and using the installed IE11/Edge browser (i.e. not running in a Docker container). However it’s too complex FTM and I’d instead move this to later and instead focus on running XWiki on various DBs. * So this means that we’ll still need manual tests for IE11/Edge/Safari for the moment... Thanks -Vincent
On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote:
Status as of today:
* I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines.
* I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting
Next steps:
* Commit the work one (in the coming days) * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests.
Feedback most welcome.
Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
Hi, It will be very useful to be able to run functional tests on all supported browsers. However I wonder if it will also be as useful for different servlet containers and databases (at least when the database schema doesn't change). I tried to run the tests on my machine but I got an error: https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you might have an idea why I'm getting this). Also it's a bit disturbing to wait for the docker images to be downloaded without knowing it (we should at least have a message saying "Downloading images..." or even better, the docker output). Thanks, Adel On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <[email protected]> wrote:
Important note: * We’ll only be able to test Firefox and Chrome browsers since there’s no Docker containers for windows/mac having IE11/Edge and Safari (they’re not supported ATM even on windows machines). * We could work around this by having some agents running on Windows and using the installed IE11/Edge browser (i.e. not running in a Docker container). However it’s too complex FTM and I’d instead move this to later and instead focus on running XWiki on various DBs. * So this means that we’ll still need manual tests for IE11/Edge/Safari for the moment...
Thanks -Vincent
On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote:
Status as of today:
* I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines.
* I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting
Next steps:
* Commit the work one (in the coming days) * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests.
Feedback most welcome.
Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
Also I hope it will not make CI builds a lot longer. Adel Atallah Product developer intern [email protected] tel: +33 (0)6 12 96 35 06 On Mon, Jul 2, 2018 at 10:52 AM, Adel Atallah <[email protected]> wrote:
Hi,
It will be very useful to be able to run functional tests on all supported browsers. However I wonder if it will also be as useful for different servlet containers and databases (at least when the database schema doesn't change). I tried to run the tests on my machine but I got an error: https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you might have an idea why I'm getting this). Also it's a bit disturbing to wait for the docker images to be downloaded without knowing it (we should at least have a message saying "Downloading images..." or even better, the docker output).
Thanks, Adel
On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <[email protected]> wrote:
Important note: * We’ll only be able to test Firefox and Chrome browsers since there’s no Docker containers for windows/mac having IE11/Edge and Safari (they’re not supported ATM even on windows machines). * We could work around this by having some agents running on Windows and using the installed IE11/Edge browser (i.e. not running in a Docker container). However it’s too complex FTM and I’d instead move this to later and instead focus on running XWiki on various DBs. * So this means that we’ll still need manual tests for IE11/Edge/Safari for the moment...
Thanks -Vincent
On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote:
Status as of today:
* I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines.
* I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting
Next steps:
* Commit the work one (in the coming days) * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests.
Feedback most welcome.
Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
On 2 Jul 2018, at 10:58, Adel Atallah <[email protected]> wrote:
Also I hope it will not make CI builds a lot longer.
It should make it slower to start the container but only marginally. What can be slower is that it starts a VNC server which we were not starting before. Globally that should add a couple of seconds. Thanks -Vincent
Adel Atallah Product developer intern [email protected] tel: +33 (0)6 12 96 35 06
On Mon, Jul 2, 2018 at 10:52 AM, Adel Atallah <[email protected]> wrote:
Hi,
It will be very useful to be able to run functional tests on all supported browsers. However I wonder if it will also be as useful for different servlet containers and databases (at least when the database schema doesn't change). I tried to run the tests on my machine but I got an error: https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you might have an idea why I'm getting this). Also it's a bit disturbing to wait for the docker images to be downloaded without knowing it (we should at least have a message saying "Downloading images..." or even better, the docker output).
Thanks, Adel
On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <[email protected]> wrote:
Important note: * We’ll only be able to test Firefox and Chrome browsers since there’s no Docker containers for windows/mac having IE11/Edge and Safari (they’re not supported ATM even on windows machines). * We could work around this by having some agents running on Windows and using the installed IE11/Edge browser (i.e. not running in a Docker container). However it’s too complex FTM and I’d instead move this to later and instead focus on running XWiki on various DBs. * So this means that we’ll still need manual tests for IE11/Edge/Safari for the moment...
Thanks -Vincent
On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote:
Status as of today:
* I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines.
* I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting
Next steps:
* Commit the work one (in the coming days) * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests.
Feedback most welcome.
Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
On Mon, Jul 2, 2018 at 10:52 AM, Adel Atallah <[email protected]> wrote:
Hi,
It will be very useful to be able to run functional tests on all supported browsers. However I wonder if it will also be as useful for different servlet containers and databases (at least when the database schema doesn't change).
I don't think it's very useful for different containers but we very often have issues with differences between databases that we discover only after the release (like Oracle and its f... empty string=null or MySQL and its weird handling of escaping) so it will definitely be useful.
I tried to run the tests on my machine but I got an error: https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you might have an idea why I'm getting this). Also it's a bit disturbing to wait for the docker images to be downloaded without knowing it (we should at least have a message saying "Downloading images..." or even better, the docker output).
Thanks, Adel
On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <[email protected]> wrote:
Important note: * We’ll only be able to test Firefox and Chrome browsers since there’s no Docker containers for windows/mac having IE11/Edge and Safari (they’re not supported ATM even on windows machines). * We could work around this by having some agents running on Windows and using the installed IE11/Edge browser (i.e. not running in a Docker container). However it’s too complex FTM and I’d instead move this to later and instead focus on running XWiki on various DBs. * So this means that we’ll still need manual tests for IE11/Edge/Safari for the moment...
Thanks -Vincent
On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote:
Status as of today:
* I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines.
* I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting
Next steps:
* Commit the work one (in the coming days) * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests.
Feedback most welcome.
Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
-- Thomas Mortagne
Hi Adel, Thanks for trying it! :)
On 2 Jul 2018, at 10:52, Adel Atallah <[email protected]> wrote:
Hi,
It will be very useful to be able to run functional tests on all supported browsers. However I wonder if it will also be as useful for different servlet containers and databases (at least when the database schema doesn't change). I tried to run the tests on my machine but I got an error: https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you might have an idea why I'm getting this).
Yes I know why. It’s because I need to find the IP so that the browser in the docker container can open a URL to your XWiki running on your host. It seems in your case that the IP resolving that I do lead to 127.0.0.1 which is wrong since that points to the container and not outside. I’ve pushed a fix for my code, see https://github.com/xwiki/xwiki-platform/commit/380c803357d49231eaf4f513d5c2d... Note that it’s also possible to pass the system property “xwiki.test.baseURL”, e.g. "xwiki.test.baseURL=http://192.168.0.45” (without the port).
Also it's a bit disturbing to wait for the docker images to be downloaded without knowing it (we should at least have a message saying "Downloading images..." or even better, the docker output).
Note that it’s only the first time. Re displaying some progress/info, I’ll need to check if that can be configured in TestContainers but I doubt it. Thanks -Vincent
Thanks, Adel
On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <[email protected]> wrote:
Important note: * We’ll only be able to test Firefox and Chrome browsers since there’s no Docker containers for windows/mac having IE11/Edge and Safari (they’re not supported ATM even on windows machines). * We could work around this by having some agents running on Windows and using the installed IE11/Edge browser (i.e. not running in a Docker container). However it’s too complex FTM and I’d instead move this to later and instead focus on running XWiki on various DBs. * So this means that we’ll still need manual tests for IE11/Edge/Safari for the moment...
Thanks -Vincent
On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote:
Status as of today:
* I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines.
* I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting
Next steps:
* Commit the work one (in the coming days) * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests.
Feedback most welcome.
Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
On 2 Jul 2018, at 10:52, Adel Atallah <[email protected]> wrote:
Hi,
It will be very useful to be able to run functional tests on all supported browsers. However I wonder if it will also be as useful for different servlet containers and databases (at least when the database schema doesn't change).
Re the DB Thomas has replied already and it’s very useful (we regularly have regressions because of that). Re the container, we need to decide and write a Servlet Container Support Strategy document as we have for DB and Browsers. Right now it’s a grey area. We document some installation instructions for some servlet containers on https://www.xwiki.org/xwiki/bin/view/Documentation/AdminGuide/Installation/I... but we don’t verify they work. And they don’t for several of them… for example we’ve received several forum posts and JIRAs about XWiki not starting fine on Glassfish for ex. We had the same issue on JBoss in the past (we even bundle some Jboss config files in our WEB-INF/ directory as a result ;)). So for me, even for servlet containers, it would be useful if we really want to ensure that XWiki deploys fine in them and thus increase user installation satisfaction. Thanks -Vincent
I tried to run the tests on my machine but I got an error: https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you might have an idea why I'm getting this). Also it's a bit disturbing to wait for the docker images to be downloaded without knowing it (we should at least have a message saying "Downloading images..." or even better, the docker output).
Thanks, Adel
On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <[email protected]> wrote:
Important note: * We’ll only be able to test Firefox and Chrome browsers since there’s no Docker containers for windows/mac having IE11/Edge and Safari (they’re not supported ATM even on windows machines). * We could work around this by having some agents running on Windows and using the installed IE11/Edge browser (i.e. not running in a Docker container). However it’s too complex FTM and I’d instead move this to later and instead focus on running XWiki on various DBs. * So this means that we’ll still need manual tests for IE11/Edge/Safari for the moment...
Thanks -Vincent
On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote:
Status as of today:
* I have a working POC for running XWiki functional tests in various browsers (chrome and firefox FTM), using the latest Selenium and latest Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires docker to be installed so I’ll create a “docker” profile in the build so that the build doesn’t break on agents which don’t have docker installed yet. It’ll allow you to start testing it on your local machines.
* I have updated the design page at http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting
Next steps:
* Commit the work one (in the coming days) * I have started trying to deploy XWiki in an embedded undertow. Still not working at this stage (but I’ve progressed). I still don’t know if this is going to work or not. I hope it will since that would save me from having to write the equivalent of the Packager plugin. * Integrate ability to choose the DB to use for the tests.
Feedback most welcome.
Thanks -Vincent
On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote:
Hi devs,
I’d like to give you some info about what I’ve started working on and verify you like the direction I’m proposing to take for the future of functional testing on the xwiki project.
Needs =====
* Be able to test xwiki on multiple environments
Context ======
* Right now we test only in 1 env (Jetty+HSQLDB) * I've started some docker images in xwiki-contrib * I’ve also started some experiment through https://jira.xwiki.org/browse/XWIKI-14929 and https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread "[Brainstorming] Implementing multi-environment tests - Take 2” and https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This email supersedes the "[Brainstorming] Implementing multi-environment tests - Take 2” thread. * Initially I imagined doing the multi env testing in Jenkins thanks to the Jenkins Docker plugin/library. However I realized that it would be better to be able to run that on the dev machines and thus decided instead to implement it at the maven level thanks to the Fabric8 Maven plugin.
Proposal =======
* The new proposal is to stop trying to do it at the maven level and instead do it at the Java level, i.e. be able to control (start/stop the various docker images for the DB, Servlet Container/XWiki and the Browser from within the java junit/selenium tests). * There are several java libraries existing to control docker from within java. For example: https://github.com/docker-java/docker-java * I got convinced when finding this awesome library that combines JUnit5/Selenium and Docker for multi-browser testing: https://bonigarcia.github.io/selenium-jupiter/ ** Note that this relies on the browser docker images provided by the Selenoid project: https://aerokube.com/selenoid/latest/ * So the idea is to extend that to be able to control the other 2 docker containers for the DB + ServletContainer/XWiki.
Pros ====
* Very simple setup to start/stop functional tests (and to debug them). Only requires Docker to be installed locally. * Very simple to test any combination of DB/Servlet Container/Browser. * Always up to date images with the latest version (we can depend on LATEST of Browser images, MySQL, Tomcat, etc). * Using JUnit5 and thus the latest features * Moving to the latest Selenium version too * Also supports manually executing tests in a given running xwiki instance
Implementation ============
Something like:
--> XWikiSeleniumExtension extends SeleniumExtension
@ExtendWith(XWikiSeleniumExtension.class) public class Test
@Test public void xxx(XWikiWebDriver driver) { … }
And be able to configure the DB to use, the Servlet container to use, and the packaging to use from system properties (and also from the test itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver).
The idea is to reimplement the XWiki Packaging Maven plugin as a java lib using Aether and to just start our functional tests using pure junit without anymore more. All the hard work will be performed by the JUnit5 extension (create the packaging if not already exist, update some part of it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, download the docker images).
The packaging will be configurable. Some ideas of options: * use an already running xwiki instance * docker created from full XS zip from URL * docker created from XS zip from maven artifact * docker created from computed based on pom in current dir
Migration =======
Once a first version is working, it’ll be easy to use it only for a single platform functional tests and then slowly move each module to use the new way for its functional tests.
WDYT?
I’m planning to continue my investigation/development of this. So please let me know if you have feedback.
Thanks -Vincent
participants (3)
-
Adel Atallah -
Thomas Mortagne -
Vincent Massol