[xwiki-devs] Status on jenkins work and experimentations

Hi devs, I wanted to share with you what’s been done and what’s next, related to the CI improvements and especially jenkins pipelines. Done: ===== * A) Share pipeline script is configured on ci.xwiki.org and available at https://github.com/xwiki/xwiki-jenkins-pipeline * B) Contrib projects are using a Jenkinsfile and a Github Organization job is set up on ci.xwiki.org (automatically create and remove jobs based on branches and PRs) * C) The share pipeline script now supports screenshot attachment to the job test report page + avoids false positives emails Ongoing experiment: ================ * D) I started moving the commons/rendering/platform/enterprise jobs to use our pipeline script but this is not possible right now because there’s an important missing feature ATM in pipeline: the ability to have incremental maven builds (ie only rebuild the module touched by the commit and its dependencies). Without this we would rebuild the whole repo which would lead to more load on the agents and slower feedback times. What I’m planning to work on next: =========================== * E) Create a new pipeline script to perform releases of XE (commons + rendering + platform + enterprise). The idea is to move part of our release script into a pipeline that would get executed either at each commit (if we have enough agents) or once per day or more. The idea is that once we’re ready to cut the release we could just promote the pre-built release and win several hours on our release process. This is what’s left to do in order to be able to do this IMO: ** Set the “xwiki.compatibility.previous.version” automatically in the script if not correct (this is step 1 of the release plan) ** Move the translation script into the pipeline script and only apply translations for non bug fix releases. Note that the RM would still need to validate visually that no spam was introduced in translations at release time by reviewing the job logs. ** Create a flickering “database”, either in jira or in xwiki.org (my proposal is to use xwiki.org and have an XObject per flicker), and in the pipeline script verify automatically that any failing test is part of the flicker database or fail the release otherwise. ** Execute the release:prepare and release:perform goals in the pipeline script and release to a staging nexus repo on nexus.xwiki.org. The RM would just need to release from this repo at release time ** Have some nexus scheduled task to remove non-released staging repos, or do this using the nexus REST API inside the pipeline script if need be (or do this manually by the RM initially) * F) Experiment using the docker pipeline feature to run XWiki functional tests inside docker. The goal is to be able to test xwiki in various setups (various DBs, with LibreOffice server, ideally clustering but that’s harder, etc). ** For this I think I’ll need to develop a new docker image that doesn’t include XWiki and map the local directory where the XWiki distribution has been created by the packager plugin into the docker image. If you have any remarks please let me know and if someone is interested in participating let me know too :) Thanks -Vincent