[xwiki-devs] [Status] Improving the CI stability - Status report

Hi devs, Marius and myself have been spending the week trying to improve our CI and especially making sure that mails we receive are real problems. Here's what we've achieved: * Configured the jenkins email to have more interesting content and make sure all jobs have the same notification configuration. Current situation: - an email is sent on every Failure - no email is sent on build fixed - only the list is notified ATM * Fixed several flickers * XINFRA-71: Automate creation of Jenkins jobs when we create a new branch. This is a by product which will be useful when we perform releases * XINFRA-74: Update Agents to use FF 17 + updated to Selenium 2.28 * XINFRA-72: Don't send CI emails when the problem is a CI issue. We've covered the following messages in the log (when these message appear no email is sent and the build is marked with a warning icon): - JVM crash: ".*A fatal error has been detected by the Java Runtime Environment.*" - VNC crash: ".*Error: cannot open display: :1.0.*" - VNC crash: ".*java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment.*" - Git issue: ".hudson.plugins.git.GitException: Could not fetch from any repository.*" - Browser crash: ".*Error communicating with the remote browser. It may have died..*" - Machine slowness: ".*Failed to start XWiki in .* seconds.*" While we're close to a stable build it's still not fully achieved since we sill have flickers in ui-tests. What it means is that from now one we need to pay attention to any Outstanding tasks: * XINFRA-75: Start VNC by the job * XINFRA-76: Fix slow XWiki startup on CI * XINFRA-70: Put Jenkins configuration under SCM * XWIKI-8631: Stop XWiki process in functional tests if XWiki is already started when the test starts * XINFRA-78: Set up Timeout Jenkins plugin to stop stuck jobs * XINFRA-79: Link to the screenshot directory in functional test job emails when there are failures * There are still some test failures in test-ui: http://ci.xwiki.org/view/Functional%20Tests/ Next steps: * Continue working on the listed issues above. If you have some other ideas not listed here please mention them. Thanks -Vincent