[xwiki-devs] [Proposal] Rerun flaky tests several times?

Hi devs, I was looking for a rerun optin in JUnit5 or Surefire but it seems it doesn’t exist for Junit5: * https://github.com/junit-team/junit5/issues/1558 * https://issues.apache.org/jira/browse/SUREFIRE-1584 However, there’s an extension to do that: * https://github.com/artsok/rerunner-jupiter Also WIP for JUnit Pioneer: * https://github.com/junit-pioneer/junit-pioneer/pull/110 So my proposal FTM is to try https://github.com/artsok/rerunner-jupiter and to start putting a repeat of 3 for JUnit5 tests that we know are flaky. For JUnit4 tests we could simply use: * https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-fail… Note that this would also allow us to correct flaky tests that don’t set up the initial conditions (fixture) correctly. Second note: This is not an endorsement that we should accept flaky tests. They’re a plague and we need to work on them and fix them. This is just a momentary workaround to identify how flaky a test is and to try to have a better solution towards moving to a Continuous Delivery solution. WDYT? Thanks -Vincent