Hi, I would really like to see something like this in our build, specially since I've always wanted us to enable the surefire flag that would re-execute a failed test. Now, it's not very clear to me, but both the JUnit Pioneer thing and the rerunner-jupiter one seem to be proposing an annotation that needs to be set apriori on a test, otherwise the plugin/feature does not do its job? I understand that, due to JUnit5, things might still be work in progress, but I'd like to clarify on the intent that, AFAIU, any test that fails should be rerun to check if it's a flaky test or an actual failure and not only tests marked before (since it would slightly defeat the purpose of automatically detecting and marking flaky tests). +1 to try to automate at least the flagging of such plagues (that are flaky tests) :) Thanks, Eduard On Mon, Jun 10, 2019 at 12:43 PM Vincent Massol <[email protected]> wrote:

Hi devs,

I was looking for a rerun optin in JUnit5 or Surefire but it seems it doesn’t exist for Junit5: * https://github.com/junit-team/junit5/issues/1558 * https://issues.apache.org/jira/browse/SUREFIRE-1584

However, there’s an extension to do that: * https://github.com/artsok/rerunner-jupiter

Also WIP for JUnit Pioneer: * https://github.com/junit-pioneer/junit-pioneer/pull/110

So my proposal FTM is to try https://github.com/artsok/rerunner-jupiter and to start putting a repeat of 3 for JUnit5 tests that we know are flaky.

For JUnit4 tests we could simply use: * https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-faili...

Note that this would also allow us to correct flaky tests that don’t set up the initial conditions (fixture) correctly.

Second note: This is not an endorsement that we should accept flaky tests. They’re a plague and we need to work on them and fix them. This is just a momentary workaround to identify how flaky a test is and to try to have a better solution towards moving to a Continuous Delivery solution.

WDYT?

Thanks -Vincent