10 Jun
2019
10 Jun
'19
4:54 p.m.
Hi,
I would really like to see something like this in our build, specially
since I've always wanted us to enable the surefire flag that would
re-execute a failed test.
Now, it's not very clear to me, but both the JUnit Pioneer thing and the
rerunner-jupiter one seem to be proposing an annotation that needs to be
set apriori on a test, otherwise the plugin/feature does not do its job? I
understand that, due to JUnit5, things might still be work in progress, but
I'd like to clarify on the intent that, AFAIU, any test that fails should
be rerun to check if it's a flaky test or an actual failure and not only
tests marked before (since it would slightly defeat the purpose of
automatically detecting and marking flaky tests).
+1 to try to automate at least the flagging of such plagues (that are flaky
tests) :)
Thanks,
Eduard
On Mon, Jun 10, 2019 at 12:43 PM Vincent Massol <vincent(a)massol.net> wrote:
Hi devs,
I was looking for a rerun optin in JUnit5 or Surefire but it seems it
doesn’t exist for Junit5:
* https://github.com/junit-team/junit5/issues/1558
* https://issues.apache.org/jira/browse/SUREFIRE-1584
However, there’s an extension to do that:
* https://github.com/artsok/rerunner-jupiter
Also WIP for JUnit Pioneer:
* https://github.com/junit-pioneer/junit-pioneer/pull/110
So my proposal FTM is to try https://github.com/artsok/rerunner-jupiter
and to start putting a repeat of 3 for JUnit5 tests that we know are flaky.
For JUnit4 tests we could simply use:
*
https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-fail…
Note that this would also allow us to correct flaky tests that don’t set
up the initial conditions (fixture) correctly.
Second note: This is not an endorsement that we should accept flaky tests.
They’re a plague and we need to work on them and fix them. This is just a
momentary workaround to identify how flaky a test is and to try to have a
better solution towards moving to a Continuous Delivery solution.
WDYT?
Thanks
-Vincent