On 12 Feb 2019, at 14:49, Thomas Mortagne <[email protected]> wrote:

On Tue, Feb 12, 2019 at 2:45 PM Vincent Massol <[email protected]> wrote:

...

Hi,

Now that JUnit 5.4.0 is out we have test ordering. I propose the following best practices for TC-based docker tests:

@UITest @TestMethodOrder(OrderAnnotation.class) public class MenuIT { @Test @Order(1) public void verifyMenuInApplicationsIndex(TestUtils setup) { ... }

@Test @Order(2) public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup) { ... }

@Test @Order(3) public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception { … }

Instead of the current:

@Test public void verifyMenu(TestUtils setup) throws Exception { verifyMenuInApplicationsIndex(setup); verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup); verifyMenuIsAvailableInAdministration(setup); }

Pros: * Easier to run each test * Easier to debug and view recorded video for a specific failing test * More in sync with JUnit’s practices * It’s still a scenario and thus doesn’t incur penalty of extra test setup

Cons: * Starts and stop the VNC docker container for each test. It takes between 1 and 2s to start the VNC container and the stop (i.e. video recording save) should be the same as before.

So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too much (between 2% and 3% more).

WDYT?

Why only docker based tests and not all UI tests ?

Yes, all UI tests, not just the docker-based ones. I mentioned the docker-based ones because they are the most controversial ones (with the VNC start/stop), which is the only reason I see for not wanting to do this. Thanks -Vincent

...

Thanks -Vincent

-- Thomas Mortagne

Improvement thanks to Simon, I’ve now moved the 2 new annotations to the UITest annotation: @Documented @Retention(RUNTIME) @Target({ TYPE, METHOD, ANNOTATION_TYPE }) @ExtendWith(XWikiDockerExtension.class) @TestInstance(TestInstance.Lifecycle.PER_CLASS) @TestMethodOrder(MethodOrderer.OrderAnnotation.class) public @interface UITest Example usage: @UITest public class MenuIT { @Test @Order(1) … Simpler, thanks Simon! -Vincent

On 12 Feb 2019, at 15:04, Vincent Massol <[email protected]> wrote:

Actually to be even more rigorous, we should use:

@UITest @TestMethodOrder(OrderAnnotation.class) @TestInstance(Lifecycle.PER_CLASS) public class MenuIT

See https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-instanc... for more details.

Thanks -Vincent

...

On 12 Feb 2019, at 14:44, Vincent Massol <[email protected]> wrote:

Hi,

Now that JUnit 5.4.0 is out we have test ordering. I propose the following best practices for TC-based docker tests:

@UITest @TestMethodOrder(OrderAnnotation.class) public class MenuIT { @Test @Order(1) public void verifyMenuInApplicationsIndex(TestUtils setup) { ... }

@Test @Order(2) public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup) { ... }

@Test @Order(3) public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception { … }

Instead of the current:

@Test public void verifyMenu(TestUtils setup) throws Exception { verifyMenuInApplicationsIndex(setup); verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup); verifyMenuIsAvailableInAdministration(setup); }

Pros: * Easier to run each test * Easier to debug and view recorded video for a specific failing test * More in sync with JUnit’s practices * It’s still a scenario and thus doesn’t incur penalty of extra test setup

Cons: * Starts and stop the VNC docker container for each test. It takes between 1 and 2s to start the VNC container and the stop (i.e. video recording save) should be the same as before.

So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too much (between 2% and 3% more).

WDYT?

Thanks -Vincent