Re: [xwiki-devs] [Proposal] New Functional test best practice for docker-based tests

12 Feb 2019

      Improvement thanks to Simon, I’ve now moved the 2 new annotations to the UITest annotation:

@Documented
@Retention(RUNTIME)
@Target({ TYPE, METHOD, ANNOTATION_TYPE })
@ExtendWith(XWikiDockerExtension.class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public @interface UITest

Example usage:

@UITest
public class MenuIT
{
    @Test
    @Order(1)
…

Simpler, thanks Simon!

-Vincent
...
On 12 Feb 2019, at 15:04, Vincent Massol <[email protected]> wrote:
Actually to be even more rigorous, we should use:
@UITest
@TestMethodOrder(OrderAnnotation.class)
@TestInstance(Lifecycle.PER_CLASS)
public class MenuIT
See https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-instanc... for more details.
Thanks
-Vincent
...
On 12 Feb 2019, at 14:44, Vincent Massol <[email protected]> wrote:
Hi,
Now that JUnit 5.4.0 is out we have test ordering. I propose the following best practices for TC-based docker tests:
@UITest
@TestMethodOrder(OrderAnnotation.class)
public class MenuIT
{
  @Test
  @Order(1)
  public void verifyMenuInApplicationsIndex(TestUtils setup)
  {
...
  }
@Test
  @Order(2)
  public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
  {
...
  }
@Test
  @Order(3)
  public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception
  {
…
  }
Instead of the current:
@Test
  public void verifyMenu(TestUtils setup) throws Exception
  {
      verifyMenuInApplicationsIndex(setup);
      verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
      verifyMenuIsAvailableInAdministration(setup);
  }
Pros:
* Easier to run each test
* Easier to debug and view recorded video for a specific failing test
* More in sync with JUnit’s practices
* It’s still a scenario and thus doesn’t incur penalty of extra test setup
Cons:
* Starts and stop the VNC docker container for each test. It takes between 1 and 2s to start the VNC container and the stop (i.e. video recording save) should be the same as before.
So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too much (between 2% and 3% more).
WDYT?
Thanks
-Vincent