26 Mar
2019
26 Mar
'19
10:55 a.m.
Hi Simon and all,
On 26 Mar 2019, at 10:31, Simon Urli
<simon.urli(a)xwiki.com> wrote:
Hi everyone,
I was checking our list of flickering tests in JIRA
(https://jira.xwiki.org/issues/?jql=labels%20%3D%20flickering%20AND%20status…)
and I noticed that we had somehow old flickering test issue concerning test that I've
never seen failing.
So I propose we close some of them as inactive: the ones that we don't remember
having seen for a while. The ideal would be to have a mechanism to update the issue when
the CI fails on a flicker, but it takes time to do properly and it's not a priority.
On the contrary I propose to trust our memory: if we're wrong because we have closed
a flicker that is still happening, it will allow us to remind that we have this flicker to
fix and we can easily reopen the issue.
As Thomas mentioned on the chat, we should also update the release plan to include the
inactive flickers in the list of issue to check.
So for now I propose to close the following list of issues as inactive:
* XWIKI-14399: AddRemoveTagsTest#addAndDeleteTagFromTagPage is flickering
(https://jira.xwiki.org/browse/XWIKI-14399)
* XWIKI-14396: AnnotationsTest#addAndDeleteAnnotations is flickering
(https://jira.xwiki.org/browse/XWIKI-14396)
* XWIKI-14394: SectionTest.testSectionEditInWikiEditorWhenSyntax2x
(xwiki-enterprise-test-ui) is flaky (https://jira.xwiki.org/browse/XWIKI-14394)
* XWIKI-14386: appwithinminutes.AppsLiveTableTest.testEditApplication is possibly flaky
(https://jira.xwiki.org/browse/XWIKI-14386)
* XWIKI-14835: DeletePageTest#deletePageIsImpossibleWhenNoDeleteRights is flickering
(https://jira.xwiki.org/browse/XWIKI-14835)
* XWIKI-14860: LoginTest#testDataIsPreservedAfterLogin is flickering
(https://jira.xwiki.org/browse/XWIKI-14860)
And I propose in general to close the flickers we don't remember having seen after a
cycle as inactive.
WDYT?
Thanks for looking into this. I’d feel more comfortable if we can do some local testing
before closing any test issue.
For ex, I’d propose to run it N times (can also be done on a docker container agent, it’s
very easy to start a new container on ks* or locally) and if it doesn’t fail at all then
close it if it’s old enough.
Note: To repeat N times a test:
* For junit5: @RepeatedTest(N)
* For junit4: @Intermittent(repetition=N) - see
https://dev.xwiki.org/xwiki/bin/view/Community/Testing/
* At the shell level:
for i in {1..50}
do
mvn test -D…
done
Some rules I propose:
* N = 50 (for a test taking 3 minutes, that means 2.5 hours)
* Last update date of the test jira issue < Now() - 6 months
WDYT? Too hard?
Thanks
-Vincent
Simon
--
Simon Urli
Software Engineer at XWiki SAS
simon.urli(a)xwiki.com
More about us at http://www.xwiki.com