26 Mar
2019
26 Mar
'19
10:58 a.m.
On 26 Mar 2019, at 10:31, Simon Urli
<simon.urli(a)xwiki.com> wrote:
Hi everyone,
I was checking our list of flickering tests in JIRA
(https://jira.xwiki.org/issues/?jql=labels%20%3D%20flickering%20AND%20status…)
and I noticed that we had somehow old flickering test issue concerning test that I've
never seen failing.
So I propose we close some of them as inactive: the ones that we don't remember
having seen for a while. The ideal would be to have a mechanism to update the issue when
the CI fails on a flicker, but it takes time to do properly and it's not a priority.
On the contrary I propose to trust our memory: if we're wrong because we have closed
a flicker that is still happening, it will allow us to remind that we have this flicker to
fix and we can easily reopen the issue.
As Thomas mentioned on the chat, we should also update the release plan to include the
inactive flickers in the list of issue to check.
I should be able to easily create a report when any test fails inside our jenkins pipeline
and make it available similar to our clover report. I could indicate if it’s a known
flicker or not too in this report. That could compensate for the fact that we only keep 7
days of records in our jobs.
Would need to define the report format, whether it’s the same file updated at each run or
a different one. If the same one, then either:
* I’d need to parse it first in memory, add the new tests and overwrite the file
* or add to the bottom of the file which will grow quite large quickly
WDYT?
Thanks
-Vincent
So for now I propose to close the following list of issues as inactive:
* XWIKI-14399: AddRemoveTagsTest#addAndDeleteTagFromTagPage is flickering
(https://jira.xwiki.org/browse/XWIKI-14399)
* XWIKI-14396: AnnotationsTest#addAndDeleteAnnotations is flickering
(https://jira.xwiki.org/browse/XWIKI-14396)
* XWIKI-14394: SectionTest.testSectionEditInWikiEditorWhenSyntax2x
(xwiki-enterprise-test-ui) is flaky (https://jira.xwiki.org/browse/XWIKI-14394)
* XWIKI-14386: appwithinminutes.AppsLiveTableTest.testEditApplication is possibly flaky
(https://jira.xwiki.org/browse/XWIKI-14386)
* XWIKI-14835: DeletePageTest#deletePageIsImpossibleWhenNoDeleteRights is flickering
(https://jira.xwiki.org/browse/XWIKI-14835)
* XWIKI-14860: LoginTest#testDataIsPreservedAfterLogin is flickering
(https://jira.xwiki.org/browse/XWIKI-14860)
And I propose in general to close the flickers we don't remember having seen after a
cycle as inactive.
WDYT?
Simon
--
Simon Urli
Software Engineer at XWiki SAS
simon.urli(a)xwiki.com
More about us at http://www.xwiki.com