Re: [xwiki-devs] [Proposal] Cleaning of flickering tests

On 6 Sep 2019, at 10:27, Simon Urli <simon.urli(a)xwiki.com> wrote: Hi all, On 05/09/2019 17:40, Simon Urli wrote: > On 05/09/2019 17:24, Thomas Mortagne wrote: >> On Thu, Sep 5, 2019 at 3:43 PM Simon Urli <simon.urli(a)xwiki.com> wrote: >>> >>> Hi everyone, >>> >>> reopening this thread since I started to close some flicker issues as >>> part of BFD and got comments for those. >>> >>> So the last mails on this threads suggested to close the flicker issues >>> if we didn't manage to reproduce them locally after a repeated tests, >>> and that we didn't see them after a while. >>> >>> We didn't vote for those suggestion and I assumed a bit quick that I >>> could close some flicker issues that I personally don't remember about >>> on the CI after having tested them locally. >>> My point for doing that is the same as for the first mail I posted on >>> this thread: those flickers are old, and the code did change enough for >>> those to be fixed in a way or another. >> >> Being old does not always means the code leading to those failures >> changed that much. >> >>> >>> Now I might be completely wrong, and the flicker to happen again, but I >>> don't think it's a problem since we can really easily open back the >>> issues if it's the case. >>> >>> The other solution IMO is to indeed keep the issue open and in fact to >>> never really close them, because we just don't have time to investigate >>> each of them properly. >>> >>> I really don't see any value of keeping things open and don't act on >>> them, that's why I suggest to close them after doing the checks we >>> suggested before: >>> 1. try to repeat locally the failure; >> >> This is totally useless IMO unless you make sure that your computer is >> made super slow some way since that's the reason for most of the >> flickering tests. >> >>> 2. check that we didn't encounter those flickers since last cycle. >> >> This one is enough for me but the hard part is to knowing that. > Ok, so the proposal is now to check only the age since last time we saw them of the open flickers before closing them. >> >>> >>> So first question, do we all agree on that? >>> >>> Then for the second check, Vincent suggested to add some tooling: it >>> will be best, but it takes time to do. So on the meantime, as Thomas >>> also suggested, we could add a check in the release plan to create or >>> update all jira issues that concerns flickers. It would allow us to keep >>> some information about the liveness of our flickers. >>> >>> So second question, do you agree on that? >> >> Depends what it exactly means. Have some dedicated jira field to >> indicate when you saw it last ? Comment that you just saw that test >> failing again ? > My suggestion was about a dedicated JIRA field if possible. So, ok if I create a new custom field in JIRA for flickers, called "Date of last failure for flicker”?