Re: [xwiki-devs] [Proposal] Cleaning of flickering tests

> On 6 Sep 2019, at 10:35, Thomas Mortagne <thomas.mortagne(a)xwiki.com> wrote: > > On Fri, Sep 6, 2019 at 10:32 AM Vincent Massol <vincent(a)massol.net> wrote: >> >> Hi Simon, >> >>> On 6 Sep 2019, at 10:27, Simon Urli <simon.urli(a)xwiki.com> wrote: >>> >>> Hi all, >>> >>> On 05/09/2019 17:40, Simon Urli wrote: >>>> On 05/09/2019 17:24, Thomas Mortagne wrote: >>>>> On Thu, Sep 5, 2019 at 3:43 PM Simon Urli <simon.urli(a)xwiki.com> wrote: >>>>>> >>>>>> Hi everyone, >>>>>> >>>>>> reopening this thread since I started to close some flicker issues as >>>>>> part of BFD and got comments for those. >>>>>> >>>>>> So the last mails on this threads suggested to close the flicker issues >>>>>> if we didn't manage to reproduce them locally after a repeated tests, >>>>>> and that we didn't see them after a while. >>>>>> >>>>>> We didn't vote for those suggestion and I assumed a bit quick that I >>>>>> could close some flicker issues that I personally don't remember about >>>>>> on the CI after having tested them locally. >>>>>> My point for doing that is the same as for the first mail I posted on >>>>>> this thread: those flickers are old, and the code did change enough for >>>>>> those to be fixed in a way or another. >>>>> >>>>> Being old does not always means the code leading to those failures >>>>> changed that much. >>>>> >>>>>> >>>>>> Now I might be completely wrong, and the flicker to happen again, but I >>>>>> don't think it's a problem since we can really easily open back the >>>>>> issues if it's the case. >>>>>> >>>>>> The other solution IMO is to indeed keep the issue open and in fact to >>>>>> never really close them, because we just don't have time to investigate >>>>>> each of them properly. >>>>>> >>>>>> I really don't see any value of keeping things open and don't act on >>>>>> them, that's why I suggest to close them after doing the checks we >>>>>> suggested before: >>>>>> 1. try to repeat locally the failure; >>>>> >>>>> This is totally useless IMO unless you make sure that your computer is >>>>> made super slow some way since that's the reason for most of the >>>>> flickering tests. >>>>> >>>>>> 2. check that we didn't encounter those flickers since last cycle. >>>>> >>>>> This one is enough for me but the hard part is to knowing that. >>>> Ok, so the proposal is now to check only the age since last time we saw them of the open flickers before closing them. >>>>> >>>>>> >>>>>> So first question, do we all agree on that? >>>>>> >>>>>> Then for the second check, Vincent suggested to add some tooling: it >>>>>> will be best, but it takes time to do. So on the meantime, as Thomas >>>>>> also suggested, we could add a check in the release plan to create or >>>>>> update all jira issues that concerns flickers. It would allow us to keep >>>>>> some information about the liveness of our flickers. >>>>>> >>>>>> So second question, do you agree on that? >>>>> >>>>> Depends what it exactly means. Have some dedicated jira field to >>>>> indicate when you saw it last ? Comment that you just saw that test >>>>> failing again ? >>>> My suggestion was about a dedicated JIRA field if possible. >>> >>> So, ok if I create a new custom field in JIRA for flickers, called "Date of last failure for flicker”? >> >> [snip] >> >> I don’t see how it’ll help since it’ll never be up to date, and the old value will remain making us think it’s not been flickering for a long time. > > In my mind the idea is not so much to use this field as a criteria to > close an issue but as a criteria to not close it. ok, as long as we don’t use it for closing, I’m fine :) Thanks -Vincent > >> >> Thanks >> -Vincent >> > > > -- > Thomas Mortagne