[xwiki-devs] [Discussion] Release strategy and slippages - Timeboxing
Hi devs, We've been quite bad at following the release strategy we've defined so far: timeboxing. We've been constantly slipping our release dates during all the past releases (and not by 1 or 2 days, but by 2 weeks or more). Personally I find this not professional of us and for me it means that the date we give are just a joke now. I don't even see why we're giving dates since they can only be misleading to anyone who would act on them… At the current slippage rate, we should only give a month estimation to have a chance of being correct ;) There's only one reason we're really slipping IMO: we've forgotten that we're doing time boxing. Let me remind us what time boxing is and how it can be made to work: * It means releasing on a fixed date and releasing whatever is ready at that date. * The idea is also to do quick releases so that if a given feature is not in a release it'll be in the next one coming soon (we used to do 2 to 3 weeks releases at some points and recently we've been doing instead 3-5 weeks instead) * The reason for releasing often is because this pushes the bug fixes and new stuff to users ASAP and thus let them experiment with them and give us feedback (bugs, usability, etc) * The way to make timeboxing work is by having automated functional tests so that we can release safely knowing that our test suite will catch the problems if any. This means that whenever a commit is done, in the same commit there are tests proving that what is committed is working (and not several weeks later which would be anti-timeboxing). * it doesn't mean we cannot have a roadmap. What it means is that we should the maximum of what is in the roadmap in a given time slot. What's not done doesn't get released. This means that large features should be divided into small one that will fit. Developers need to think about what they're doing by constantly checking the release date and verifying what they can achieve for that date and make absolutely sure that what they have committed is working for that date (even if what they have committed is not finished). * Timeboxing is not about how many men/days you have available or whether people are on holidays or not. If you have less men/days you do less work. It doesn't change the release date. Basically you can only do timeboxing if you have good and automated quality control. The opposite of timeboxing is feature releases which lead to the following: * People committing not working stuff or without tests because they know they'll have time to fix it later on before the release (easy to think this since there's no release date, it's only when the features are done that it'll get released) * Release date become not important since they keep being pushed since what's important is to release planned features * Build failing all the time and developers not caring about it ("it can always be fixed later when we get close to the release date" - that's easy to say since there' no fixed release date) * Users seeing less frequent releases and giving less feedback to developers (thus helping less) * In general, quality dropping over time So we have several choices: 1) Forget timeboxing and do feature-based releases, i.e. we list features and we only release when they're done 2) Start doing real timeboxing again I've seen so many projects do 1) in the past with such bad results that for me there's no doubt that the only good strategy is 2). Especially for an open source project which has a strong community and we should be releasing stuff regularly and quickly to this community to get good feedback. Doing timeboxing is also a great way to improve the quality of our code. Timeboxing can only work fine if everyone agrees with it (we can revert stuff from a developer if it breaks things but it's a pain) and believe in the release date, so we'd need everyone's agreements. So WDYT, are we ok to resume doing timeboxing and go back on track? On my side I'm ok to do it and help enforce it. If we agree we should start now by releasing RC1 ASAP and give a new date for 3.4 final and release on that date. Thanks -Vincent
Hi Vincent, On Mon, Jan 16, 2012 at 10:09 AM, Vincent Massol <[email protected]> wrote:
Hi devs,
We've been quite bad at following the release strategy we've defined so far: timeboxing.
We've been constantly slipping our release dates during all the past releases (and not by 1 or 2 days, but by 2 weeks or more).
Personally I find this not professional of us and for me it means that the date we give are just a joke now. I don't even see why we're giving dates since they can only be misleading to anyone who would act on them… At the current slippage rate, we should only give a month estimation to have a chance of being correct ;)
There's only one reason we're really slipping IMO: we've forgotten that we're doing time boxing.
I don't believe this is the only reason, not even the main one. I don't think 3.4M1 was delayed because we wanted to do more. What I know is that the build was unstable during the winter holidays when most of the committers were offline so there wasn't anyone to investigate and fix the failing tests on CI. You can argue that the build was unstable because we don't do timeboxing, but even with timeboxing you can have: * integration tests failing because a commit in module A (which has its own unit tests passing) has side effects on module B. If both modules are under development, with lost of commits, then it's hard to track which commit broke the build. Jenkins is not very helpful regarding this. * flickering functional tests, both due to bad test coding but also due to the instability of Selenium (I don't feel it 100% reliable).
Let me remind us what time boxing is and how it can be made to work: * It means releasing on a fixed date and releasing whatever is ready at that date. * The idea is also to do quick releases so that if a given feature is not in a release it'll be in the next one coming soon (we used to do 2 to 3 weeks releases at some points and recently we've been doing instead 3-5 weeks instead) * The reason for releasing often is because this pushes the bug fixes and new stuff to users ASAP and thus let them experiment with them and give us feedback (bugs, usability, etc) * The way to make timeboxing work is by having automated functional tests so that we can release safely knowing that our test suite will catch the problems if any. This means that whenever a commit is done, in the same commit there are tests proving that what is committed is working (and not several weeks later which would be anti-timeboxing). * it doesn't mean we cannot have a roadmap. What it means is that we should the maximum of what is in the roadmap in a given time slot. What's not done doesn't get released. This means that large features should be divided into small one that will fit. Developers need to think about what they're doing by constantly checking the release date and verifying what they can achieve for that date and make absolutely sure that what they have committed is working for that date (even if what they have committed is not finished).
* Timeboxing is not about how many men/days you have available or whether people are on holidays or not. If you have less men/days you do less work. It doesn't change the release date.
You still need someone do to the release and we aren't that many committers so if we all take the winter holidays (as it is normal) then we have to adjust the release date.
Basically you can only do timeboxing if you have good and automated quality control.
The opposite of timeboxing is feature releases which lead to the following: * People committing not working stuff or without tests because they know they'll have time to fix it later on before the release (easy to think this since there's no release date, it's only when the features are done that it'll get released) * Release date become not important since they keep being pushed since what's important is to release planned features * Build failing all the time and developers not caring about it ("it can always be fixed later when we get close to the release date" - that's easy to say since there' no fixed release date) * Users seeing less frequent releases and giving less feedback to developers (thus helping less) * In general, quality dropping over time
So we have several choices:
1) Forget timeboxing and do feature-based releases, i.e. we list features and we only release when they're done 2) Start doing real timeboxing again
I've seen so many projects do 1) in the past with such bad results that for me there's no doubt that the only good strategy is 2). Especially for an open source project which has a strong community and we should be releasing stuff regularly and quickly to this community to get good feedback. Doing timeboxing is also a great way to improve the quality of our code.
Timeboxing can only work fine if everyone agrees with it (we can revert stuff from a developer if it breaks things but it's a pain) and believe in the release date, so we'd need everyone's agreements.
So WDYT, are we ok to resume doing timeboxing and go back on track?
I'm definitely for timeboxing. Thanks, Marius
On my side I'm ok to do it and help enforce it. If we agree we should start now by releasing RC1 ASAP and give a new date for 3.4 final and release on that date.
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Jan 16, 2012, at 10:14 AM, Marius Dumitru Florea wrote:
Hi Vincent,
On Mon, Jan 16, 2012 at 10:09 AM, Vincent Massol <[email protected]> wrote:
Hi devs,
We've been quite bad at following the release strategy we've defined so far: timeboxing.
We've been constantly slipping our release dates during all the past releases (and not by 1 or 2 days, but by 2 weeks or more).
Personally I find this not professional of us and for me it means that the date we give are just a joke now. I don't even see why we're giving dates since they can only be misleading to anyone who would act on them… At the current slippage rate, we should only give a month estimation to have a chance of being correct ;)
There's only one reason we're really slipping IMO: we've forgotten that we're doing time boxing.
I don't believe this is the only reason, not even the main one. I don't think 3.4M1 was delayed because we wanted to do more. What I know is that the build was unstable during the winter holidays when most of the committers were offline so there wasn't anyone to investigate and fix the failing tests on CI. You can argue that the build was unstable because we don't do timeboxing, but even with timeboxing you can have:
* integration tests failing because a commit in module A (which has its own unit tests passing) has side effects on module B. If both modules are under development, with lost of commits, then it's hard to track which commit broke the build. Jenkins is not very helpful regarding this.
Actually it does help since you can know exactly which commits breaks even if it's in another module.
* flickering functional tests, both due to bad test coding but also due to the instability of Selenium (I don't feel it 100% reliable).
So far I've never seen any instabillity caused by Selenium. Every time it's been our fault. There were a few times it case caused by selenium but it wasn't flickering, it was not working all the time ;) Having shorter releases help fix this because the longer you wait the more stuff accumulates. BTW I was not talking about 3.4M1 but about the past 20 or so releases we've done (if not more)… :) (I didn't get the time to compute the exact variation for past releases) Thanks -Vincent
Let me remind us what time boxing is and how it can be made to work: * It means releasing on a fixed date and releasing whatever is ready at that date. * The idea is also to do quick releases so that if a given feature is not in a release it'll be in the next one coming soon (we used to do 2 to 3 weeks releases at some points and recently we've been doing instead 3-5 weeks instead) * The reason for releasing often is because this pushes the bug fixes and new stuff to users ASAP and thus let them experiment with them and give us feedback (bugs, usability, etc) * The way to make timeboxing work is by having automated functional tests so that we can release safely knowing that our test suite will catch the problems if any. This means that whenever a commit is done, in the same commit there are tests proving that what is committed is working (and not several weeks later which would be anti-timeboxing). * it doesn't mean we cannot have a roadmap. What it means is that we should the maximum of what is in the roadmap in a given time slot. What's not done doesn't get released. This means that large features should be divided into small one that will fit. Developers need to think about what they're doing by constantly checking the release date and verifying what they can achieve for that date and make absolutely sure that what they have committed is working for that date (even if what they have committed is not finished).
* Timeboxing is not about how many men/days you have available or whether people are on holidays or not. If you have less men/days you do less work. It doesn't change the release date.
You still need someone do to the release and we aren't that many committers so if we all take the winter holidays (as it is normal) then we have to adjust the release date.
Basically you can only do timeboxing if you have good and automated quality control.
The opposite of timeboxing is feature releases which lead to the following: * People committing not working stuff or without tests because they know they'll have time to fix it later on before the release (easy to think this since there's no release date, it's only when the features are done that it'll get released) * Release date become not important since they keep being pushed since what's important is to release planned features * Build failing all the time and developers not caring about it ("it can always be fixed later when we get close to the release date" - that's easy to say since there' no fixed release date) * Users seeing less frequent releases and giving less feedback to developers (thus helping less) * In general, quality dropping over time
So we have several choices:
1) Forget timeboxing and do feature-based releases, i.e. we list features and we only release when they're done 2) Start doing real timeboxing again
I've seen so many projects do 1) in the past with such bad results that for me there's no doubt that the only good strategy is 2). Especially for an open source project which has a strong community and we should be releasing stuff regularly and quickly to this community to get good feedback. Doing timeboxing is also a great way to improve the quality of our code.
Timeboxing can only work fine if everyone agrees with it (we can revert stuff from a developer if it breaks things but it's a pain) and believe in the release date, so we'd need everyone's agreements.
So WDYT, are we ok to resume doing timeboxing and go back on track?
I'm definitely for timeboxing.
Thanks, Marius
On my side I'm ok to do it and help enforce it. If we agree we should start now by releasing RC1 ASAP and give a new date for 3.4 final and release on that date.
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
+1 for time boxing On Mon, Jan 16, 2012 at 11:37, Vincent Massol <[email protected]> wrote:
On Jan 16, 2012, at 10:14 AM, Marius Dumitru Florea wrote:
Hi Vincent,
On Mon, Jan 16, 2012 at 10:09 AM, Vincent Massol <[email protected]> wrote:
Hi devs,
We've been quite bad at following the release strategy we've defined so far: timeboxing.
We've been constantly slipping our release dates during all the past releases (and not by 1 or 2 days, but by 2 weeks or more).
Personally I find this not professional of us and for me it means that the date we give are just a joke now. I don't even see why we're giving dates since they can only be misleading to anyone who would act on them… At the current slippage rate, we should only give a month estimation to have a chance of being correct ;)
There's only one reason we're really slipping IMO: we've forgotten that we're doing time boxing.
I don't believe this is the only reason, not even the main one. I don't think 3.4M1 was delayed because we wanted to do more. What I know is that the build was unstable during the winter holidays when most of the committers were offline so there wasn't anyone to investigate and fix the failing tests on CI. You can argue that the build was unstable because we don't do timeboxing, but even with timeboxing you can have:
* integration tests failing because a commit in module A (which has its own unit tests passing) has side effects on module B. If both modules are under development, with lost of commits, then it's hard to track which commit broke the build. Jenkins is not very helpful regarding this.
Actually it does help since you can know exactly which commits breaks even if it's in another module.
* flickering functional tests, both due to bad test coding but also due to the instability of Selenium (I don't feel it 100% reliable).
So far I've never seen any instabillity caused by Selenium. Every time it's been our fault. There were a few times it case caused by selenium but it wasn't flickering, it was not working all the time ;)
Having shorter releases help fix this because the longer you wait the more stuff accumulates.
The only problem with shorter releases is that this means more work for the release manager and less time for him to concentrate on the things he has to do. Some organizations have release engineers and managers full time. Another problem regarding tests, I think we should enforce the rule to: - add features, improvements just in milestone (M1, M2) - concentrate on bug fixes for the features added in the RCs (RC1, RC2) Sometimes we add features in the last moment of a RC and we don't have enough time to verify the integration or they introduce a new series of falling test. Thanks, Caty
BTW I was not talking about 3.4M1 but about the past 20 or so releases we've done (if not more)… :) (I didn't get the time to compute the exact variation for past releases)
Thanks -Vincent
Let me remind us what time boxing is and how it can be made to work: * It means releasing on a fixed date and releasing whatever is ready at that date. * The idea is also to do quick releases so that if a given feature is not in a release it'll be in the next one coming soon (we used to do 2 to 3 weeks releases at some points and recently we've been doing instead 3-5 weeks instead) * The reason for releasing often is because this pushes the bug fixes and new stuff to users ASAP and thus let them experiment with them and give us feedback (bugs, usability, etc) * The way to make timeboxing work is by having automated functional tests so that we can release safely knowing that our test suite will catch the problems if any. This means that whenever a commit is done, in the same commit there are tests proving that what is committed is working (and not several weeks later which would be anti-timeboxing). * it doesn't mean we cannot have a roadmap. What it means is that we should the maximum of what is in the roadmap in a given time slot. What's not done doesn't get released. This means that large features should be divided into small one that will fit. Developers need to think about what they're doing by constantly checking the release date and verifying what they can achieve for that date and make absolutely sure that what they have committed is working for that date (even if what they have committed is not finished).
* Timeboxing is not about how many men/days you have available or whether people are on holidays or not. If you have less men/days you do less work. It doesn't change the release date.
You still need someone do to the release and we aren't that many committers so if we all take the winter holidays (as it is normal) then we have to adjust the release date.
Basically you can only do timeboxing if you have good and automated
quality control.
The opposite of timeboxing is feature releases which lead to the
following:
* People committing not working stuff or without tests because they know they'll have time to fix it later on before the release (easy to think this since there's no release date, it's only when the features are done that it'll get released) * Release date become not important since they keep being pushed since what's important is to release planned features * Build failing all the time and developers not caring about it ("it can always be fixed later when we get close to the release date" - that's easy to say since there' no fixed release date) * Users seeing less frequent releases and giving less feedback to developers (thus helping less) * In general, quality dropping over time
So we have several choices:
1) Forget timeboxing and do feature-based releases, i.e. we list features and we only release when they're done 2) Start doing real timeboxing again
I've seen so many projects do 1) in the past with such bad results that for me there's no doubt that the only good strategy is 2). Especially for an open source project which has a strong community and we should be releasing stuff regularly and quickly to this community to get good feedback. Doing timeboxing is also a great way to improve the quality of our code.
Timeboxing can only work fine if everyone agrees with it (we can revert stuff from a developer if it breaks things but it's a pain) and believe in the release date, so we'd need everyone's agreements.
So WDYT, are we ok to resume doing timeboxing and go back on track?
I'm definitely for timeboxing.
Thanks, Marius
On my side I'm ok to do it and help enforce it. If we agree we should
start now by releasing RC1 ASAP and give a new date for 3.4 final and release on that date.
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Jan 16, 2012, at 11:33 AM, Ecaterina Moraru (Valica) wrote:
+1 for time boxing
On Mon, Jan 16, 2012 at 11:37, Vincent Massol <[email protected]> wrote:
On Jan 16, 2012, at 10:14 AM, Marius Dumitru Florea wrote:
Hi Vincent,
On Mon, Jan 16, 2012 at 10:09 AM, Vincent Massol <[email protected]> wrote:
Hi devs,
We've been quite bad at following the release strategy we've defined so far: timeboxing.
We've been constantly slipping our release dates during all the past releases (and not by 1 or 2 days, but by 2 weeks or more).
Personally I find this not professional of us and for me it means that the date we give are just a joke now. I don't even see why we're giving dates since they can only be misleading to anyone who would act on them… At the current slippage rate, we should only give a month estimation to have a chance of being correct ;)
There's only one reason we're really slipping IMO: we've forgotten that we're doing time boxing.
I don't believe this is the only reason, not even the main one. I don't think 3.4M1 was delayed because we wanted to do more. What I know is that the build was unstable during the winter holidays when most of the committers were offline so there wasn't anyone to investigate and fix the failing tests on CI. You can argue that the build was unstable because we don't do timeboxing, but even with timeboxing you can have:
* integration tests failing because a commit in module A (which has its own unit tests passing) has side effects on module B. If both modules are under development, with lost of commits, then it's hard to track which commit broke the build. Jenkins is not very helpful regarding this.
Actually it does help since you can know exactly which commits breaks even if it's in another module.
* flickering functional tests, both due to bad test coding but also due to the instability of Selenium (I don't feel it 100% reliable).
So far I've never seen any instabillity caused by Selenium. Every time it's been our fault. There were a few times it case caused by selenium but it wasn't flickering, it was not working all the time ;)
Having shorter releases help fix this because the longer you wait the more stuff accumulates.
The only problem with shorter releases is that this means more work for the release manager and less time for him to concentrate on the things he has to do
Yes which is why we absolutely need to work no reducing the time it takes to release. But with Sergiu's script we're going in the good direction. It would be interesting to assess the full time we need for a release now that we have Sergiu's scripts and work on improving the parts that still take time. Thanks -Vincent
. Some organizations have release engineers and managers full time.
Another problem regarding tests, I think we should enforce the rule to: - add features, improvements just in milestone (M1, M2) - concentrate on bug fixes for the features added in the RCs (RC1, RC2) Sometimes we add features in the last moment of a RC and we don't have enough time to verify the integration or they introduce a new series of falling test.
Thanks, Caty
BTW I was not talking about 3.4M1 but about the past 20 or so releases we've done (if not more)… :) (I didn't get the time to compute the exact variation for past releases)
Thanks -Vincent
Let me remind us what time boxing is and how it can be made to work: * It means releasing on a fixed date and releasing whatever is ready at that date. * The idea is also to do quick releases so that if a given feature is not in a release it'll be in the next one coming soon (we used to do 2 to 3 weeks releases at some points and recently we've been doing instead 3-5 weeks instead) * The reason for releasing often is because this pushes the bug fixes and new stuff to users ASAP and thus let them experiment with them and give us feedback (bugs, usability, etc) * The way to make timeboxing work is by having automated functional tests so that we can release safely knowing that our test suite will catch the problems if any. This means that whenever a commit is done, in the same commit there are tests proving that what is committed is working (and not several weeks later which would be anti-timeboxing). * it doesn't mean we cannot have a roadmap. What it means is that we should the maximum of what is in the roadmap in a given time slot. What's not done doesn't get released. This means that large features should be divided into small one that will fit. Developers need to think about what they're doing by constantly checking the release date and verifying what they can achieve for that date and make absolutely sure that what they have committed is working for that date (even if what they have committed is not finished).
* Timeboxing is not about how many men/days you have available or whether people are on holidays or not. If you have less men/days you do less work. It doesn't change the release date.
You still need someone do to the release and we aren't that many committers so if we all take the winter holidays (as it is normal) then we have to adjust the release date.
Basically you can only do timeboxing if you have good and automated
quality control.
The opposite of timeboxing is feature releases which lead to the
following:
* People committing not working stuff or without tests because they know they'll have time to fix it later on before the release (easy to think this since there's no release date, it's only when the features are done that it'll get released) * Release date become not important since they keep being pushed since what's important is to release planned features * Build failing all the time and developers not caring about it ("it can always be fixed later when we get close to the release date" - that's easy to say since there' no fixed release date) * Users seeing less frequent releases and giving less feedback to developers (thus helping less) * In general, quality dropping over time
So we have several choices:
1) Forget timeboxing and do feature-based releases, i.e. we list features and we only release when they're done 2) Start doing real timeboxing again
I've seen so many projects do 1) in the past with such bad results that for me there's no doubt that the only good strategy is 2). Especially for an open source project which has a strong community and we should be releasing stuff regularly and quickly to this community to get good feedback. Doing timeboxing is also a great way to improve the quality of our code.
Timeboxing can only work fine if everyone agrees with it (we can revert stuff from a developer if it breaks things but it's a pain) and believe in the release date, so we'd need everyone's agreements.
So WDYT, are we ok to resume doing timeboxing and go back on track?
I'm definitely for timeboxing.
Thanks, Marius
On my side I'm ok to do it and help enforce it. If we agree we should
start now by releasing RC1 ASAP and give a new date for 3.4 final and release on that date.
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
participants (3)
-
Ecaterina Moraru (Valica) -
Marius Dumitru Florea -
Vincent Massol