[xwiki-devs] [Important] More stability needed - To all committers
Hi everyone, I don't know if you're like me but I have the strong feeling we should be better at not introducing regressions. One of the goal of XE 1.3 and now again XE 1.4 was more stability and more automated tests. Several potential problems are occurring now: 1) People using XWiki are expecting more stability when they upgrade. This is due to the fact that XWiki is improving in general and more people use it. It's expected that it'll just work and that's a reasonable expectations 2) We've introduced several important regressions for the past *4* releases (login/logout/RMUI/Escapes and more). We're following a bas trend. 3) We've recently introduced several storage changes (Sergiu and Artem) and I haven't seen tests that would prove what was working before is still working. I'm not saying it isn't but last time we made a change to the storage area it took us several months to stabilize it and we cannot do that again. 4) We're committing more code than tests meaning the overall quality of XWiki is degrading :( Thus I'd like to propose that: A) We become very very careful when committing things and we only commit when we can *guarantee* that what we've done is working (with a given level of confidence of course). This can only be achieved through tests being committed at the same time as the code is committed B) We stop putting stuff that are NOT critical in point releases. For example dangerous changes were done in the WYSIWYG editor for 1.3.1 and I'm not confident this was a good thing, and certainly not with so little tests and verifications since we know that whenever we touch that editor we introduce problems elsewhere. C) In general we reduce the number of changes that we commit and instead we focus on tests and stability. This is indeed one of the general goals for 1.4. WDYT? Thanks -Vincent
On Sun, Apr 6, 2008 at 1:13 PM, Vincent Massol <[email protected]> wrote:
Hi everyone,
I don't know if you're like me but I have the strong feeling we should be better at not introducing regressions. One of the goal of XE 1.3 and now again XE 1.4 was more stability and more automated tests.
WDYT?
Thanks -Vincent
I'm not a developer but a user, nevertheless I'd like to give you my 2 cents: I would prefer if you would do minor changes with only small or no impact to the user (simply copy the new files into this and this folder and you get a performance boost or higher security or whatsoever; or even an integrated update-manager a la firefox) to the stable branch (maybe with somebody who is responsible for the stable one) and beside this branch a developing branch with a longer release-cycle. The benefit would be, that you have more time to develop stuff (maybe bigger changes) and it does not affect the general user because he is supposed to use the stable branch. Right now the release-cycle is too fast for me as a user (sound strange but it's true for me) to keep up with Xwiki. Cheers, Squirrel
Hi Squirrel, On Apr 6, 2008, at 8:51 PM, Squirrel wrote:
On Sun, Apr 6, 2008 at 1:13 PM, Vincent Massol <[email protected]> wrote:
Hi everyone,
I don't know if you're like me but I have the strong feeling we should be better at not introducing regressions. One of the goal of XE 1.3 and now again XE 1.4 was more stability and more automated tests.
WDYT?
Thanks -Vincent
I'm not a developer but a user, nevertheless I'd like to give you my 2 cents:
I would prefer if you would do minor changes with only small or no impact to the user (simply copy the new files into this and this folder and you get a performance boost or higher security or whatsoever;
If it were that simple! There's no silver bullet unfortunately. Adding stuff that suddenly makes the application faster, better, etc with no change to the application is very rare... :) Or maybe I missed what you were saying?
or even an integrated update-manager a la firefox) to the stable branch (maybe with somebody who is responsible for the stable one) and beside this branch a developing branch with a longer release-cycle. The benefit would be, that you have more time to develop stuff (maybe bigger changes) and it does not affect the general user because he is supposed to use the stable branch. Right now the release-cycle is too fast for me as a user (sound strange but it's true for me) to keep up with Xwiki.
That's an interesting statement that requires more input: 1) Why do you need to upgrade? 2) Why is hard to upgrade? Re the stable/branch/etc that's exactly what we do... :) We have the milestones/RC that last for about 3-4 months so we only release a new version every 3-4 months. We can hardly call this quick! Moreover you should know that the longer a release the less stable it is in general since there are more stuff in it. From a user point of view the best is to get the shortest possible release cycle. This is what ensures the best quality in general. The Hudson project for example releases every day or every 2 days and that's the greatest possible. I'd love that we be able to do that with XWiki and this is my personal target. Thanks for your feedback. -Vincent
On Sun, Apr 6, 2008 at 2:03 PM, Vincent Massol <[email protected]> wrote:
Hi Squirrel,
If it were that simple! There's no silver bullet unfortunately. Adding stuff that suddenly makes the application faster, better, etc with no change to the application is very rare... :) Or maybe I missed what you were saying?
Well, I didn't mean no change to the application, I mean no bigger changes which would force the user to do some changes too. Ie. if you change some tables or changes new things / features who would break the customization the user did to the skin ie.. It's only general speaking, but as a user I don't want any surprises ;-) That's an interesting statement that requires more input:
1) Why do you need to upgrade? 2) Why is hard to upgrade?
1) If something is new you can bet your a..money that the user want it. Beside often there are improvements which the user drools over. Sometimes they are security updates so you should update (I'm speaking now generally) 2) I'm coming from the Joomla! crowd and for me it was a lot easier to update my Joomla installation than the Xwiki. Also the installation itself was way more straight forward (ie. the config file was set up during the installation and stable even after updates). Nevertheless, I'm now a Xwiki user ;-)
Re the stable/branch/etc that's exactly what we do... :) We have the milestones/RC that last for about 3-4 months so we only release a new version every 3-4 months. We can hardly call this quick! Moreover you should know that the longer a release the less stable it is in general since there are more stuff in it. From a user point of view the best is to get the shortest possible release cycle. This is what ensures the best quality in general. The Hudson project for example releases every day or every 2 days and that's the greatest possible. I'd love that we be able to do that with XWiki and this is my personal target.
Thanks for your feedback. -Vincent
I'm coming from the Joomla/Drupal corner and therefore I'm more used to longer release cycles for bigger improvements. I have nothing against frequent updates on the stable branch (security fixes, minor improvements) as long as they don't break things and somehow I have the feeling if Xwiki gets more attention from the general user side (not developers or advanced users) that could lead into trouble as users will start howling (but right now the entry barrier is too high for the average Joomla!-user, IMHO). I'm not so sure if it's indeed the best thing for the user to get daily updates, in the labor this might work, but out in the wild there are maybe way more side-effects and they need time to pop-up... However, my statements are not really qualified, so I'm sure you will do the right thing and I hope you get my statements not wrong in anyway. :-)
On Apr 6, 2008, at 9:44 PM, Squirrel wrote:
On Sun, Apr 6, 2008 at 2:03 PM, Vincent Massol <[email protected]> wrote:
Hi Squirrel,
If it were that simple! There's no silver bullet unfortunately. Adding stuff that suddenly makes the application faster, better, etc with no change to the application is very rare... :) Or maybe I missed what you were saying?
Well, I didn't mean no change to the application, I mean no bigger changes which would force the user to do some changes too. Ie. if you change some tables or changes new things / features who would break the customization the user did to the skin ie.. It's only general speaking, but as a user I don't want any surprises ;-)
Sure, I don't think anyone would disagree with this. Re the database we have automated upgrades exactly for this purpose. As a user you don't have to do anything (unless we missed something and there's a bug of course).
That's an interesting statement that requires more input:
1) Why do you need to upgrade? 2) Why is hard to upgrade?
1) If something is new you can bet your a..money that the user want it.
Right so you agree that faster release cycles are better as otherwise that means no new stuff for a very long time ;)
Beside often there are improvements which the user drools over. Sometimes they are security updates so you should update (I'm speaking now generally)
2) I'm coming from the Joomla! crowd and for me it was a lot easier to update my Joomla installation than the Xwiki. Also the installation itself was way more straight forward (ie. the config file was set up during the installation and stable even after updates). Nevertheless, I'm now a Xwiki user ;-)
What do you find hard to upgrade xwiki? We can certainly improve but it's pretty straightforward right now: you replace your current WAR with the new one and off you go... (apart from keeping your xwiki.cfg and hibernate.cfg.xml files). Re the 2 config files, I'd also like to move them out of the WAR so that upgrading the WAR doesn't involve saving them and restoring them. I'm currently working on the new configuration component that'll allow this.
Re the stable/branch/etc that's exactly what we do... :) We have the milestones/RC that last for about 3-4 months so we only release a new version every 3-4 months. We can hardly call this quick! Moreover you should know that the longer a release the less stable it is in general since there are more stuff in it. From a user point of view the best is to get the shortest possible release cycle. This is what ensures the best quality in general. The Hudson project for example releases every day or every 2 days and that's the greatest possible. I'd love that we be able to do that with XWiki and this is my personal target.
Thanks for your feedback. -Vincent
I'm coming from the Joomla/Drupal corner and therefore I'm more used to longer release cycles for bigger improvements. I have nothing against frequent updates on the stable branch (security fixes, minor improvements) as long as they don't break things and somehow I have the feeling if Xwiki gets more attention from the general user side (not developers or advanced users) that could lead into trouble as users will start howling (but right now the entry barrier is too high for the average Joomla!-user, IMHO). I'm not so sure if it's indeed the best thing for the user to get daily updates,
Sure but it's not because there are updates that you *have* to upgrade... The user still decides when he upgrades.
in the labor this might work, but out in the wild there are maybe way more side-effects and they need time to pop-up...
However, my statements are not really qualified, so I'm sure you will do the right thing and I hope you get my statements not wrong in anyway. :-)
Not at all. Feedback is always good, thanks :) -Vincent
On Sun, Apr 6, 2008 at 2:59 PM, Vincent Massol <[email protected]> wrote:
Right so you agree that faster release cycles are better as otherwise that means no new stuff for a very long time ;)
Hehe, owned.
What do you find hard to upgrade xwiki?
Let me think over that again... ;-)
Sure but it's not because there are updates that you *have* to
upgrade... The user still decides when he upgrades.
Agreed. Hmmm...maybe I'm just not sure about how much customization I can do without breaking anything (after the next update/upgrade)...the customization I do (on macros, classes and so forth), are they stored in files or in the DB? I guess in the DB, that's what confuses me as a Joomla! user...I guess I have to read more in the docs... Cheers, Squirrel PS: I really like the way the Xwiki team talks to the community!
From my server architect's point of view, I see XWiki as a still young and very active project... Not yet fully stable in its perimeter and final aims: lots of things are still opened and being studied. For the time being, new releases don't only involve bug correction but new functional and design features. There is still this euphoric progression in the project and this is what I was looking for when I chose XWiki. New ideas are coming every day and they are integrated progressively but it means new releases bring sometimes new feature that, as a user, you want to integrate for sure. But if there are some regressions as Vincent explained, the eager user can be deceived. As a developer, I can also see new crucial modifications of the core such as the component design which is very important for me as a potential XWiki plugin/app/core developer and logically, I know this kind of evolution always brings lots of unforeseen regressions. Anyway, I don't find the releases are too fast or too slow... I trust the xwiki team to solve the small regression problems :)
I'm not against faster releases... Just keep identifying important, meaningful and stable releases. Considering the config files, good thing to put them outside the WAR. You could even let the hibernate.cfg.xml (from the installation point of view, knowing hibernate is behind all of this is not important) in the war and put some configuration parameters linked to it in the xwiki.cfg regards Pascal On 4/6/08, Squirrel <[email protected]> wrote:
On Sun, Apr 6, 2008 at 2:59 PM, Vincent Massol <[email protected]> wrote:
Right so you agree that faster release cycles are better as otherwise that means no new stuff for a very long time ;)
Hehe, owned.
What do you find hard to upgrade xwiki?
Let me think over that again... ;-)
Sure but it's not because there are updates that you *have* to
upgrade... The user still decides when he upgrades.
Agreed. Hmmm...maybe I'm just not sure about how much customization I can do without breaking anything (after the next update/upgrade)...the customization I do (on macros, classes and so forth), are they stored in files or in the DB? I guess in the DB, that's what confuses me as a Joomla! user...I guess I have to read more in the docs...
Cheers, Squirrel
PS: I really like the way the Xwiki team talks to the community! _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Vincent Massol wrote:
Hi everyone,
I don't know if you're like me but I have the strong feeling we should be better at not introducing regressions. One of the goal of XE 1.3 and now again XE 1.4 was more stability and more automated tests.
Several potential problems are occurring now:
1) People using XWiki are expecting more stability when they upgrade. This is due to the fact that XWiki is improving in general and more people use it. It's expected that it'll just work and that's a reasonable expectations 2) We've introduced several important regressions for the past *4* releases (login/logout/RMUI/Escapes and more). We're following a bas trend. 3) We've recently introduced several storage changes (Sergiu and Artem) and I haven't seen tests that would prove what was working before is still working. I'm not saying it isn't but last time we made a change to the storage area it took us several months to stabilize it and we cannot do that again. 4) We're committing more code than tests meaning the overall quality of XWiki is degrading :(
Thus I'd like to propose that:
A) We become very very careful when committing things and we only commit when we can *guarantee* that what we've done is working (with a given level of confidence of course). This can only be achieved through tests being committed at the same time as the code is committed B) We stop putting stuff that are NOT critical in point releases. For example dangerous changes were done in the WYSIWYG editor for 1.3.1 and I'm not confident this was a good thing, and certainly not with so little tests and verifications since we know that whenever we touch that editor we introduce problems elsewhere. C) In general we reduce the number of changes that we commit and instead we focus on tests and stability. This is indeed one of the general goals for 1.4.
WDYT?
While I would like that very much, most of the time I find it almost impossible to write (meaningful) tests. Sure, when dealing with user interface features, we can write selenium tests to click here and there and see what happens, and when dealing with algorithmic stuff we can write a unit test to see if an input gives the expected output. But there are things that can't be tested that way, because they need a running wiki. For example, how am I to test the attachments storage? Mocking the storage is not the right thing, as that is what I am testing. Selenium tests are not suited for this, as I can't upload files. I could abuse the selenium framework to write tests using programming, mimicking the upload action, but that also requires too much work and code duplication. I hope that when we'll have everything as a component, testing will be a lot easier. -- Sergiu Dumitriu http://purl.org/net/sergiu/
On Apr 7, 2008, at 11:35 AM, Sergiu Dumitriu wrote: [snip]
While I would like that very much, most of the time I find it almost impossible to write (meaningful) tests. Sure, when dealing with user interface features, we can write selenium tests to click here and there and see what happens, and when dealing with algorithmic stuff we can write a unit test to see if an input gives the expected output. But there are things that can't be tested that way, because they need a running wiki. For example, how am I to test the attachments storage?
I haven't thought much about this specific use case but I can think of several ideas: 1) Use DBUnit 2) Have automated tests that manipulates attachments through our UI (create attachment, delete attachment, rollback doc version, etc) 3) We need to have several databases available (from xwiki.org, from curriki for example) in different versions: 0.9.840, 1.0, 1.1, 1.2 and 1.3 and we need to run the test suite on all of them. Right now we must do 2) for sure and someone need to work on 3). I don't think we need 1). But my mail was not really about this. It's a more general mail that is asking everyone to be careful and whenever we make a change to ensure that we have tests *as much as possible*, think whether it can potentially break something, etc. To be more cautious. I've seen several changes recently that were committed quickly without tests and this is what I'd like to fight against. Thanks -Vincent
Mocking the storage is not the right thing, as that is what I am testing. Selenium tests are not suited for this, as I can't upload files. I could abuse the selenium framework to write tests using programming, mimicking the upload action, but that also requires too much work and code duplication.
I hope that when we'll have everything as a component, testing will be a lot easier. -- Sergiu Dumitriu http://purl.org/net/sergiu/
Perhaps I shouldn't intrude in this thread but I ran across this maven plugin for a test tool called DBUnit http://dbunit.sourceforge.net/ http://mojo.codehaus.org/dbunit-maven-plugin/ I haven't used this tool however, I was thinking you could perhaps test 2 things at once. You could start up an XWiki using ant exec and then write some junit test for the XMLRPC interface to execute an api that saves data into database then use the dbunit compare to compare the a dataset saved in the database with a previously set of (verified) data extracted with dbunit export. Just a thought. - Glenn Everitt -- View this message in context: http://www.nabble.com/-Important--More-stability-needed---To-all-committers-... Sent from the XWiki- Dev mailing list archive at Nabble.com.
Hi Vincent, Vincent Massol wrote:
Hi everyone,
I don't know if you're like me but I have the strong feeling we should be better at not introducing regressions. One of the goal of XE 1.3 and now again XE 1.4 was more stability and more automated tests.
Several potential problems are occurring now:
[...] 4) We're committing more code than tests meaning the overall quality of XWiki is degrading :(
not really adequate to the topic of the thread, but also an important aspect:
From my experiment in developing a plugin for XWiki (an additional link filter, in fact), I observed some design flaws in the core, that make it very hard to test single components/classes without having dependencies to the whole system including the database stuff.
Some few design changes and refactorings in the core could improve the testability strongly. If you (or other commiters) are interested in, I'll give you you some examples...
Thus I'd like to propose that:
A) We become very very careful when committing things and we only commit when we can *guarantee* that what we've done is working (with a given level of confidence of course). This can only be achieved through tests being committed at the same time as the code is committed [...] C) In general we reduce the number of changes that we commit and instead we focus on tests and stability. This is indeed one of the general goals for 1.4.
WDYT? +1
-Marco
participants (6)
-
Glenn Everitt -
Marco K. -
Pascal Voitot -
Sergiu Dumitriu -
Squirrel -
Vincent Massol