Hi, Here's a list Catalin and I came up with while brainstorming about ideas around testing for XWiki. Please feel free to comment, argue, suggest new ideas, etc. 0) Agree that we want to have our functional tests written in Java. Vs using Selenium IDE only Pros: - DSL -> easy to write and low maintenance (if a UI changes it's fixed in one place) - ability to write more complex tests with branch control (if, loop, etc) - ability to write tests easily (try creating a test that types some content in our Tiny MCE editor and you'll understand that part ;-)) - run automatically as part of our build/CI Cons: - require a minimal knowledge of Java. Thus it's going to prevent us from having non technical people write functional tests for us. I'm currently 100% convinced that our Java choice is right but I know Ludovic isn't fully convinced which is why I'm listing this point here to see what others think. 1) Finish setting up our Java functional test fwk. Goals: - make it as easy as possible to use. - make it low maintenance (ie tests shouldn't fail often and if some UI is changed it should be very easy to fix the tests) Tasks: - Continue improving our DSL for testing. Specifically add the ability to test over several skins and create one DSL per skin. - Add DSL methods for asserting the rendered HTML (cf Catalin's tests in http://jira.xwiki.org/jira/browse/XWIKI-1207) 2) Add clover support in the build to assert Test Coverage (TC) We need this to evaluate easily what we are not testing. It should report on both unit and functional tests execution. Note: This is easy to do with the nice Clover plugin for m2 ;-) (I'm the author of that plugin) 3) Add functional tests to cover 30% of the code under test. This is a first milestone that we should aim towards. For this we need to have 2) to see where we stand. Our target goal should be in the range of 60-70% TC. For my past experience this is a good ballpark figure and having this results in: a) well tested code and confidence something new doesn't break what exists. This is very interesting for us in the 1.x time frame as we don't want to add regressions to 1.0. But also it gives us greater confidence for redesigning internal parts of XWiki for implementing the 2.0 architecture b) the ability to release in production at each release (provided new features get tests added when they're added) 4) Explore in-VM testing The idea is to have several VM (Virtual Machines) running different OSes for example (Windows, linux, Mac) and a combination of Databases (Oracle, MySQL, PostGre). You run the XWiki maven build in them so that the tests get executed in VM. This allows 2 things: a) test for different environment. For example this tests our start/ stop scripts, our database setups b) this allows more complex tests to be written like: virtual wikis c) this allows to implement hard to write tests like the "add attachment" one. For this you need the attachment to be on your file system. With a VM you control the full setup of the VM and you can restore the VM at each run 5) create a test plan listing tests to be written We need this written in JIRA. This goes with 3). 6) Explore using Reality QA This is a nice tool/service created by Patrick Lightbody (who's a friend). See http://www.realityqa.com/screencasts/realitycheck_tutorial/ RealityCheck.htm to see it in action. Right now the main issue is that it won't run selenium tests written in Java but Patrick says it's coming: http://community.realityqa.com/thread.jspa?threadID=1010 The other thing that could be nice is the ability to upload our app inside their VM: http://community.realityqa.com/thread/1011 (but we can host our VMs to start with). What Reality QA buys us is the ability to run all our tests on all platforms/browsers. 7) Improve the maven build For example: - ability to run the app on a defined port so that it doesn't interfere with an already running xwiki instance - stop the container when a test fails 8) Finish setting up our CI tool Tests are pretty useless without CI. I'm working on setting up TeamCity. I'm having some issues that I'm debugging with jetbrains (at first glance seems caused by ObjectWeb SVN). Catalin will work on some of these points as part of GSOC (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/ FunctionalTestSuite). We haven't decided yet which points he'll work on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5 being the most important items. WDYT about all these? Thanks -Vincent
Hi, and sorry for the delayed response on this. On 5/31/07, Vincent Massol <[email protected]> wrote:
Hi,
Here's a list Catalin and I came up with while brainstorming about ideas around testing for XWiki. Please feel free to comment, argue, suggest new ideas, etc.
0) Agree that we want to have our functional tests written in Java. Vs using Selenium IDE only
Pros: - DSL -> easy to write and low maintenance (if a UI changes it's fixed in one place)
I also think that writing the functional tests in Java would make them easier to maintain. The tests written in Selenese (the language used by default by Selenium IDE; each Selenese program is actually a HTML table with Nx3 cells) are extremely "fragile". No matter whether they are just recordings or tests carefully written by hand, as soon as some part of the interface changes each of them would need to be changed individually, while with Java the change would occur in only one place.
- ability to write more complex tests with branch control (if, loop, etc)
Branches and loops are actually available for Selenium as an extension (http://wiki.openqa.org/display/SEL/flowControl), and might one day make it into the official package. The way I see it, the main feature which will always be missing in Selenese is abstraction, which makes copy-paste the only way to achieve code reuse.
- ability to write tests easily
I agree, once all the infrastructure is in place (the DSLs for all skins) creating new tests with Java will be "piece-of-cake". Still creating this infrastructure will require a bigger initial overhead ... as opposed to Selenese where we wouldn't (and won't be able to) create such an infrastructure at all. Anyway, I think that this initial investment in testing infrastructure will pay off when writing tests and even more when maintaining them.
(try creating a test that types some content in our Tiny MCE editor and you'll understand that part ;-))
This is actually not so hard to do, once you know how it should be done, but for someone who does not know it it's almost impossible to discover: <tr> <td>type</td> <td>mceSpanFonts</td> <td>Some text to be inserted in the Tiny MCE editor</td> </tr>
- run automatically as part of our build/CI
Is this not also the case for Selenese tests?
Cons: - require a minimal knowledge of Java. Thus it's going to prevent us from having non technical people write functional tests for us.
In my opinion writing the tests in our DSL will be a lot easier than _writing_ them in Selenese, even for someone who does not know Java at all. However, with the Selenium IDE you are also able to _record_ tests, which we definitely won't be able to do with Java. The Selenium IDE can export a Java version of a recorded test, but the code will be a translation of Selenese into Java, so if we commit to using a DSL we shouldn't use such fragile tests. What is also possible Selenium IDE, and is unfortunately not (yet) possible in our Java framework is easily debugging functional tests. With Selenium IDE you can set breakpoints, run a test step-by-step, see tracing information, inspect the html currently rendered by the browser (with firebug for example), for debugging your tests. For the Java tests now you can only run all of them as part of the Maven build and see whether some of them fail or not. If they fail you get a stack trace, which is never sufficient to realize what went wrong. We should aim to do a lot better than this. It should be possible to run and debug the Java functional tests in Eclipse or Idea like usual JUnit tests. The setup and teardown phases will be costly (starting a xwiki server and a browser) and we should log a lot more tracing information, but other than this I don't see why this wouldn't work. We should be able to debug Java tests at least as easily as it is we debug Selenese tests in Selenium IDE. Otherwise we will find ourselves more an more often manually converting Java tests into Selenese just to be able to debug them, and this is unacceptable. One (definitely bad) alternative would be for the Java tests not to access Selenium directly but just generate Selenese tests which are then run separately. This would at least allow one to debug these generated tests in Selenium IDE, but I think we should consider it only if it turns out that what I wrote in the previous paragraph is for some reason not achievable.
I'm currently 100% convinced that our Java choice is right but I know Ludovic isn't fully convinced which is why I'm listing this point here to see what others think.
I would also prefer Java tests, provided that we can also debug them easily.
1) Finish setting up our Java functional test fwk.
Goals: - make it as easy as possible to use. - make it low maintenance (ie tests shouldn't fail often and if some UI is changed it should be very easy to fix the tests)
Tasks: - Continue improving our DSL for testing. Specifically add the ability to test over several skins and create one DSL per skin. - Add DSL methods for asserting the rendered HTML (cf Catalin's tests in http://jira.xwiki.org/jira/browse/XWIKI-1207)
More tasks: - Make it easy to debug the Java functional tests - Add useful tracing and error log messages
2) Add clover support in the build to assert Test Coverage (TC)
We need this to evaluate easily what we are not testing. It should report on both unit and functional tests execution.
Note: This is easy to do with the nice Clover plugin for m2 ;-) (I'm the author of that plugin)
3) Add functional tests to cover 30% of the code under test.
This is a first milestone that we should aim towards. For this we need to have 2) to see where we stand. Our target goal should be in the range of 60-70% TC. For my past experience this is a good ballpark figure and having this results in:
a) well tested code and confidence something new doesn't break what exists. This is very interesting for us in the 1.x time frame as we don't want to add regressions to 1.0. But also it gives us greater confidence for redesigning internal parts of XWiki for implementing the 2.0 architecture b) the ability to release in production at each release (provided new features get tests added when they're added)
4) Explore in-VM testing
The idea is to have several VM (Virtual Machines) running different OSes for example (Windows, linux, Mac) and a combination of Databases (Oracle, MySQL, PostGre). You run the XWiki maven build in them so that the tests get executed in VM. This allows 2 things:
a) test for different environment. For example this tests our start/ stop scripts, our database setups b) this allows more complex tests to be written like: virtual wikis
c) this allows to implement hard to write tests like the "add attachment" one. For this you need the attachment to be on your file system. With a VM you control the full setup of the VM and you can restore the VM at each run
Just wondering: how many tests like this are there? At first sight it seems that not many. What is most important and not on this list is: d) In-VM testing would also allow us to test on more web browsers on more platforms. This is related to 6), maybe another way to achieve the same thing. Unless we mean different things by "in-VM testing" :)
5) create a test plan listing tests to be written
We need this written in JIRA. This goes with 3).
Here the opinion of the xwiki developers and users would be greatly appreciated. Which are the features you think that should receive the most care, or which are the test cases which we should not be leave out? Please enter your opinions here: http://www.xwiki.org/xwiki/bin/view/Idea/FunctionalTestPlanListing After the discussion with Vincent it seems that we should focus on features that are very important for end-users but the developers tend to overlook when testing ... like the WYSIWYG editor. What do you think?
6) Explore using Reality QA
This is a nice tool/service created by Patrick Lightbody (who's a friend). See http://www.realityqa.com/screencasts/realitycheck_tutorial/ RealityCheck.htm to see it in action.
Right now the main issue is that it won't run selenium tests written in Java but Patrick says it's coming: http://community.realityqa.com/thread.jspa?threadID=1010
The other thing that could be nice is the ability to upload our app inside their VM: http://community.realityqa.com/thread/1011 (but we can host our VMs to start with).
What Reality QA buys us is the ability to run all our tests on all platforms/browsers.
7) Improve the maven build
For example: - ability to run the app on a defined port so that it doesn't interfere with an already running xwiki instance - stop the container when a test fails
8) Finish setting up our CI tool
Tests are pretty useless without CI. I'm working on setting up TeamCity. I'm having some issues that I'm debugging with jetbrains (at first glance seems caused by ObjectWeb SVN).
9) Write a tutorial for newcomers on how to create Java selenium tests. 10) Small enhancements to the the XWiki user interface (for example adding IDs to elements) would make it more easily testable and the tests less fragile. Selectors like //div[4]/a/strong for the delete button should never be used in our tests. 11) Make it also possible to run the tests, without Firefox/IE/etc windows popping all over the place. This is important since we want the developers to actually run these tests (and also since we want to integrate them into a CI tool). They already take a long time to run (5 minutes on my machine), but if they would at least let me continue working while they run, I would run them a lot more often. According to this http://wiki.openqa.org/display/SRC/Selenium-RC+and+Continuous+Integration running tests in "headless mode" should be possible at least on UNIX machines.
Catalin will work on some of these points as part of GSOC (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/ FunctionalTestSuite). We haven't decided yet which points he'll work on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5 being the most important items.
I agree that 1 (including debugging support), 3 and 5 are the points where I could help the most, and probably also 9 and 10. Time and knowledge permitting I could also try to help with 4 and others. Regards, Catalin
On Jun 8, 2007, at 11:33 AM, Catalin Hritcu wrote: [snip]
(try creating a test that types some content in our Tiny MCE editor and you'll understand that part ;-))
This is actually not so hard to do, once you know how it should be done, but for someone who does not know it it's almost impossible to discover: <tr> <td>type</td> <td>mceSpanFonts</td> <td>Some text to be inserted in the Tiny MCE editor</td> </tr>
ok, now try to clear the content of a page being editer in WYSIWYG mode ;-)
- run automatically as part of our build/CI
Is this not also the case for Selenese tests?
No the maven plugin we're using works with Selenium RC. There's a mavenium plugin for HTML tests though but I haven't tried it. [snip]
In my opinion writing the tests in our DSL will be a lot easier than _writing_ them in Selenese, even for someone who does not know Java at all. However, with the Selenium IDE you are also able to _record_ tests, which we definitely won't be able to do with Java.
I'm now finding myself writing more and more tests directly in java without going through the Selenium IDE. I think it's a matter of having the DSL in place mostly. And we already have a good start. [snip]
What is also possible Selenium IDE, and is unfortunately not (yet) possible in our Java framework is easily debugging functional tests. With Selenium IDE you can set breakpoints, run a test step-by-step, see tracing information, inspect the html currently rendered by the browser (with firebug for example), for debugging your tests. For the Java tests now you can only run all of them as part of the Maven build and see whether some of them fail or not. If they fail you get a stack trace, which is never sufficient to realize what went wrong.
We should aim to do a lot better than this. It should be possible to run and debug the Java functional tests in Eclipse or Idea like usual JUnit tests. The setup and teardown phases will be costly (starting a xwiki server and a browser) and we should log a lot more tracing information, but other than this I don't see why this wouldn't work. We should be able to debug Java tests at least as easily as it is we debug Selenese tests in Selenium IDE. Otherwise we will find ourselves more an more often manually converting Java tests into Selenese just to be able to debug them, and this is unacceptable.
One (definitely bad) alternative would be for the Java tests not to access Selenium directly but just generate Selenese tests which are then run separately. This would at least allow one to debug these generated tests in Selenium IDE, but I think we should consider it only if it turns out that what I wrote in the previous paragraph is for some reason not achievable.
We can do debugging! I'm doing it. Here's the current way of doing it (we should improve it): * go in application-test/, edit the pom.xml to replace <background>true</background> with false. Note: We need to find a way to pass this value on the maven command line. This is possible, I just haven't really looked into that. * start the xwiki version you wish to test * go in your IDE, find the test you wish to run, put breakpoints and run it in debug mode. Not only you'll get the Java debugging power but you can also see the browser in action, verify stuff in the browser, etc. It works real well.
I'm currently 100% convinced that our Java choice is right but I know Ludovic isn't fully convinced which is why I'm listing this point here to see what others think.
I would also prefer Java tests, provided that we can also debug them easily.
Your wish has come true! :)
1) Finish setting up our Java functional test fwk.
Goals: - make it as easy as possible to use. - make it low maintenance (ie tests shouldn't fail often and if some UI is changed it should be very easy to fix the tests)
Tasks: - Continue improving our DSL for testing. Specifically add the ability to test over several skins and create one DSL per skin. - Add DSL methods for asserting the rendered HTML (cf Catalin's tests in http://jira.xwiki.org/jira/browse/XWIKI-1207)
More tasks: - Make it easy to debug the Java functional tests
done
- Add useful tracing and error log messages
Not sure what this is about. There's already a debug feature of the selenium plugin we're using. What do you want exactly? [snip]
4) Explore in-VM testing
The idea is to have several VM (Virtual Machines) running different OSes for example (Windows, linux, Mac) and a combination of Databases (Oracle, MySQL, PostGre). You run the XWiki maven build in them so that the tests get executed in VM. This allows 2 things:
a) test for different environment. For example this tests our start/ stop scripts, our database setups b) this allows more complex tests to be written like: virtual wikis
c) this allows to implement hard to write tests like the "add attachment" one. For this you need the attachment to be on your file system. With a VM you control the full setup of the VM and you can restore the VM at each run
Just wondering: how many tests like this are there? At first sight it seems that not many.
A lot... The problem is that they are the most important... For example ensuring that our build runs on different DB, OSes, etc is quite crucial.
What is most important and not on this list is: d) In-VM testing would also allow us to test on more web browsers on more platforms.
Yes . However we don't need VM for this. This is provided by OpenQA as of now. See below.
This is related to 6), maybe another way to achieve the same thing. Unless we mean different things by "in-VM testing" :)
5) create a test plan listing tests to be written
We need this written in JIRA. This goes with 3).
Here the opinion of the xwiki developers and users would be greatly appreciated. Which are the features you think that should receive the most care, or which are the test cases which we should not be leave out? Please enter your opinions here: http://www.xwiki.org/xwiki/bin/view/Idea/FunctionalTestPlanListing
I think you need to provide a first draft with more details and then based on this the community can react.
After the discussion with Vincent it seems that we should focus on features that are very important for end-users but the developers tend to overlook when testing ... like the WYSIWYG editor. What do you think?
Sure.
6) Explore using Reality QA
This is a nice tool/service created by Patrick Lightbody (who's a friend). See http://www.realityqa.com/screencasts/realitycheck_tutorial/ RealityCheck.htm to see it in action.
Right now the main issue is that it won't run selenium tests written in Java but Patrick says it's coming: http://community.realityqa.com/thread.jspa?threadID=1010
The other thing that could be nice is the ability to upload our app inside their VM: http://community.realityqa.com/thread/1011 (but we can host our VMs to start with).
What Reality QA buys us is the ability to run all our tests on all platforms/browsers.
Patrick just told me he has implemented executing Java tests inside openQA so that's really cool. I'm going to try this real soon. See http://community.realityqa.com/thread.jspa?threadID=1010 for updates.
7) Improve the maven build
For example: - ability to run the app on a defined port so that it doesn't interfere with an already running xwiki instance - stop the container when a test fails
8) Finish setting up our CI tool
Tests are pretty useless without CI. I'm working on setting up TeamCity. I'm having some issues that I'm debugging with jetbrains (at first glance seems caused by ObjectWeb SVN).
9) Write a tutorial for newcomers on how to create Java selenium tests.
Yes. I have started here: http://www.xwiki.org/xwiki/bin/view/Community/Testing We need to continue improving that page. Help is most welcome.
10) Small enhancements to the the XWiki user interface (for example adding IDs to elements) would make it more easily testable and the tests less fragile. Selectors like //div[4]/a/strong for the delete button should never be used in our tests.
Right. BTW I'm using "//div[@id='tmDelete']/a" for the delete button which isn't that bad. Maybe you're talking about another delete button. I think this should be fixed as we write the tests.
11) Make it also possible to run the tests, without Firefox/IE/etc windows popping all over the place. This is important since we want the developers to actually run these tests (and also since we want to integrate them into a CI tool).
BTW we get this if we use openQA.
They already take a long time to run (5 minutes on my machine), but if they would at least let me continue working while they run, I would run them a lot more often. According to this http://wiki.openqa.org/display/SRC/Selenium-RC+and +Continuous+Integration running tests in "headless mode" should be possible at least on UNIX machines.
Yes that would be nice to investigate as we'll need this for our CI.
Catalin will work on some of these points as part of GSOC (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/ FunctionalTestSuite). We haven't decided yet which points he'll work on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5 being the most important items.
I agree that 1 (including debugging support), 3 and 5 are the points where I could help the most, and probably also 9 and 10. Time and knowledge permitting I could also try to help with 4 and others.
Great stuff. Let us know your planning so that we set some milestones and progress in a directed manner rather than in a best-effort one. I can help too. Thanks Catalin! -Vincent
Hi, On 6/8/07, Vincent Massol <[email protected]> wrote:
On Jun 8, 2007, at 11:33 AM, Catalin Hritcu wrote:
[snip]
(try creating a test that types some content in our Tiny MCE editor and you'll understand that part ;-))
This is actually not so hard to do, once you know how it should be done, but for someone who does not know it it's almost impossible to discover: <tr> <td>type</td> <td>mceSpanFonts</td> <td>Some text to be inserted in the Tiny MCE editor</td> </tr>
ok, now try to clear the content of a page being editer in WYSIWYG mode ;-)
I'm not trying to annoy you, but this is the same :) <tr> <td>type</td> <td>mceSpanFonts</td> <td></td> </tr> [snip]
We can do debugging! I'm doing it. Here's the current way of doing it (we should improve it):
* go in application-test/, edit the pom.xml to replace <background>true</background> with false. Note: We need to find a way to pass this value on the maven command line. This is possible, I just haven't really looked into that. * start the xwiki version you wish to test * go in your IDE, find the test you wish to run, put breakpoints and run it in debug mode. Not only you'll get the Java debugging power but you can also see the browser in action, verify stuff in the browser, etc. It works real well.
Sorry but I was not aware of this already working. I tried debugging a Java test like this and it works smoothly. Thank you. [snip]
- Add useful tracing and error log messages
Not sure what this is about. There's already a debug feature of the selenium plugin we're using. What do you want exactly?
When a test fails I want to be able to get more information than just a stack trace, something like the Log tab at the bottom of Selenium IDE: # [info] Executing: |open | /mail/?ik=6d3a96df6c ... | | # [info] Executing: |selectFrame | main | | # [info] Executing: |selectFrame | v1 | | # [info] Executing: |type | ta_compose | Hi, On 6/8/07, Vincent Massol ... | | [This is me replaying the recording I made when writing this email] Didn't know about it until now, but one can start maven with the "-Ddebug=true" switch and then all messages from the selenium server are saved to: application-test/target/selenium/server.log I wonder if there is an easy way to have only level info and higher logged (because with "-Ddebug=true" the useful messages are borrowed under lots of debug messages)? [snip]
What is most important and not on this list is: d) In-VM testing would also allow us to test on more web browsers on more platforms.
Yes . However we don't need VM for this. This is provided by OpenQA as of now. See below.
I wonder, with OpenQA will be also able to directly debug the tests on different browsers, or only run them as a batch on their (virtual) machines ? [snip]
5) create a test plan listing tests to be written
We need this written in JIRA. This goes with 3).
Here the opinion of the xwiki developers and users would be greatly appreciated. Which are the features you think that should receive the most care, or which are the test cases which we should not be leave out? Please enter your opinions here: http://www.xwiki.org/xwiki/bin/view/Idea/FunctionalTestPlanListing
I think you need to provide a first draft with more details and then based on this the community can react.
One has to start somewhere :) . I'll add more ideas to the page as they come, and when it's in better shape announce it in a separate thread on xwiki-users. [snip]
Patrick just told me he has implemented executing Java tests inside openQA so that's really cool. I'm going to try this real soon. See http://community.realityqa.com/thread.jspa?threadID=1010 for updates.
Nice :) [snip]
10) Small enhancements to the the XWiki user interface (for example adding IDs to elements) would make it more easily testable and the tests less fragile. Selectors like //div[4]/a/strong for the delete button should never be used in our tests.
Right.
BTW I'm using "//div[@id='tmDelete']/a" for the delete button which isn't that bad. Maybe you're talking about another delete button.
If you record deleting a page using selenium ide you get "//div[4]/a/strong". [snip]
Catalin will work on some of these points as part of GSOC (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/ FunctionalTestSuite). We haven't decided yet which points he'll work on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5 being the most important items.
I agree that 1 (including debugging support), 3 and 5 are the points where I could help the most, and probably also 9 and 10. Time and knowledge permitting I could also try to help with 4 and others.
Great stuff.
Let us know your planning so that we set some milestones and progress in a directed manner rather than in a best-effort one.
I will. Still for a couple of weeks it will still be best-effort as I am overwhelmed with other work.
I can help too.
Thank you. Regards, Catalin
On 6/8/07, Catalin Hritcu <[email protected]> wrote:
10) Small enhancements to the the XWiki user interface (for example adding IDs to elements) would make it more easily testable and the tests less fragile. Selectors like //div[4]/a/strong for the delete button should never be used in our tests.
I try to minimize the use of id-s, as they have little semantics. Instead, we should use 'role', 'class' and 'name', and only have an id for elements that we are sure they will always be unique. For example, the cancel/preview/save buttons aren't unique, as in some editors/skins they are both before and after the editable content. But I also agree that stuff like div[4]//li[5]/span/a is not a good way of working. Complain about this in a JIRA issue and link it to the skin cleanup task. Sergiu -- http://purl.org/net/sergiu
participants (3)
-
Catalin Hritcu -
Sergiu Dumitriu -
Vincent Massol