Hi, and sorry for the delayed response on this. On 5/31/07, Vincent Massol <[email protected]> wrote:
Hi,
Here's a list Catalin and I came up with while brainstorming about ideas around testing for XWiki. Please feel free to comment, argue, suggest new ideas, etc.
0) Agree that we want to have our functional tests written in Java. Vs using Selenium IDE only
Pros: - DSL -> easy to write and low maintenance (if a UI changes it's fixed in one place)
I also think that writing the functional tests in Java would make them easier to maintain. The tests written in Selenese (the language used by default by Selenium IDE; each Selenese program is actually a HTML table with Nx3 cells) are extremely "fragile". No matter whether they are just recordings or tests carefully written by hand, as soon as some part of the interface changes each of them would need to be changed individually, while with Java the change would occur in only one place.
- ability to write more complex tests with branch control (if, loop, etc)
Branches and loops are actually available for Selenium as an extension (http://wiki.openqa.org/display/SEL/flowControl), and might one day make it into the official package. The way I see it, the main feature which will always be missing in Selenese is abstraction, which makes copy-paste the only way to achieve code reuse.
- ability to write tests easily
I agree, once all the infrastructure is in place (the DSLs for all skins) creating new tests with Java will be "piece-of-cake". Still creating this infrastructure will require a bigger initial overhead ... as opposed to Selenese where we wouldn't (and won't be able to) create such an infrastructure at all. Anyway, I think that this initial investment in testing infrastructure will pay off when writing tests and even more when maintaining them.
(try creating a test that types some content in our Tiny MCE editor and you'll understand that part ;-))
This is actually not so hard to do, once you know how it should be done, but for someone who does not know it it's almost impossible to discover: <tr> <td>type</td> <td>mceSpanFonts</td> <td>Some text to be inserted in the Tiny MCE editor</td> </tr>
- run automatically as part of our build/CI
Is this not also the case for Selenese tests?
Cons: - require a minimal knowledge of Java. Thus it's going to prevent us from having non technical people write functional tests for us.
In my opinion writing the tests in our DSL will be a lot easier than _writing_ them in Selenese, even for someone who does not know Java at all. However, with the Selenium IDE you are also able to _record_ tests, which we definitely won't be able to do with Java. The Selenium IDE can export a Java version of a recorded test, but the code will be a translation of Selenese into Java, so if we commit to using a DSL we shouldn't use such fragile tests. What is also possible Selenium IDE, and is unfortunately not (yet) possible in our Java framework is easily debugging functional tests. With Selenium IDE you can set breakpoints, run a test step-by-step, see tracing information, inspect the html currently rendered by the browser (with firebug for example), for debugging your tests. For the Java tests now you can only run all of them as part of the Maven build and see whether some of them fail or not. If they fail you get a stack trace, which is never sufficient to realize what went wrong. We should aim to do a lot better than this. It should be possible to run and debug the Java functional tests in Eclipse or Idea like usual JUnit tests. The setup and teardown phases will be costly (starting a xwiki server and a browser) and we should log a lot more tracing information, but other than this I don't see why this wouldn't work. We should be able to debug Java tests at least as easily as it is we debug Selenese tests in Selenium IDE. Otherwise we will find ourselves more an more often manually converting Java tests into Selenese just to be able to debug them, and this is unacceptable. One (definitely bad) alternative would be for the Java tests not to access Selenium directly but just generate Selenese tests which are then run separately. This would at least allow one to debug these generated tests in Selenium IDE, but I think we should consider it only if it turns out that what I wrote in the previous paragraph is for some reason not achievable.
I'm currently 100% convinced that our Java choice is right but I know Ludovic isn't fully convinced which is why I'm listing this point here to see what others think.
I would also prefer Java tests, provided that we can also debug them easily.
1) Finish setting up our Java functional test fwk.
Goals: - make it as easy as possible to use. - make it low maintenance (ie tests shouldn't fail often and if some UI is changed it should be very easy to fix the tests)
Tasks: - Continue improving our DSL for testing. Specifically add the ability to test over several skins and create one DSL per skin. - Add DSL methods for asserting the rendered HTML (cf Catalin's tests in http://jira.xwiki.org/jira/browse/XWIKI-1207)
More tasks: - Make it easy to debug the Java functional tests - Add useful tracing and error log messages
2) Add clover support in the build to assert Test Coverage (TC)
We need this to evaluate easily what we are not testing. It should report on both unit and functional tests execution.
Note: This is easy to do with the nice Clover plugin for m2 ;-) (I'm the author of that plugin)
3) Add functional tests to cover 30% of the code under test.
This is a first milestone that we should aim towards. For this we need to have 2) to see where we stand. Our target goal should be in the range of 60-70% TC. For my past experience this is a good ballpark figure and having this results in:
a) well tested code and confidence something new doesn't break what exists. This is very interesting for us in the 1.x time frame as we don't want to add regressions to 1.0. But also it gives us greater confidence for redesigning internal parts of XWiki for implementing the 2.0 architecture b) the ability to release in production at each release (provided new features get tests added when they're added)
4) Explore in-VM testing
The idea is to have several VM (Virtual Machines) running different OSes for example (Windows, linux, Mac) and a combination of Databases (Oracle, MySQL, PostGre). You run the XWiki maven build in them so that the tests get executed in VM. This allows 2 things:
a) test for different environment. For example this tests our start/ stop scripts, our database setups b) this allows more complex tests to be written like: virtual wikis
c) this allows to implement hard to write tests like the "add attachment" one. For this you need the attachment to be on your file system. With a VM you control the full setup of the VM and you can restore the VM at each run
Just wondering: how many tests like this are there? At first sight it seems that not many. What is most important and not on this list is: d) In-VM testing would also allow us to test on more web browsers on more platforms. This is related to 6), maybe another way to achieve the same thing. Unless we mean different things by "in-VM testing" :)
5) create a test plan listing tests to be written
We need this written in JIRA. This goes with 3).
Here the opinion of the xwiki developers and users would be greatly appreciated. Which are the features you think that should receive the most care, or which are the test cases which we should not be leave out? Please enter your opinions here: http://www.xwiki.org/xwiki/bin/view/Idea/FunctionalTestPlanListing After the discussion with Vincent it seems that we should focus on features that are very important for end-users but the developers tend to overlook when testing ... like the WYSIWYG editor. What do you think?
6) Explore using Reality QA
This is a nice tool/service created by Patrick Lightbody (who's a friend). See http://www.realityqa.com/screencasts/realitycheck_tutorial/ RealityCheck.htm to see it in action.
Right now the main issue is that it won't run selenium tests written in Java but Patrick says it's coming: http://community.realityqa.com/thread.jspa?threadID=1010
The other thing that could be nice is the ability to upload our app inside their VM: http://community.realityqa.com/thread/1011 (but we can host our VMs to start with).
What Reality QA buys us is the ability to run all our tests on all platforms/browsers.
7) Improve the maven build
For example: - ability to run the app on a defined port so that it doesn't interfere with an already running xwiki instance - stop the container when a test fails
8) Finish setting up our CI tool
Tests are pretty useless without CI. I'm working on setting up TeamCity. I'm having some issues that I'm debugging with jetbrains (at first glance seems caused by ObjectWeb SVN).
9) Write a tutorial for newcomers on how to create Java selenium tests. 10) Small enhancements to the the XWiki user interface (for example adding IDs to elements) would make it more easily testable and the tests less fragile. Selectors like //div[4]/a/strong for the delete button should never be used in our tests. 11) Make it also possible to run the tests, without Firefox/IE/etc windows popping all over the place. This is important since we want the developers to actually run these tests (and also since we want to integrate them into a CI tool). They already take a long time to run (5 minutes on my machine), but if they would at least let me continue working while they run, I would run them a lot more often. According to this http://wiki.openqa.org/display/SRC/Selenium-RC+and+Continuous+Integration running tests in "headless mode" should be possible at least on UNIX machines.
Catalin will work on some of these points as part of GSOC (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/ FunctionalTestSuite). We haven't decided yet which points he'll work on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5 being the most important items.
I agree that 1 (including debugging support), 3 and 5 are the points where I could help the most, and probably also 9 and 10. Time and knowledge permitting I could also try to help with 4 and others. Regards, Catalin