Testing Ideas for XWiki

Hi, Here's a list Catalin and I came up with while brainstorming about ideas around testing for XWiki. Please feel free to comment, argue, suggest new ideas, etc. 0) Agree that we want to have our functional tests written in Java. Vs using Selenium IDE only Pros: - DSL -> easy to write and low maintenance (if a UI changes it's fixed in one place) - ability to write more complex tests with branch control (if, loop, etc) - ability to write tests easily (try creating a test that types some content in our Tiny MCE editor and you'll understand that part ;-)) - run automatically as part of our build/CI Cons: - require a minimal knowledge of Java. Thus it's going to prevent us from having non technical people write functional tests for us. I'm currently 100% convinced that our Java choice is right but I know Ludovic isn't fully convinced which is why I'm listing this point here to see what others think. 1) Finish setting up our Java functional test fwk. Goals: - make it as easy as possible to use. - make it low maintenance (ie tests shouldn't fail often and if some UI is changed it should be very easy to fix the tests) Tasks: - Continue improving our DSL for testing. Specifically add the ability to test over several skins and create one DSL per skin. - Add DSL methods for asserting the rendered HTML (cf Catalin's tests in http://jira.xwiki.org/jira/browse/XWIKI-1207) 2) Add clover support in the build to assert Test Coverage (TC) We need this to evaluate easily what we are not testing. It should report on both unit and functional tests execution. Note: This is easy to do with the nice Clover plugin for m2 ;-) (I'm the author of that plugin) 3) Add functional tests to cover 30% of the code under test. This is a first milestone that we should aim towards. For this we need to have 2) to see where we stand. Our target goal should be in the range of 60-70% TC. For my past experience this is a good ballpark figure and having this results in: a) well tested code and confidence something new doesn't break what exists. This is very interesting for us in the 1.x time frame as we don't want to add regressions to 1.0. But also it gives us greater confidence for redesigning internal parts of XWiki for implementing the 2.0 architecture b) the ability to release in production at each release (provided new features get tests added when they're added) 4) Explore in-VM testing The idea is to have several VM (Virtual Machines) running different OSes for example (Windows, linux, Mac) and a combination of Databases (Oracle, MySQL, PostGre). You run the XWiki maven build in them so that the tests get executed in VM. This allows 2 things: a) test for different environment. For example this tests our start/ stop scripts, our database setups b) this allows more complex tests to be written like: virtual wikis c) this allows to implement hard to write tests like the "add attachment" one. For this you need the attachment to be on your file system. With a VM you control the full setup of the VM and you can restore the VM at each run 5) create a test plan listing tests to be written We need this written in JIRA. This goes with 3). 6) Explore using Reality QA This is a nice tool/service created by Patrick Lightbody (who's a friend). See http://www.realityqa.com/screencasts/realitycheck_tutorial/ RealityCheck.htm to see it in action. Right now the main issue is that it won't run selenium tests written in Java but Patrick says it's coming: http://community.realityqa.com/thread.jspa?threadID=1010 The other thing that could be nice is the ability to upload our app inside their VM: http://community.realityqa.com/thread/1011 (but we can host our VMs to start with). What Reality QA buys us is the ability to run all our tests on all platforms/browsers. 7) Improve the maven build For example: - ability to run the app on a defined port so that it doesn't interfere with an already running xwiki instance - stop the container when a test fails 8) Finish setting up our CI tool Tests are pretty useless without CI. I'm working on setting up TeamCity. I'm having some issues that I'm debugging with jetbrains (at first glance seems caused by ObjectWeb SVN). Catalin will work on some of these points as part of GSOC (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/ FunctionalTestSuite). We haven't decided yet which points he'll work on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5 being the most important items. WDYT about all these? Thanks -Vincent