8 Jun
2007
8 Jun
'07
11:33 a.m.
Hi,
and sorry for the delayed response on this.
On 5/31/07, Vincent Massol <vincent(a)massol.net> wrote:
Hi,
Here's a list Catalin and I came up with while brainstorming about
ideas around testing for XWiki. Please feel free to comment, argue,
suggest new ideas, etc.
0) Agree that we want to have our functional tests written in Java.
Vs using Selenium IDE only
Pros:
- DSL -> easy to write and low maintenance (if a UI changes it's
fixed in one place)
I also think that writing the functional tests in Java would make them
easier to maintain. The tests written in Selenese (the language used
by default by Selenium IDE; each Selenese program is actually a HTML
table with Nx3 cells) are extremely "fragile". No matter whether they
are just recordings or tests carefully written by hand, as soon as
some part of the interface changes each of them would need to be
changed individually, while with Java the change would occur in only
one place.
- ability to write more complex tests with branch
control (if, loop,
etc)
Branches and loops are actually available for Selenium as an extension
(http://wiki.openqa.org/display/SEL/flowControl), and might one day
make it into the official package. The way I see it, the main feature
which will always be missing in Selenese is abstraction, which makes
copy-paste the only way to achieve code reuse.
- ability to write tests easily
I agree, once all the infrastructure is in place (the DSLs for all
skins) creating new tests with Java will be "piece-of-cake". Still
creating this infrastructure will require a bigger initial overhead
... as opposed to Selenese where we wouldn't (and won't be able to)
create such an infrastructure at all. Anyway, I think that this
initial investment in testing infrastructure will pay off when writing
tests and even more when maintaining them.
(try creating a test that types some
content in our Tiny MCE editor and you'll understand that part ;-))
This is actually not so hard to do, once you know how it should be
done, but for someone who does not know it it's almost impossible to
discover:
<tr>
<td>type</td>
<td>mceSpanFonts</td>
<td>Some text to be inserted in the Tiny MCE editor</td>
</tr>
- run automatically as part of our build/CI
Is this not also the case for Selenese tests?
Cons:
- require a minimal knowledge of Java. Thus it's going to prevent us
from having non technical people write functional tests for us.
In my opinion writing the tests in our DSL will be a lot easier than
_writing_ them in Selenese, even for someone who does not know Java at
all. However, with the Selenium IDE you are also able to _record_
tests, which we definitely won't be able to do with Java. The Selenium
IDE can export a Java version of a recorded test, but the code will be
a translation of Selenese into Java, so if we commit to using a DSL we
shouldn't use such fragile tests.
What is also possible Selenium IDE, and is unfortunately not (yet)
possible in our Java framework is easily debugging functional tests.
With Selenium IDE you can set breakpoints, run a test step-by-step,
see tracing information, inspect the html currently rendered by the
browser (with firebug for example), for debugging your tests. For the
Java tests now you can only run all of them as part of the Maven build
and see whether some of them fail or not. If they fail you get a stack
trace, which is never sufficient to realize what went wrong.
We should aim to do a lot better than this. It should be possible to
run and debug the Java functional tests in Eclipse or Idea like usual
JUnit tests. The setup and teardown phases will be costly (starting a
xwiki server and a browser) and we should log a lot more tracing
information, but other than this I don't see why this wouldn't work.
We should be able to debug Java tests at least as easily as it is we
debug Selenese tests in Selenium IDE. Otherwise we will find ourselves
more an more often manually converting Java tests into Selenese just
to be able to debug them, and this is unacceptable.
One (definitely bad) alternative would be for the Java tests not to
access Selenium directly but just generate Selenese tests which are
then run separately. This would at least allow one to debug these
generated tests in Selenium IDE, but I think we should consider it
only if it turns out that what I wrote in the previous paragraph is
for some reason not achievable.
I'm currently 100% convinced that our Java choice
is right but I know
Ludovic isn't fully convinced which is why I'm listing this point
here to see what others think.
I would also prefer Java tests, provided that we can also debug them easily.
1) Finish setting up our Java functional test fwk.
Goals:
- make it as easy as possible to use.
- make it low maintenance (ie tests shouldn't fail often and if some
UI is changed it should be very easy to fix the tests)
Tasks:
- Continue improving our DSL for testing. Specifically add the
ability to test over several skins and create one DSL per skin.
- Add DSL methods for asserting the rendered HTML (cf Catalin's tests
in http://jira.xwiki.org/jira/browse/XWIKI-1207)
More tasks:
- Make it easy to debug the Java functional tests
- Add useful tracing and error log messages
2) Add clover support in the build to assert Test
Coverage (TC)
We need this to evaluate easily what we are not testing. It should
report on both unit and functional tests execution.
Note: This is easy to do with the nice Clover plugin for m2 ;-) (I'm
the author of that plugin)
3) Add functional tests to cover 30% of the code under test.
This is a first milestone that we should aim towards. For this we
need to have 2) to see where we stand. Our target goal should be in
the range of 60-70% TC. For my past experience this is a good
ballpark figure and having this results in:
a) well tested code and confidence something new doesn't break what
exists. This is very interesting for us in the 1.x time frame as we
don't want to add regressions to 1.0. But also it gives us greater
confidence for redesigning internal parts of XWiki for implementing
the 2.0 architecture
b) the ability to release in production at each release (provided new
features get tests added when they're added)
4) Explore in-VM testing
The idea is to have several VM (Virtual Machines) running different
OSes for example (Windows, linux, Mac) and a combination of Databases
(Oracle, MySQL, PostGre). You run the XWiki maven build in them so
that the tests get executed in VM. This allows 2 things:
a) test for different environment. For example this tests our start/
stop scripts, our database setups
b) this allows more complex tests to be written like: virtual wikis
c) this allows to implement hard to write tests like
the "add
attachment" one. For this you need the attachment to be on your file
system. With a VM you control the full setup of the VM and you can
restore the VM at each run
Just wondering: how many tests like this are there? At first sight it
seems that not many.
What is most important and not on this list is:
d) In-VM testing would also allow us to test on more web browsers on
more platforms.
This is related to 6), maybe another way to achieve the same thing.
Unless we mean different things by "in-VM testing" :)
5) create a test plan listing tests to be written
We need this written in JIRA. This goes with 3).
Here the opinion of the xwiki developers and users would be greatly
appreciated. Which are the features you think that should receive the
most care, or which are the test cases which we should not be leave
out? Please enter your opinions here:
http://www.xwiki.org/xwiki/bin/view/Idea/FunctionalTestPlanListing
After the discussion with Vincent it seems that we should focus on
features that are very important for end-users but the developers tend
to overlook when testing ... like the WYSIWYG editor. What do you
think?
6) Explore using Reality QA
This is a nice tool/service created by Patrick Lightbody (who's a
friend).
See http://www.realityqa.com/screencasts/realitycheck_tutorial/
RealityCheck.htm to see it in action.
Right now the main issue is that it won't run selenium tests written
in Java but Patrick says it's coming:
http://community.realityqa.com/thread.jspa?threadID=1010
The other thing that could be nice is the ability to upload our app
inside their VM:
http://community.realityqa.com/thread/1011 (but we can host our VMs
to start with).
What Reality QA buys us is the ability to run all our tests on all
platforms/browsers.
7) Improve the maven build
For example:
- ability to run the app on a defined port so that it doesn't
interfere with an already running xwiki instance
- stop the container when a test fails
8) Finish setting up our CI tool
Tests are pretty useless without CI. I'm working on setting up
TeamCity. I'm having some issues that I'm debugging with jetbrains
(at first glance seems caused by ObjectWeb SVN).
9) Write a tutorial for newcomers on how to create Java selenium tests.
10) Small enhancements to the the XWiki user interface (for example
adding IDs to elements) would make it more easily testable and the
tests less fragile. Selectors like //div[4]/a/strong for the delete
button should never be used in our tests.
11) Make it also possible to run the tests, without Firefox/IE/etc
windows popping all over the place. This is important since we want
the developers to actually run these tests (and also since we want to
integrate them into a CI tool). They already take a long time to run
(5 minutes on my machine), but if they would at least let me continue
working while they run, I would run them a lot more often. According
to this http://wiki.openqa.org/display/SRC/Selenium-RC+and+Continuous+Integration
running tests in "headless mode" should be possible at least on UNIX
machines.
Catalin will work on some of these points as part of GSOC
(http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/
FunctionalTestSuite). We haven't decided yet which points he'll work
on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5
being the most important items.
I agree that 1 (including debugging support), 3 and 5 are the points
where I could help the most, and probably also 9 and 10. Time and
knowledge permitting I could also try to help with 4 and others.
Regards,
Catalin