8 Jun
2007
8 Jun
'07
12:06 p.m.
On Jun 8, 2007, at 11:33 AM, Catalin Hritcu wrote:
[snip]
ok, now try to clear the content of a page being editer in WYSIWYG
mode ;-)
No the maven plugin we're using works with Selenium RC. There's a
mavenium plugin for HTML tests though but I haven't tried it.
[snip]
Your wish has come true! :)
done
A lot... The problem is that they are the most important... For
example ensuring that our build runs on different DB, OSes, etc is
quite crucial.
Yes. I have started here:
http://www.xwiki.org/xwiki/bin/view/Community/Testing
We need to continue improving that page. Help is most welcome.
Great stuff.
Let us know your planning so that we set some milestones and progress
in a directed manner rather than in a best-effort one.
I can help too.
Thanks Catalin!
-Vincent
(try creating
a test that types some
content in our Tiny MCE editor and you'll understand that part ;-))
This is actually not so hard to do, once you know how it should be
done, but for someone who does not know it it's almost impossible to
discover:
<tr>
<td>type</td>
<td>mceSpanFonts</td>
<td>Some text to be inserted in the Tiny MCE editor</td>
</tr> - run
automatically as part of our build/CI
Is this not also the case for Selenese tests? In my opinion writing the tests in our DSL will be a
lot easier than
_writing_ them in Selenese, even for someone who does not know Java at
all. However, with the Selenium IDE you are also able to _record_
tests, which we definitely won't be able to do with Java.
I'm now finding myself writing more and more tests directly in java
without going through the Selenium IDE. I think it's a matter of
having the DSL in place mostly. And we already have a good start.
[snip]
What is also possible Selenium IDE, and is
unfortunately not (yet)
possible in our Java framework is easily debugging functional tests.
With Selenium IDE you can set breakpoints, run a test step-by-step,
see tracing information, inspect the html currently rendered by the
browser (with firebug for example), for debugging your tests. For the
Java tests now you can only run all of them as part of the Maven build
and see whether some of them fail or not. If they fail you get a stack
trace, which is never sufficient to realize what went wrong.
We should aim to do a lot better than this. It should be possible to
run and debug the Java functional tests in Eclipse or Idea like usual
JUnit tests. The setup and teardown phases will be costly (starting a
xwiki server and a browser) and we should log a lot more tracing
information, but other than this I don't see why this wouldn't work.
We should be able to debug Java tests at least as easily as it is we
debug Selenese tests in Selenium IDE. Otherwise we will find ourselves
more an more often manually converting Java tests into Selenese just
to be able to debug them, and this is unacceptable.
One (definitely bad) alternative would be for the Java tests not to
access Selenium directly but just generate Selenese tests which are
then run separately. This would at least allow one to debug these
generated tests in Selenium IDE, but I think we should consider it
only if it turns out that what I wrote in the previous paragraph is
for some reason not achievable.
We can do debugging! I'm doing it. Here's the current way of doing it
(we should improve it):
* go in application-test/, edit the pom.xml to replace
<background>true</background> with false. Note: We need to find a way
to pass this value on the maven command line. This is possible, I
just haven't really looked into that.
* start the xwiki version you wish to test
* go in your IDE, find the test you wish to run, put breakpoints and
run it in debug mode. Not only you'll get the Java debugging power
but you can also see the browser in action, verify stuff in the
browser, etc. It works real well.
I'm
currently 100% convinced that our Java choice is right but I know
Ludovic isn't fully convinced which is why I'm listing this point
here to see what others think.
I would also prefer Java tests, provided that we can also debug
them easily. 1) Finish
setting up our Java functional test fwk.
Goals:
- make it as easy as possible to use.
- make it low maintenance (ie tests shouldn't fail often and if some
UI is changed it should be very easy to fix the tests)
Tasks:
- Continue improving our DSL for testing. Specifically add the
ability to test over several skins and create one DSL per skin.
- Add DSL methods for asserting the rendered HTML (cf Catalin's tests
in http://jira.xwiki.org/jira/browse/XWIKI-1207)
More tasks:
- Make it easy to debug the Java functional tests - Add useful tracing and error log messages
Not sure what this is about. There's already a debug feature of the
selenium plugin we're using. What do you want exactly?
[snip]
4) Explore
in-VM testing
The idea is to have several VM (Virtual Machines) running different
OSes for example (Windows, linux, Mac) and a combination of Databases
(Oracle, MySQL, PostGre). You run the XWiki maven build in them so
that the tests get executed in VM. This allows 2 things:
a) test for different environment. For example this tests our start/
stop scripts, our database setups
b) this allows more complex tests to be written like: virtual wikis
c) this allows to implement hard to write tests
like the "add
attachment" one. For this you need the attachment to be on your file
system. With a VM you control the full setup of the VM and you can
restore the VM at each run
Just wondering: how many tests like this are there? At first sight it
seems that not many. What is most important and not on this list is:
d) In-VM testing would also allow us to test on more web browsers on
more platforms.
Yes . However we don't need VM for this. This is provided by OpenQA
as of now. See below.
This is related to 6), maybe another way to achieve
the same thing.
Unless we mean different things by "in-VM testing" :)
I think you need to provide a first draft with more details and then
based on this the community can react.
5) create a test plan listing tests to be
written
We need this written in JIRA. This goes with 3).
Here the opinion of the xwiki developers and users would be greatly
appreciated. Which are the features you think that should receive the
most care, or which are the test cases which we should not be leave
out? Please enter your opinions here:
http://www.xwiki.org/xwiki/bin/view/Idea/FunctionalTestPlanListing After the discussion with Vincent it seems that we
should focus on
features that are very important for end-users but the developers tend
to overlook when testing ... like the WYSIWYG editor. What do you
think?
Sure.
> 6) Explore using Reality QA
>
> This is a nice tool/service created by Patrick Lightbody (who's a
> friend).
> See http://www.realityqa.com/screencasts/realitycheck_tutorial/
> RealityCheck.htm to see it in action.
>
> Right now the main issue is that it won't run selenium tests written
> in Java but Patrick says it's coming:
> http://community.realityqa.com/thread.jspa?threadID=1010
>
> The other thing that could be nice is the ability to upload our app
> inside their VM:
> http://community.realityqa.com/thread/1011 (but we can host our VMs
> to start with).
>
> What Reality QA buys us is the ability to run all our tests on all
> platforms/browsers.
Patrick just told me he has implemented executing Java tests inside
openQA so that's really cool. I'm going to try this real soon.
See http://community.realityqa.com/thread.jspa?threadID=1010 for
updates.
7) Improve the maven build
For example:
- ability to run the app on a defined port so that it doesn't
interfere with an already running xwiki instance
- stop the container when a test fails
8) Finish setting up our CI tool
Tests are pretty useless without CI. I'm working on setting up
TeamCity. I'm having some issues that I'm debugging with jetbrains
(at first glance seems caused by ObjectWeb SVN).
9) Write a tutorial for newcomers on how to create Java selenium
tests.
10) Small enhancements to the the XWiki user interface (for example
adding IDs to elements) would make it more easily testable and the
tests less fragile. Selectors like //div[4]/a/strong for the delete
button should never be used in our tests.
Right.
BTW I'm using "//div[@id='tmDelete']/a" for the delete button which
isn't that bad. Maybe you're talking about another delete button.
I think this should be fixed as we write the tests.
11) Make it also possible to run the tests, without
Firefox/IE/etc
windows popping all over the place. This is important since we want
the developers to actually run these tests (and also since we want to
integrate them into a CI tool).
BTW we get this if we use openQA.
They already take a long time to run
(5 minutes on my machine), but if they would at least let me continue
working while they run, I would run them a lot more often. According
to this http://wiki.openqa.org/display/SRC/Selenium-RC+and
+Continuous+Integration
running tests in "headless mode" should be possible at least on UNIX
machines.
Yes that would be nice to investigate as we'll need this for our CI.
Catalin will work on some of these points as part of GSOC
(http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/
FunctionalTestSuite). We haven't decided yet which points he'll work
on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5
being the most important items.
I agree that 1 (including debugging support), 3 and 5 are the points
where I could help the most, and probably also 9 and 10. Time and
knowledge permitting I could also try to help with 4 and others.