[xwiki-dev] Testing Ideas for XWiki

[Vincent Massol]

On Jun 8, 2007, at 11:33 AM, Catalin Hritcu wrote:

[snip]

>> (try creating a test that types some
>> content in our Tiny MCE editor and you'll understand that part ;-))
>>
> This is actually not so hard to do, once you know how it should be
> done, but for someone who does not know it it's almost impossible to
> discover:
> <tr>
> 	<td>type</td>
> 	<td>mceSpanFonts</td>
> 	<td>Some text to be inserted in the Tiny MCE editor</td>
> </tr>

ok, now try to clear the content of a page being editer in WYSIWYG  
mode ;-)

>> - run automatically as part of our build/CI
>>
> Is this not also the case for Selenese tests?

No the maven plugin we're using works with Selenium RC. There's a  
mavenium plugin for HTML tests though but I haven't tried it.

[snip]

> In my opinion writing the tests in our DSL will be a lot easier than
> _writing_ them in Selenese, even for someone who does not know Java at
> all. However, with the Selenium IDE you are also able to _record_
> tests, which we definitely won't be able to do with Java.

I'm now finding myself writing more and more tests directly in java  
without going through the Selenium IDE. I think it's a matter of  
having the DSL in place mostly. And we already have a good start.

[snip]

> What is also possible Selenium IDE, and is unfortunately not (yet)
> possible in our Java framework is easily debugging functional tests.
> With Selenium IDE you can set breakpoints, run a test step-by-step,
> see tracing information, inspect the html currently rendered by the
> browser (with firebug for example), for debugging your tests. For the
> Java tests now you can only run all of them as part of the Maven build
> and see whether some of them fail or not. If they fail you get a stack
> trace, which is never sufficient to realize what went wrong.
>
> We should aim to do a lot better than this. It should be possible to
> run and debug the Java functional tests in Eclipse or Idea like usual
> JUnit tests. The setup and teardown phases will be costly (starting a
> xwiki server and a browser) and we should log a lot more tracing
> information, but other than this I don't see why this wouldn't work.
> We should be able to debug Java tests at least as easily as it is we
> debug Selenese tests in Selenium IDE. Otherwise we will find ourselves
> more an more often manually converting Java tests into Selenese just
> to be able to debug them, and this is unacceptable.
>
> One (definitely bad) alternative would be for the Java tests not to
> access Selenium directly but just generate Selenese tests which are
> then run separately. This would at least allow one to debug these
> generated tests in Selenium IDE, but I think we should consider it
> only if it turns out that what I wrote in the previous paragraph is
> for some reason not achievable.

We can do debugging! I'm doing it. Here's the current way of doing it  
(we should improve it):

* go in application-test/, edit the pom.xml to replace  
<background>true</background> with false. Note: We need to find a way  
to pass this value on the maven command line. This is possible, I  
just haven't really looked into that.
* start the xwiki version you wish to test
* go in your IDE, find the test you wish to run, put breakpoints and  
run it in debug mode. Not only you'll get the Java debugging power  
but you can also see the browser in action, verify stuff in the  
browser, etc. It works real well.

>> I'm currently 100% convinced that our Java choice is right but I know
>> Ludovic isn't fully convinced which is why I'm listing this point
>> here to see what others think.
>>
> I would also prefer Java tests, provided that we can also debug  
> them easily.

Your wish has come true! :)

>> 1) Finish setting up our Java functional test fwk.
>>
>> Goals:
>> - make it as easy as possible to use.
>> - make it low maintenance (ie tests shouldn't fail often and if some
>> UI is changed it should be very easy to fix the tests)
>>
>> Tasks:
>> - Continue improving our DSL for testing. Specifically add the
>> ability to test over several skins and create one DSL per skin.
>> - Add DSL methods for asserting the rendered HTML (cf Catalin's tests
>> in http://jira.xwiki.org/jira/browse/XWIKI-1207)
>>
> More tasks:
> - Make it easy to debug the Java functional tests

done

> - Add useful tracing and error log messages

Not sure what this is about. There's already a debug feature of the  
selenium plugin we're using. What do you want exactly?

[snip]

>> 4) Explore in-VM testing
>>
>> The idea is to have several VM (Virtual Machines) running different
>> OSes for example (Windows, linux, Mac) and a combination of Databases
>> (Oracle, MySQL, PostGre). You run the XWiki maven build in them so
>> that the tests get executed in VM. This allows 2 things:
>>
>> a) test for different environment. For example this tests our start/
>> stop scripts, our database setups
>> b) this allows more complex tests to be written like: virtual wikis
>
>> c) this allows to implement hard to write tests like the "add
>> attachment" one. For this you need the attachment to be on your file
>> system. With a VM you control the full setup of the VM and you can
>> restore the VM at each run
>>
> Just wondering: how many tests like this are there? At first sight it
> seems that not many.

A lot... The problem is that they are the most important... For  
example ensuring that our build runs on different DB, OSes, etc is  
quite crucial.

> What is most important and not on this list is:
> d) In-VM testing would also allow us to test on more web browsers on
> more platforms.

Yes . However we don't need VM for this. This is provided by OpenQA  
as of now. See below.

> This is related to 6), maybe another way to achieve the same thing.
> Unless we mean different things by "in-VM testing" :)
>
>> 5) create a test plan listing tests to be written
>>
>> We need this written in JIRA. This goes with 3).
>>
> Here the opinion of the xwiki developers and users would be greatly
> appreciated. Which are the features you think that should receive the
> most care, or which are the test cases which we should not be leave
> out? Please enter your opinions here:
> http://www.xwiki.org/xwiki/bin/view/Idea/FunctionalTestPlanListing

I think you need to provide a first draft with more details and then  
based on this the community can react.

> After the discussion with Vincent it seems that we should focus on
> features that are very important for end-users but the developers tend
> to overlook when testing ... like the WYSIWYG editor. What do you
> think?

Sure.

>
>> 6) Explore using Reality QA
>>
>> This is a nice tool/service created by Patrick Lightbody (who's a
>> friend).
>> See http://www.realityqa.com/screencasts/realitycheck_tutorial/
>> RealityCheck.htm to see it in action.
>>
>> Right now the main issue is that it won't run selenium tests written
>> in Java but Patrick says it's coming:
>> http://community.realityqa.com/thread.jspa?threadID=1010
>>
>> The other thing that could be nice is the ability to upload our app
>> inside their VM:
>> http://community.realityqa.com/thread/1011 (but we can host our VMs
>> to start with).
>>
>> What Reality QA buys us is the ability to run all our tests on all
>> platforms/browsers.

Patrick just told me he has implemented executing Java tests inside  
openQA so that's really cool. I'm going to try this real soon.
See http://community.realityqa.com/thread.jspa?threadID=1010 for  
updates.

>>
>> 7) Improve the maven build
>>
>> For example:
>> - ability to run the app on a defined port so that it doesn't
>> interfere with an already running xwiki instance
>> - stop the container when a test fails
>>
>> 8) Finish setting up our CI tool
>>
>> Tests are pretty useless without CI. I'm working on setting up
>> TeamCity. I'm having some issues that I'm debugging with jetbrains
>> (at first glance seems caused by ObjectWeb SVN).
>>
>
> 9) Write a tutorial for newcomers on how to create Java selenium  
> tests.

Yes. I have started here:
http://www.xwiki.org/xwiki/bin/view/Community/Testing

We need to continue improving that page. Help is most welcome.

>
> 10)  Small enhancements to the the XWiki user interface (for example
> adding IDs to  elements) would make it more easily testable and the
> tests less fragile. Selectors like //div[4]/a/strong for the delete
> button should never be used in our tests.

Right.

BTW I'm using "//div[@id='tmDelete']/a" for the delete button which  
isn't that bad. Maybe you're talking about another delete button.

I think this should be fixed as we write the tests.

> 11) Make it also possible to run the tests, without Firefox/IE/etc
> windows popping all over the place. This is important since we want
> the developers to actually run these tests (and also since we want to
> integrate them into a CI tool).

BTW we get this if we use openQA.

> They already take a long time to run
> (5 minutes on my machine), but if they would at least let me continue
> working while they run, I would run them a lot more often. According
> to this http://wiki.openqa.org/display/SRC/Selenium-RC+and 
> +Continuous+Integration
> running tests in "headless mode" should be possible at least on UNIX
> machines.

Yes that would be nice to investigate as we'll need this for our CI.

>>
>> Catalin will work on some of these points as part of GSOC
>> (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/
>> FunctionalTestSuite). We haven't decided yet which points he'll work
>> on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5
>> being the most important items.
>>
> I agree that 1 (including debugging support), 3 and 5 are the points
> where I could help the most, and probably also 9 and 10. Time and
> knowledge permitting I could also try to help with 4 and others.

Great stuff.

Let us know your planning so that we set some milestones and progress  
in a directed manner rather than in a best-effort one.

I can help too.

Thanks Catalin!
-Vincent