[xwiki-dev] Testing Ideas for XWiki

[Catalin Hritcu]

Hi,

On 6/8/07, Vincent Massol <vincent at massol.net> wrote:
>
> On Jun 8, 2007, at 11:33 AM, Catalin Hritcu wrote:
>
> [snip]
>
> >> (try creating a test that types some
> >> content in our Tiny MCE editor and you'll understand that part ;-))
> >>
> > This is actually not so hard to do, once you know how it should be
> > done, but for someone who does not know it it's almost impossible to
> > discover:
> > <tr>
> >       <td>type</td>
> >       <td>mceSpanFonts</td>
> >       <td>Some text to be inserted in the Tiny MCE editor</td>
> > </tr>
>
> ok, now try to clear the content of a page being editer in WYSIWYG
> mode ;-)
>
I'm not trying to annoy you, but this is the same :)
<tr>
	<td>type</td>
	<td>mceSpanFonts</td>
	<td></td>
</tr>

[snip]

> We can do debugging! I'm doing it. Here's the current way of doing it
> (we should improve it):
>
> * go in application-test/, edit the pom.xml to replace
> <background>true</background> with false. Note: We need to find a way
> to pass this value on the maven command line. This is possible, I
> just haven't really looked into that.
> * start the xwiki version you wish to test
> * go in your IDE, find the test you wish to run, put breakpoints and
> run it in debug mode. Not only you'll get the Java debugging power
> but you can also see the browser in action, verify stuff in the
> browser, etc. It works real well.
>
Sorry but I was not aware of this already working. I tried debugging a
Java test like this and it works smoothly. Thank you.

[snip]

> > - Add useful tracing and error log messages
>
> Not sure what this is about. There's already a debug feature of the
> selenium plugin we're using. What do you want exactly?
>
When a test fails I want to be able to get more information than just
a stack trace, something like the Log tab at the bottom of Selenium
IDE:
# [info] Executing: |open | /mail/?ik=6d3a96df6c ... | |
# [info] Executing: |selectFrame | main | |
# [info] Executing: |selectFrame | v1 | |
# [info] Executing: |type | ta_compose | Hi, On 6/8/07, Vincent Massol ... | |
[This is me replaying the recording I made when writing this email]

Didn't know about it until now, but one can start maven with the
"-Ddebug=true" switch and then all messages from the selenium server
are saved to:
application-test/target/selenium/server.log

I wonder if there is an easy way to have only level info and higher
logged (because with "-Ddebug=true" the useful messages are borrowed
under lots of debug messages)?

[snip]

> > What is most important and not on this list is:
> > d) In-VM testing would also allow us to test on more web browsers on
> > more platforms.
>
> Yes . However we don't need VM for this. This is provided by OpenQA
> as of now. See below.
>
I wonder, with OpenQA will be also able to directly debug the tests on
different browsers, or only run them as a batch on their (virtual)
machines ?

[snip]

> >> 5) create a test plan listing tests to be written
> >>
> >> We need this written in JIRA. This goes with 3).
> >>
> > Here the opinion of the xwiki developers and users would be greatly
> > appreciated. Which are the features you think that should receive the
> > most care, or which are the test cases which we should not be leave
> > out? Please enter your opinions here:
> > http://www.xwiki.org/xwiki/bin/view/Idea/FunctionalTestPlanListing
>
> I think you need to provide a first draft with more details and then
> based on this the community can react.
>
One has to start somewhere  :) . I'll add more ideas to the page as
they come, and when it's in better shape announce it in a separate
thread on xwiki-users.

[snip]

> Patrick just told me he has implemented executing Java tests inside
> openQA so that's really cool. I'm going to try this real soon.
> See http://community.realityqa.com/thread.jspa?threadID=1010 for
> updates.
>
Nice :)

[snip]

> > 10)  Small enhancements to the the XWiki user interface (for example
> > adding IDs to  elements) would make it more easily testable and the
> > tests less fragile. Selectors like //div[4]/a/strong for the delete
> > button should never be used in our tests.
>
> Right.
>
> BTW I'm using "//div[@id='tmDelete']/a" for the delete button which
> isn't that bad. Maybe you're talking about another delete button.
>
If you record deleting a page using selenium ide you get "//div[4]/a/strong".

[snip]

> >>
> >> Catalin will work on some of these points as part of GSOC
> >> (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/
> >> FunctionalTestSuite). We haven't decided yet which points he'll work
> >> on but at least he'll do/help on the following: 1, 3 and 5. 3 and 5
> >> being the most important items.
> >>
> > I agree that 1 (including debugging support), 3 and 5 are the points
> > where I could help the most, and probably also 9 and 10. Time and
> > knowledge permitting I could also try to help with 4 and others.
>
> Great stuff.
>
> Let us know your planning so that we set some milestones and progress
> in a directed manner rather than in a best-effort one.
>
I will. Still for a couple of weeks it will still be best-effort as I
am overwhelmed with other work.

> I can help too.
>
Thank you.

Regards,
Catalin