[xwiki-devs] [Proposal] Introduce automated escaping tests

Hello devs, I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests. Currently, escaping tests are usually written using selenium2, see e.g.: http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-te… This is a bad solution, slow and tricky to write. Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions. The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily. The implementation can be found in the sandbox: https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-e… It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")). Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though). Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run: mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install Some implementation details: org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite. org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc. org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported. I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise). WDYT? Regards, Alex