[xwiki-devs] [Proposal] Introduce automated escaping tests
Hello devs, I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests. Currently, escaping tests are usually written using selenium2, see e.g.: http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-tes... This is a bad solution, slow and tricky to write. Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions. The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily. The implementation can be found in the sandbox: https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-es... It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")). Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though). Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run: mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install Some implementation details: org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite. org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc. org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported. I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise). WDYT? Regards, Alex
+1 On Fri, Jun 25, 2010 at 18:08, Alex Busenius <[email protected]> wrote:
Hello devs,
I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests.
Currently, escaping tests are usually written using selenium2, see e.g.: http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-tes... This is a bad solution, slow and tricky to write.
Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions.
The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily.
The implementation can be found in the sandbox:
https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-es...
It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")).
Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though).
Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run:
mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install
Some implementation details:
org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite.
org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc.
org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported.
I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise).
WDYT?
Regards,
Alex _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
+1 Sounds great! - Asiri On Fri, Jun 25, 2010 at 9:38 PM, Alex Busenius <[email protected]>wrote:
Hello devs,
I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests.
Currently, escaping tests are usually written using selenium2, see e.g.:
http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-tes... This is a bad solution, slow and tricky to write.
Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions.
The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily.
The implementation can be found in the sandbox:
https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-es...
It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")).
Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though).
Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run:
mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install
Some implementation details:
org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite.
org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc.
org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported.
I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise).
WDYT?
Regards,
Alex _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
+1 On 06/25/2010 06:08 PM, Alex Busenius wrote:
Hello devs,
I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests.
Currently, escaping tests are usually written using selenium2, see e.g.: http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-tes... This is a bad solution, slow and tricky to write.
Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions.
The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily.
The implementation can be found in the sandbox:
https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-es...
It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")).
Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though).
Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run:
mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install
Some implementation details:
org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite.
org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc.
org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported.
I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise).
-- Sergiu Dumitriu http://purl.org/net/sergiu/
+1 Thanks, Marius On 06/25/2010 07:08 PM, Alex Busenius wrote:
Hello devs,
I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests.
Currently, escaping tests are usually written using selenium2, see e.g.: http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-tes... This is a bad solution, slow and tricky to write.
Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions.
The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily.
The implementation can be found in the sandbox:
https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-es...
It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")).
Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though).
Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run:
mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install
Some implementation details:
org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite.
org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc.
org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported.
I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise).
WDYT?
Regards,
Alex _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
+1 Caleb Marius Dumitru Florea wrote:
+1
Thanks, Marius
On 06/25/2010 07:08 PM, Alex Busenius wrote:
Hello devs,
I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests.
Currently, escaping tests are usually written using selenium2, see e.g.: http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-tes... This is a bad solution, slow and tricky to write.
Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions.
The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily.
The implementation can be found in the sandbox:
https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-es...
It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")).
Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though).
Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run:
mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install
Some implementation details:
org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite.
org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc.
org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported.
I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise).
WDYT?
Regards,
Alex _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
The corresponding issue is here: http://jira.xwiki.org/jira/browse/XE-685 Would be great if somebody could take a look at it. Thanks, Alex On 06/25/2010 06:08 PM, Alex Busenius wrote:
Hello devs,
I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests.
Currently, escaping tests are usually written using selenium2, see e.g.: http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-tes... This is a bad solution, slow and tricky to write.
Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions.
The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily.
The implementation can be found in the sandbox:
https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-es...
It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")).
Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though).
Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run:
mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install
Some implementation details:
org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite.
org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc.
org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported.
I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise).
WDYT?
Regards,
Alex _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
+1 These tests take 15 minutes on a duel core and the time cost could be expected to increase as more potential attack vectors are discovered. Would it be advisable to make this a separate project on hudson? Caleb Alex Busenius wrote:
The corresponding issue is here:
http://jira.xwiki.org/jira/browse/XE-685
Would be great if somebody could take a look at it.
Thanks, Alex
On 06/25/2010 06:08 PM, Alex Busenius wrote:
Hello devs,
I'd like to add a new test framework aimed on catching XML escaping bugs that we currently have almost everywhere. Escaping bugs cause broken, unusable pages when special characters are used in space/page/category/user/whatever names and often present serious security issues. This test framework will run automatic tests and also allow for writing manual tests.
Currently, escaping tests are usually written using selenium2, see e.g.: http://svn.xwiki.org/svnroot/xwiki/enterprise/trunk/distribution-test/ui-tes... This is a bad solution, slow and tricky to write.
Writing unit tests for such bugs is also not a good solution, because most of the time they originate in velocity templates. It is hard to test a single template in isolation, they usually require access to many API functions.
The solution I propose works similar to webstandards tests. The framework searches for all templates and documents in XWiki Enterprise *.xar and *.war and generates automatic escaping tests for each of them. Each test then requests URLs from XWiki server with different parameters and runs a custom validator that searches for not escaped data in the response. Manual tests can also be added easily.
The implementation can be found in the sandbox:
https://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-enterprise-test-es...
It currently implements some simple automated escaping tests on all templates and applications from the XAR. The parameters to use are chosen based on the source, using matching with regular expressions (searches for things like $request.get("something")).
Full test run takes 8 minutes on my laptop. Current results: Tests run: 499, Failures: 166, Errors: 3, Skipped: 0 (there are still some false positives that I need to track down though).
Files and test classes can be excluded from the tests using various properties set in maven (see pom.xml). Usual -Dtest and -Dpattern are also supported, e.g. to test all pages from "Blog" space, run:
mvn -Dtest=ApplicationTest -Dpattern='^Blog/' install
Some implementation details:
org.xwiki.escaping.suite It is a custom JUnit4 test suite. It reads given zip file, then creates and initializes instances of the test class for each found file. The test class must implement FileTest (defines initialize()) and annotated with @RunWith(ArchiveSuite.class). Test classes can decide based on file name and content whether they can test the given file or not. ApplicationTest and TemplateTest use this suite.
org.xwiki.escaping.framework Contains base classes for the escaping tests (implement downloading data from server, regex-based "parsing" etc.), escaping validator etc.
org.xwiki.escaping Contains the tests (2 automatic and 1 manual). Instances of the test classes (built by ArchiveSuite or manual ones) a run using the default JUnit4 test runner, so all fancy test method annotations are supported.
I propose to add this test framework to the main repository starting with 2.5 branch. This will give us more time to fix current issues (problematic files can be easily excluded to generate less noise).
WDYT?
Regards,
Alex _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
participants (6)
-
Alex Busenius -
Asiri Rathnayake -
Caleb James DeLisle -
Marius Dumitru Florea -
Sergiu Dumitriu -
Thomas Mortagne