[xwiki-devs] [Proposal] Improve the QA Manual Test Reporting
Hello XWiki Community, I would like to propose a change in the visibility of manual testing. The Problem: As you know, we have a manual test report found here: http://www.xwiki.org/xwiki/bin/view/TestReports/ManualTestReportTemplate This report is being copied for each major, minor and intermediary version. Performing a full manual test takes approximately 4-5 days, on one browser. Since we support 4 browsers and 4 databases it would take a very large amount of time to test on everything before every release, since I'm the only one following this manual test report. This is why most of time I perform partial testing on some browsers. Please note that ATM, we stage only one day the artefacts before we release them. Currently our Manual Test Reports is a huge page with a lot of tables. This makes it a little bit hard to test because for each test case, I have to manually add Passed/Failed or No Tested. At the moment there is not enough visibility because we only list the Browsers we support, but we don't give exact information on which testcase has been ran on which browser. I'd like to encourage more users from the community to participate in testing XWiki, and ATM this might be hard because everything is on a single monolithic page. In order to make it as visible as I can, I have the following proposal. The Proposal: Create a dedicated subwiki for QA (maybe qa.xwiki.org), which in future could grow with other QA related things. For the moment, I would like to break the current manual test report in several pages, and several spaces. Each single test would be a wiki page, which will have objects attached to it. The space names will be the features, like WYSIWYG, Administration Application, etc. Also, if this proposal gets voted I'd like in the future to add a new field so we can mark tests that are already automated. This would again, greatly ease my work since I know what is already tested automatically, thus reducing the work needed for manual testing. After brainstorming this idea with Marius and Silvia, we have agreed on the following model: * the MTR will be organized in 2 levels - spaces that will contain feature/testcase suites with pages that hold the individual tests. - pages that will contain individual testcases. Each singular test will also have one or multiple tags attached with the features it belongs to * the following classes to model this: QA.TestClass attached to each page containing a test description (TextArea) - Tells what is the test about e.g: "Tests that images are uploaded fine after a preview." Note that the test name will be the wiki page name and the steps to perform the test will be stored in the document content QA.ExecutionClass (this will contain all information about a test execution) platform (DBList) - linked to QA.PlatformClass browser (DBList) - linked to QA.BrowserClass passed (Boolean) - value to indicate if the test has passed or failed. Absence of an object for a specific platform/browser means not tested jira (String) - link to JIRA issue (if any) tester (User) - user that performed the test date (Date) - date when test was executed QA.PlatformClass (this will contain the version of XE that have been tested/currently being tested) name (String) XE version (String) 4.1.2 QA.BrowserClass (this will contain the browser versions that we support) name (String) Mozilla Firefox version (String) 13 In the same manner we can have this for databases in the future. Final result: - If this is implemented there will be a page with a (custom) lievtable that contains all existing tests. There will be filtering options per feature for example. - There will be also a tag cloud above the livetable (similar with what we have on extensions.xwiki.org) - Marking a test as executed would require 3 clicks maximum (AJAX) right from the LiveTable (Similar to Document/Space Index with Copy/Delete/Rename/Rights), thus improving the speed of manual testing (the "fill in what you tested" part). - Going to the page of an individual test will show all XE versions and browsers on which this has been tested - Easier for everybody to see what has been tested and what not In the future, there will also be a proposal/brainstorming about how to link this with the automated tests. I consider we could be a little bit more visible about this too. Also, there is no visible link between the manual tests and the automated tests. It would be nice if everybody could see for which manual test we have an automated one. I am waiting for your feedback since I am eager to start working on this. Regards, Sorin B.
Hi Sorin, On Tue, Jul 3, 2012 at 10:46 AM, Sorin Burjan <[email protected]>wrote:
Hello XWiki Community,
I would like to propose a change in the visibility of manual testing.
The Problem:
As you know, we have a manual test report found here: http://www.xwiki.org/xwiki/bin/view/TestReports/ManualTestReportTemplate This report is being copied for each major, minor and intermediary version. Performing a full manual test takes approximately 4-5 days, on one browser. Since we support 4 browsers and 4 databases it would take a very large amount of time to test on everything before every release, since I'm the only one following this manual test report. This is why most of time I perform partial testing on some browsers. Please note that ATM, we stage only one day the artefacts before we release them. Currently our Manual Test Reports is a huge page with a lot of tables. This makes it a little bit hard to test because for each test case, I have to manually add Passed/Failed or No Tested. At the moment there is not enough visibility because we only list the Browsers we support, but we don't give exact information on which testcase has been ran on which browser. I'd like to encourage more users from the community to participate in testing XWiki, and ATM this might be hard because everything is on a single monolithic page.
Anything that makes it even easier to know exactly what is tested and validated as working in all XWiki releases is welcome :-) In order to make it as visible as I can, I have the following proposal.
The Proposal:
Create a dedicated subwiki for QA (maybe qa.xwiki.org), which in future could grow with other QA related things. For the moment, I would like to break the current manual test report in several pages, and several spaces. Each single test would be a wiki page, which will have objects attached to it. The space names will be the features, like WYSIWYG, Administration Application, etc.
Also, if this proposal gets voted I'd like in the future to add a new field so we can mark tests that are already automated. This would again, greatly ease my work since I know what is already tested automatically, thus reducing the work needed for manual testing.
After brainstorming this idea with Marius and Silvia, we have agreed on the following model:
* the MTR will be organized in 2 levels - spaces that will contain feature/testcase suites with pages that hold the individual tests. - pages that will contain individual testcases. Each singular test will also have one or multiple tags attached with the features it belongs to
* the following classes to model this:
QA.TestClass attached to each page containing a test description (TextArea) - Tells what is the test about e.g: "Tests that images are uploaded fine after a preview."
Note that the test name will be the wiki page name and the steps to perform the test will be stored in the document content
QA.ExecutionClass (this will contain all information about a test execution) platform (DBList) - linked to QA.PlatformClass browser (DBList) - linked to QA.BrowserClass passed (Boolean) - value to indicate if the test has passed or failed. Absence of an object for a specific platform/browser means not tested jira (String) - link to JIRA issue (if any) tester (User) - user that performed the test date (Date) - date when test was executed
QA.PlatformClass (this will contain the version of XE that have been tested/currently being tested) name (String) XE version (String) 4.1.2
QA.BrowserClass (this will contain the browser versions that we support) name (String) Mozilla Firefox version (String) 13
In the same manner we can have this for databases in the future.
Final result:
- If this is implemented there will be a page with a (custom) lievtable that contains all existing tests. There will be filtering options per feature for example. - There will be also a tag cloud above the livetable (similar with what we have on extensions.xwiki.org) - Marking a test as executed would require 3 clicks maximum (AJAX) right from the LiveTable (Similar to Document/Space Index with Copy/Delete/Rename/Rights), thus improving the speed of manual testing (the "fill in what you tested" part).
You might be able to do even better here. There is a livetable extension that lets an user modify a property straight by clicking on the livetable cell (contributed by Ludovic).
- Going to the page of an individual test will show all XE versions and browsers on which this has been tested - Easier for everybody to see what has been tested and what not
Sounds good. In the future, there will also be a proposal/brainstorming about how
to link this with the automated tests. I consider we could be a little bit more visible about this too. Also, there is no visible link between the manual tests and the automated tests. It would be nice if everybody could see for which manual test we have an automated one.
I am waiting for your feedback since I am eager to start working on this.
I'm looking forward to using it :-) Guillaume Regards,
Sorin B. _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Sorin, it's not that we have loads of time to contribute here, especially as we are fairly unable to stand on the bleeding edge of the high-pace of xwiki releases, but I think your initiative is likely to be useful to Curriki. For QA management, we've been using Google docs since a while, allowing live collaborative editing. Paul Le 3 juil. 2012 à 10:46, Sorin Burjan a écrit :
Hello XWiki Community,
I would like to propose a change in the visibility of manual testing.
The Problem:
As you know, we have a manual test report found here: http://www.xwiki.org/xwiki/bin/view/TestReports/ManualTestReportTemplate This report is being copied for each major, minor and intermediary version. Performing a full manual test takes approximately 4-5 days, on one browser. Since we support 4 browsers and 4 databases it would take a very large amount of time to test on everything before every release, since I'm the only one following this manual test report. This is why most of time I perform partial testing on some browsers. Please note that ATM, we stage only one day the artefacts before we release them. Currently our Manual Test Reports is a huge page with a lot of tables. This makes it a little bit hard to test because for each test case, I have to manually add Passed/Failed or No Tested. At the moment there is not enough visibility because we only list the Browsers we support, but we don't give exact information on which testcase has been ran on which browser. I'd like to encourage more users from the community to participate in testing XWiki, and ATM this might be hard because everything is on a single monolithic page.
In order to make it as visible as I can, I have the following proposal.
The Proposal:
Create a dedicated subwiki for QA (maybe qa.xwiki.org), which in future could grow with other QA related things. For the moment, I would like to break the current manual test report in several pages, and several spaces. Each single test would be a wiki page, which will have objects attached to it. The space names will be the features, like WYSIWYG, Administration Application, etc.
Also, if this proposal gets voted I'd like in the future to add a new field so we can mark tests that are already automated. This would again, greatly ease my work since I know what is already tested automatically, thus reducing the work needed for manual testing.
After brainstorming this idea with Marius and Silvia, we have agreed on the following model:
* the MTR will be organized in 2 levels - spaces that will contain feature/testcase suites with pages that hold the individual tests. - pages that will contain individual testcases. Each singular test will also have one or multiple tags attached with the features it belongs to
* the following classes to model this:
QA.TestClass attached to each page containing a test description (TextArea) - Tells what is the test about e.g: "Tests that images are uploaded fine after a preview."
Note that the test name will be the wiki page name and the steps to perform the test will be stored in the document content
QA.ExecutionClass (this will contain all information about a test execution) platform (DBList) - linked to QA.PlatformClass browser (DBList) - linked to QA.BrowserClass passed (Boolean) - value to indicate if the test has passed or failed. Absence of an object for a specific platform/browser means not tested jira (String) - link to JIRA issue (if any) tester (User) - user that performed the test date (Date) - date when test was executed
QA.PlatformClass (this will contain the version of XE that have been tested/currently being tested) name (String) XE version (String) 4.1.2
QA.BrowserClass (this will contain the browser versions that we support) name (String) Mozilla Firefox version (String) 13
In the same manner we can have this for databases in the future.
Final result:
- If this is implemented there will be a page with a (custom) lievtable that contains all existing tests. There will be filtering options per feature for example. - There will be also a tag cloud above the livetable (similar with what we have on extensions.xwiki.org) - Marking a test as executed would require 3 clicks maximum (AJAX) right from the LiveTable (Similar to Document/Space Index with Copy/Delete/Rename/Rights), thus improving the speed of manual testing (the "fill in what you tested" part). - Going to the page of an individual test will show all XE versions and browsers on which this has been tested - Easier for everybody to see what has been tested and what not
In the future, there will also be a proposal/brainstorming about how to link this with the automated tests. I consider we could be a little bit more visible about this too. Also, there is no visible link between the manual tests and the automated tests. It would be nice if everybody could see for which manual test we have an automated one.
I am waiting for your feedback since I am eager to start working on this.
Regards, Sorin B. _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Sorin, On Jul 3, 2012, at 10:46 AM, Sorin Burjan wrote:
Hello XWiki Community,
I would like to propose a change in the visibility of manual testing.
The Problem:
As you know, we have a manual test report found here: http://www.xwiki.org/xwiki/bin/view/TestReports/ManualTestReportTemplate This report is being copied for each major, minor and intermediary version. Performing a full manual test takes approximately 4-5 days, on one browser. Since we support 4 browsers and 4 databases it would take a very large amount of time to test on everything before every release, since I'm the only one following this manual test report. This is why most of time I perform partial testing on some browsers. Please note that ATM, we stage only one day the artefacts before we release them. Currently our Manual Test Reports is a huge page with a lot of tables. This makes it a little bit hard to test because for each test case, I have to manually add Passed/Failed or No Tested. At the moment there is not enough visibility because we only list the Browsers we support, but we don't give exact information on which testcase has been ran on which browser. I'd like to encourage more users from the community to participate in testing XWiki, and ATM this might be hard because everything is on a single monolithic page.
Yes the current MTR is not that useful because: * It says passed but I've realized that it's not because it says passed that it's been tested * There's no way to know what tests have been tested on which environment * The MTR is not updated regularly based on new features/improvements added to XE * Several tests in the MTR are covered by automated tests and do not need to be manual * Several domains not covered by automated tests are not covered by manual tests
In order to make it as visible as I can, I have the following proposal.
The Proposal:
Create a dedicated subwiki for QA (maybe qa.xwiki.org), which in future could grow with other QA related things. For the moment, I would like to break the current manual test report in several pages, and several spaces. Each single test would be a wiki page, which will have objects attached to it. The space names will be the features, like WYSIWYG, Administration Application, etc.
Also, if this proposal gets voted I'd like in the future to add a new field so we can mark tests that are already automated. This would again, greatly ease my work since I know what is already tested automatically, thus reducing the work needed for manual testing.
After brainstorming this idea with Marius and Silvia, we have agreed on the following model:
* the MTR will be organized in 2 levels - spaces that will contain feature/testcase suites with pages that hold the individual tests. - pages that will contain individual testcases. Each singular test will also have one or multiple tags attached with the features it belongs to
* the following classes to model this:
QA.TestClass attached to each page containing a test description (TextArea) - Tells what is the test about e.g: "Tests that images are uploaded fine after a preview."
Note that the test name will be the wiki page name and the steps to perform the test will be stored in the document content
QA.ExecutionClass (this will contain all information about a test execution) platform (DBList) - linked to QA.PlatformClass browser (DBList) - linked to QA.BrowserClass passed (Boolean) - value to indicate if the test has passed or failed. Absence of an object for a specific platform/browser means not tested jira (String) - link to JIRA issue (if any) tester (User) - user that performed the test date (Date) - date when test was executed
QA.PlatformClass (this will contain the version of XE that have been tested/currently being tested) name (String) XE version (String) 4.1.2
QA.BrowserClass (this will contain the browser versions that we support) name (String) Mozilla Firefox version (String) 13
In the same manner we can have this for databases in the future.
Final result:
- If this is implemented there will be a page with a (custom) lievtable that contains all existing tests. There will be filtering options per feature for example. - There will be also a tag cloud above the livetable (similar with what we have on extensions.xwiki.org) - Marking a test as executed would require 3 clicks maximum (AJAX) right from the LiveTable (Similar to Document/Space Index with Copy/Delete/Rename/Rights), thus improving the speed of manual testing (the "fill in what you tested" part). - Going to the page of an individual test will show all XE versions and browsers on which this has been tested - Easier for everybody to see what has been tested and what not
In the future, there will also be a proposal/brainstorming about how to link this with the automated tests. I consider we could be a little bit more visible about this too. Also, there is no visible link between the manual tests and the automated tests. It would be nice if everybody could see for which manual test we have an automated one.
I am waiting for your feedback since I am eager to start working on this.
This is a very good idea. +1 in general for having a testing application (which we should make as generic as possible and publish on e.x.o). +1 too to let contributors create tests and/or test results in that testing application. Thanks -Vincent
participants (4)
-
Guillaume Lerouge -
Paul Libbrecht -
Sorin Burjan -
Vincent Massol