[xwiki-devs] Improved supported database testing and testing results
Hello, As you know, we have defined a supported database strategy, found here: http://dev.xwiki.org/xwiki/bin/view/Community/DatabaseSupportStrategy At the moment, in our release notes, we report the supported list of databases in the "Tested Databases" section. This is a little bit misleading, because we don't perform a full manual test on each of them. Also, at the moment, we don't have automated tests running on these databases. What I currently do is to test migration on each databases from an older version (in my case, I am using 2.7.2) to the current upcoming release. I usually perform these tests before RC1 so we make sure issues won't slip in the final release. Also, this testing is done on vanilla wiki's, so they don't reflect or simulate a large wiki migration with thousands of users and pages. What I want to propose is: 1) improve the "Tested Databases" section, so we list that we only test migrations and we don't perform a full test on each of the databases. This would mean adding a small explanation about this in the release notes pages. 2) When we release a new version, migrate myxwiki.org to the latest version, so users using it can report possible issues due to migration. This is a little hard for me to do since if something goes wrong, I could break a lot of things. Someone with more experience should do it, if there volunteers. I am waiting your feedback about this since I want to improve the visibility of testing, and also try to test migrations in a more real world environment Regards, Sorin B.
On 06/25/2012 08:40 AM, Sorin Burjan wrote:
Hello,
As you know, we have defined a supported database strategy, found here: http://dev.xwiki.org/xwiki/bin/view/Community/DatabaseSupportStrategy
At the moment, in our release notes, we report the supported list of databases in the "Tested Databases" section. This is a little bit misleading, because we don't perform a full manual test on each of them. Also, at the moment, we don't have automated tests running on these databases.
What I currently do is to test migration on each databases from an older version (in my case, I am using 2.7.2) to the current upcoming release. I usually perform these tests before RC1 so we make sure issues won't slip in the final release. Also, this testing is done on vanilla wiki's, so they don't reflect or simulate a large wiki migration with thousands of users and pages.
What I want to propose is:
1) improve the "Tested Databases" section, so we list that we only test migrations and we don't perform a full test on each of the databases. This would mean adding a small explanation about this in the release notes pages.
+1 for being as honest as we can about what we do and what we don't do.
2) When we release a new version, migrate myxwiki.org to the latest version, so users using it can report possible issues due to migration. This is a little hard for me to do since if something goes wrong, I could break a lot of things. Someone with more experience should do it, if there volunteers.
I think this is already established as standard practice, unfortunately we often lack the time or do not think to do it. I would suggest 2 additional ideas: #1 brainstorm how we can automate functional tests running on multiple databases. Currently storage-tests run with and without filesystem attachments enabled so the basic framework for tests on multiple configurations exists. What we need is a developer with the time and the dedication to want to make it happen, perhaps it could be considered as a candidate for the 4.3 roadmap. #2 brainstorm how we can automate upgrades of myxwiki.org, this is a huge project so IMO it should be broken up and resolved as a set of small issues, similarly to how the release process has been (partially) automated. Getting the issues out of the way to make this work is beneficial to users in the field as well. Thanks, Caleb
I am waiting your feedback about this since I want to improve the visibility of testing, and also try to test migrations in a more real world environment
Regards, Sorin B. _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
HI Sorin, On Jun 25, 2012, at 2:40 PM, Sorin Burjan wrote:
Hello,
As you know, we have defined a supported database strategy, found here: http://dev.xwiki.org/xwiki/bin/view/Community/DatabaseSupportStrategy
At the moment, in our release notes, we report the supported list of databases in the "Tested Databases" section. This is a little bit misleading, because we don't perform a full manual test on each of them. Also, at the moment, we don't have automated tests running on these databases.
What I currently do is to test migration on each databases from an older version (in my case, I am using 2.7.2) to the current upcoming release. I usually perform these tests before RC1 so we make sure issues won't slip in the final release. Also, this testing is done on vanilla wiki's, so they don't reflect or simulate a large wiki migration with thousands of users and pages.
What I want to propose is:
1) improve the "Tested Databases" section, so we list that we only test migrations and we don't perform a full test on each of the databases. This would mean adding a small explanation about this in the release notes pages.
I don't fully agree with this. We should do some manual tests on these DBs, in addition to testing the migration. You need to define what tests to do but at the very minimum we need to have a sanity test checklist, otherwise we cannot honestly say we have tested the DB. BTW we don't really test database migration properly as it's been shown when we tried to upgrade myxwiki.org. We get a real life DB and use it instead of using just a clean and pristine DB.
2) When we release a new version, migrate myxwiki.org to the latest version, so users using it can report possible issues due to migration. This is a little hard for me to do since if something goes wrong, I could break a lot of things. Someone with more experience should do it, if there volunteers.
I'm +1 and I actually think it should be done *before* we release as otherwise it has little value. IMO we need to migrate at least when we release RC1 so that we can fix issues for the final. Now someone need to spend the time to automate the update since it takes quite a while, especially with the cluster. Note that I've been trying to upgrade myxwiki.org manually for a week now but I cannot since our database migration is just not working ATM. We're working on this.
I am waiting your feedback about this since I want to improve the visibility of testing, and also try to test migrations in a more real world environment
+1 for more real word! :) Thanks -Vincent
On Fri, Jun 29, 2012 at 5:00 PM, Vincent Massol <[email protected]> wrote:
HI Sorin,
On Jun 25, 2012, at 2:40 PM, Sorin Burjan wrote:
Hello,
As you know, we have defined a supported database strategy, found here: http://dev.xwiki.org/xwiki/bin/view/Community/DatabaseSupportStrategy
At the moment, in our release notes, we report the supported list of databases in the "Tested Databases" section. This is a little bit misleading, because we don't perform a full manual test on each of them. Also, at the moment, we don't have automated tests running on these databases.
What I currently do is to test migration on each databases from an older version (in my case, I am using 2.7.2) to the current upcoming release. I usually perform these tests before RC1 so we make sure issues won't slip in the final release. Also, this testing is done on vanilla wiki's, so they don't reflect or simulate a large wiki migration with thousands of users and pages.
What I want to propose is:
1) improve the "Tested Databases" section, so we list that we only test migrations and we don't perform a full test on each of the databases. This would mean adding a small explanation about this in the release notes pages.
I don't fully agree with this. We should do some manual tests on these DBs, in addition to testing the migration.
I agree with this. I'm just saying that until we decide the minimal stuff to be tested on the databases, we should explicitly state on the release notes that we only tested small/vanilla wikis migration (so users with large wikis know this). Just trying to be as accurate as possible about what testing has been done so we don't mislead users. Regards, Sorin B.
You need to define what tests to do but at the very minimum we need to have a sanity test checklist, otherwise we cannot honestly say we have tested the DB.
BTW we don't really test database migration properly as it's been shown when we tried to upgrade myxwiki.org. We get a real life DB and use it instead of using just a clean and pristine DB.
2) When we release a new version, migrate myxwiki.org to the latest version, so users using it can report possible issues due to migration. This is a little hard for me to do since if something goes wrong, I could break a lot of things. Someone with more experience should do it, if there volunteers.
I'm +1 and I actually think it should be done *before* we release as otherwise it has little value. IMO we need to migrate at least when we release RC1 so that we can fix issues for the final.
Now someone need to spend the time to automate the update since it takes quite a while, especially with the cluster.
Note that I've been trying to upgrade myxwiki.org manually for a week now but I cannot since our database migration is just not working ATM. We're working on this.
I am waiting your feedback about this since I want to improve the visibility of testing, and also try to test migrations in a more real world environment
+1 for more real word! :)
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
participants (3)
-
Caleb James DeLisle -
Sorin Burjan -
Vincent Massol