[xwiki-devs] [Proposal] Our schema migration process is broken, we need to change it before 5.0
Hi devs, I have filed today a blocker issue: http://jira.xwiki.org/browse/XWIKI-9041 This serious issue is the result of our current schema migration process. Currently, managing schema changes is done at two level: 1) A automatic script is computed by hibernate, comparing the current DB schema, and the targeted schema extracted from mapping files. 2) DataMigration may change schema using liquibase (git like solution for managing schema changes) According to the Hibernate team, 1) should only be used for development purpose. The Hibernate team clearly discourage usage of 1) in production, since it does not cover every use cases. For example, it does not: - drop column, table, etc... - change datatype - add constrainst ... However, this was the first, and until 4.x, the only way we had to manage our database schema. This is why we have introduced 2) in 4.0, to be able to change more information in our schema. Since 4.3, 2) was even be splitted in two steps, one before and one after step 1) for partially solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041. While 1) is an uncontrollable monolithic step, 2) is like the underlying data migration, a chronologic process, that apply change in order. These two methods does not really work well join together, since parts of step 1) may need some part of step 2) to be done, and the reverse. For example, migration of IDs from 32bits to 64bits bases its schema changes on the hibernate mapping, and need the schema to be updated by hibernate first. A new table like xwikistringlists introduced in 5.0M2 implies the creation of new constraints against existing IDs, and therefore need the change of datatype to be processed first. There is no way out of that situation, and this is the cause of http://jira.xwiki.org/browse/XWIKI-9041. Since we do not do a lot of schema updates during a release cycle, the current situation has been manageable during the 4.x cycle. But, to avoid XWIKI-9041, it implies that user must not skip a major release during migration. This is a possible option, but it may implies some refactoring of existing migration inside a given release cycle, and probably some limitations as well. After brainstorming about the situation with Thomas, we have concluded that it would be better to get rid of step 1), to control precisely the evolution of the schema. However, we do not want to cause a breakage for existing users. So here what we proposed: A) For any existing database having a version lower than 50000, apply the automatic update of schema using hibernate, but based on the mapping file of the latest 4.5.x release, and followed by B) B) Any changes in mapping file done in 5.x or later (implying schema changes) should be backed by a corresponding DataMigration that provide Liquibase changelogs to reflect those changes on the database. C) Newly created database are setup using the latest mapping file by hibernate (since we manage versioning of our database, old liquibase change logs will never be applied to these new db) D) Advanced user will be allowed to also provide liquibase file, to manage their schema updated properly (existing hidden feature), but... E) The automatic hibernate schema update script is computed after the above steps to check that it is empty, and that the DB schema is in sync with the mapping file. Optionally we may allow this step to apply more schema updates for advanced users (at their own risk) that do not provide the liquibase change log required. (keeping the existing internal custom mapping feature, as well as the logic of our dynamic mapping). The pros: - New database will no more be updated blindly, and liquibase will provide a complete history of changes (except advanced user changes and dynamic mapping) - Existing database will start to have a complete history of schema changes starting from 5.x (and a somewhat partial history for 4.x versions) - Migration test will only be needed from pre-5.x to 5.x version, future migration will be more isolated, and can be tested more locally between the release introducing the migration and the previous one - We optionally keep existing functionalities (step E) The cons: - More work to be done while changing mapping file - Advanced user may need also to be more careful about their change in .hbm file - We keep using our versioning solution (we do not really use the liquibase way), maybe a advantage as well - We need some time to put that in place and we have already a slipping RC1 However, the only alternative I see to fix http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema changes until the above, or a similar solution is available. The bad news now. Once again, I have not planned to work on this, and my planning is already really busy. It will also require to be thoroughly tested, and Sorin will not say the opposite, migration test is time consuming. So, this will not improve our schedule to release 5.0 and I will need serious support from you guys to put that in place. WDTH ? -- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
+1 On Tue, Apr 16, 2013 at 8:36 PM, Denis Gervalle <[email protected]> wrote:
Hi devs,
I have filed today a blocker issue: http://jira.xwiki.org/browse/XWIKI-9041 This serious issue is the result of our current schema migration process.
Currently, managing schema changes is done at two level:
1) A automatic script is computed by hibernate, comparing the current DB schema, and the targeted schema extracted from mapping files. 2) DataMigration may change schema using liquibase (git like solution for managing schema changes)
According to the Hibernate team, 1) should only be used for development purpose. The Hibernate team clearly discourage usage of 1) in production, since it does not cover every use cases. For example, it does not: - drop column, table, etc... - change datatype - add constrainst ... However, this was the first, and until 4.x, the only way we had to manage our database schema. This is why we have introduced 2) in 4.0, to be able to change more information in our schema. Since 4.3, 2) was even be splitted in two steps, one before and one after step 1) for partially solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041.
While 1) is an uncontrollable monolithic step, 2) is like the underlying data migration, a chronologic process, that apply change in order. These two methods does not really work well join together, since parts of step 1) may need some part of step 2) to be done, and the reverse.
For example, migration of IDs from 32bits to 64bits bases its schema changes on the hibernate mapping, and need the schema to be updated by hibernate first. A new table like xwikistringlists introduced in 5.0M2 implies the creation of new constraints against existing IDs, and therefore need the change of datatype to be processed first. There is no way out of that situation, and this is the cause of http://jira.xwiki.org/browse/XWIKI-9041.
Since we do not do a lot of schema updates during a release cycle, the current situation has been manageable during the 4.x cycle. But, to avoid XWIKI-9041, it implies that user must not skip a major release during migration. This is a possible option, but it may implies some refactoring of existing migration inside a given release cycle, and probably some limitations as well.
After brainstorming about the situation with Thomas, we have concluded that it would be better to get rid of step 1), to control precisely the evolution of the schema. However, we do not want to cause a breakage for existing users. So here what we proposed:
A) For any existing database having a version lower than 50000, apply the automatic update of schema using hibernate, but based on the mapping file of the latest 4.5.x release, and followed by B) B) Any changes in mapping file done in 5.x or later (implying schema changes) should be backed by a corresponding DataMigration that provide Liquibase changelogs to reflect those changes on the database. C) Newly created database are setup using the latest mapping file by hibernate (since we manage versioning of our database, old liquibase change logs will never be applied to these new db) D) Advanced user will be allowed to also provide liquibase file, to manage their schema updated properly (existing hidden feature), but... E) The automatic hibernate schema update script is computed after the above steps to check that it is empty, and that the DB schema is in sync with the mapping file. Optionally we may allow this step to apply more schema updates for advanced users (at their own risk) that do not provide the liquibase change log required. (keeping the existing internal custom mapping feature, as well as the logic of our dynamic mapping).
The pros: - New database will no more be updated blindly, and liquibase will provide a complete history of changes (except advanced user changes and dynamic mapping) - Existing database will start to have a complete history of schema changes starting from 5.x (and a somewhat partial history for 4.x versions) - Migration test will only be needed from pre-5.x to 5.x version, future migration will be more isolated, and can be tested more locally between the release introducing the migration and the previous one - We optionally keep existing functionalities (step E)
The cons: - More work to be done while changing mapping file - Advanced user may need also to be more careful about their change in .hbm file - We keep using our versioning solution (we do not really use the liquibase way), maybe a advantage as well - We need some time to put that in place and we have already a slipping RC1
However, the only alternative I see to fix http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema changes until the above, or a similar solution is available.
The bad news now. Once again, I have not planned to work on this, and my planning is already really busy. It will also require to be thoroughly tested, and Sorin will not say the opposite, migration test is time consuming. So, this will not improve our schedule to release 5.0 and I will need serious support from you guys to put that in place.
WDTH ?
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
Keeping this thread to discuss details of the implementation as explain by Denis and move 5.0 related decision to http://markmail.org/message/xyr5ujxcz4rfdaug. On Wed, Apr 17, 2013 at 9:06 AM, Thomas Mortagne <[email protected]> wrote:
+1
On Tue, Apr 16, 2013 at 8:36 PM, Denis Gervalle <[email protected]> wrote:
Hi devs,
I have filed today a blocker issue: http://jira.xwiki.org/browse/XWIKI-9041 This serious issue is the result of our current schema migration process.
Currently, managing schema changes is done at two level:
1) A automatic script is computed by hibernate, comparing the current DB schema, and the targeted schema extracted from mapping files. 2) DataMigration may change schema using liquibase (git like solution for managing schema changes)
According to the Hibernate team, 1) should only be used for development purpose. The Hibernate team clearly discourage usage of 1) in production, since it does not cover every use cases. For example, it does not: - drop column, table, etc... - change datatype - add constrainst ... However, this was the first, and until 4.x, the only way we had to manage our database schema. This is why we have introduced 2) in 4.0, to be able to change more information in our schema. Since 4.3, 2) was even be splitted in two steps, one before and one after step 1) for partially solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041.
While 1) is an uncontrollable monolithic step, 2) is like the underlying data migration, a chronologic process, that apply change in order. These two methods does not really work well join together, since parts of step 1) may need some part of step 2) to be done, and the reverse.
For example, migration of IDs from 32bits to 64bits bases its schema changes on the hibernate mapping, and need the schema to be updated by hibernate first. A new table like xwikistringlists introduced in 5.0M2 implies the creation of new constraints against existing IDs, and therefore need the change of datatype to be processed first. There is no way out of that situation, and this is the cause of http://jira.xwiki.org/browse/XWIKI-9041.
Since we do not do a lot of schema updates during a release cycle, the current situation has been manageable during the 4.x cycle. But, to avoid XWIKI-9041, it implies that user must not skip a major release during migration. This is a possible option, but it may implies some refactoring of existing migration inside a given release cycle, and probably some limitations as well.
After brainstorming about the situation with Thomas, we have concluded that it would be better to get rid of step 1), to control precisely the evolution of the schema. However, we do not want to cause a breakage for existing users. So here what we proposed:
A) For any existing database having a version lower than 50000, apply the automatic update of schema using hibernate, but based on the mapping file of the latest 4.5.x release, and followed by B) B) Any changes in mapping file done in 5.x or later (implying schema changes) should be backed by a corresponding DataMigration that provide Liquibase changelogs to reflect those changes on the database. C) Newly created database are setup using the latest mapping file by hibernate (since we manage versioning of our database, old liquibase change logs will never be applied to these new db) D) Advanced user will be allowed to also provide liquibase file, to manage their schema updated properly (existing hidden feature), but... E) The automatic hibernate schema update script is computed after the above steps to check that it is empty, and that the DB schema is in sync with the mapping file. Optionally we may allow this step to apply more schema updates for advanced users (at their own risk) that do not provide the liquibase change log required. (keeping the existing internal custom mapping feature, as well as the logic of our dynamic mapping).
The pros: - New database will no more be updated blindly, and liquibase will provide a complete history of changes (except advanced user changes and dynamic mapping) - Existing database will start to have a complete history of schema changes starting from 5.x (and a somewhat partial history for 4.x versions) - Migration test will only be needed from pre-5.x to 5.x version, future migration will be more isolated, and can be tested more locally between the release introducing the migration and the previous one - We optionally keep existing functionalities (step E)
The cons: - More work to be done while changing mapping file - Advanced user may need also to be more careful about their change in .hbm file - We keep using our versioning solution (we do not really use the liquibase way), maybe a advantage as well - We need some time to put that in place and we have already a slipping RC1
However, the only alternative I see to fix http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema changes until the above, or a similar solution is available.
The bad news now. Once again, I have not planned to work on this, and my planning is already really busy. It will also require to be thoroughly tested, and Sorin will not say the opposite, migration test is time consuming. So, this will not improve our schedule to release 5.0 and I will need serious support from you guys to put that in place.
WDTH ?
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
-- Thomas Mortagne
Hi Denis, On Apr 16, 2013, at 8:36 PM, Denis Gervalle <[email protected]> wrote:
Hi devs,
I have filed today a blocker issue: http://jira.xwiki.org/browse/XWIKI-9041 This serious issue is the result of our current schema migration process.
Currently, managing schema changes is done at two level:
1) A automatic script is computed by hibernate, comparing the current DB schema, and the targeted schema extracted from mapping files. 2) DataMigration may change schema using liquibase (git like solution for managing schema changes)
According to the Hibernate team, 1) should only be used for development purpose. The Hibernate team clearly discourage usage of 1) in production, since it does not cover every use cases. For example, it does not: - drop column, table, etc... - change datatype - add constrainst ... However, this was the first, and until 4.x, the only way we had to manage our database schema. This is why we have introduced 2) in 4.0, to be able to change more information in our schema. Since 4.3, 2) was even be splitted in two steps, one before and one after step 1) for partially solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041.
While 1) is an uncontrollable monolithic step, 2) is like the underlying data migration, a chronologic process, that apply change in order. These two methods does not really work well join together, since parts of step 1) may need some part of step 2) to be done, and the reverse.
For example, migration of IDs from 32bits to 64bits bases its schema changes on the hibernate mapping, and need the schema to be updated by hibernate first. A new table like xwikistringlists introduced in 5.0M2 implies the creation of new constraints against existing IDs, and therefore need the change of datatype to be processed first. There is no way out of that situation, and this is the cause of http://jira.xwiki.org/browse/XWIKI-9041.
Since we do not do a lot of schema updates during a release cycle, the current situation has been manageable during the 4.x cycle. But, to avoid XWIKI-9041, it implies that user must not skip a major release during migration. This is a possible option, but it may implies some refactoring of existing migration inside a given release cycle, and probably some limitations as well.
After brainstorming about the situation with Thomas, we have concluded that it would be better to get rid of step 1), to control precisely the evolution of the schema. However, we do not want to cause a breakage for existing users. So here what we proposed:
A) For any existing database having a version lower than 50000, apply the automatic update of schema using hibernate, but based on the mapping file of the latest 4.5.x release, and followed by B) B) Any changes in mapping file done in 5.x or later (implying schema changes) should be backed by a corresponding DataMigration that provide Liquibase changelogs to reflect those changes on the database.
Actually the nice point of 1) was that Hibernate was taking care of differences between databases whereas if we have to write a migration we have to do that ourselves manually. If we are to create migrations for any schema change it would be nice to make it as simple as possible to write a migration. I personally haven't written one and wouldn't know how to write a good one that will work on all DBs. At the very minimum we would need a migration writing guide published on xwiki.org.
C) Newly created database are setup using the latest mapping file by hibernate (since we manage versioning of our database, old liquibase change logs will never be applied to these new db) D) Advanced user will be allowed to also provide liquibase file, to manage their schema updated properly (existing hidden feature), but... E) The automatic hibernate schema update script is computed after the above steps to check that it is empty, and that the DB schema is in sync with the mapping file. Optionally we may allow this step to apply more schema updates for advanced users (at their own risk) that do not provide the liquibase change log required. (keeping the existing internal custom mapping feature, as well as the logic of our dynamic mapping).
The pros: - New database will no more be updated blindly, and liquibase will provide a complete history of changes (except advanced user changes and dynamic mapping) - Existing database will start to have a complete history of schema changes starting from 5.x (and a somewhat partial history for 4.x versions) - Migration test will only be needed from pre-5.x to 5.x version, future migration will be more isolated, and can be tested more locally between the release introducing the migration and the previous one - We optionally keep existing functionalities (step E)
The cons: - More work to be done while changing mapping file - Advanced user may need also to be more careful about their change in .hbm file - We keep using our versioning solution (we do not really use the liquibase way), maybe a advantage as well - We need some time to put that in place and we have already a slipping RC1
However, the only alternative I see to fix http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema changes until the above, or a similar solution is available.
The bad news now. Once again, I have not planned to work on this, and my planning is already really busy. It will also require to be thoroughly tested, and Sorin will not say the opposite, migration test is time consuming. So, this will not improve our schedule to release 5.0 and I will need serious support from you guys to put that in place.
WDTH ?
I'll trust Thomas and you on the details because I'm not familiar with migrations. My only comment is that we need to release 5.0 without this big change since we're already very late. AFAIU we could put in the Release notes of 5.0 that users upgrading from before 4.x will need first to upgrade to 4.5.x and then only upgrade to 5.0 and that this will be fixed in 5.1. Correct? That said, after more thinking, when you talk about "reverting schema changes" what does it entail? Is it just reverting Sergiu's change that adds the new table or does it also include reverting my change for the oracle hbm file where I've changed from not-null="true" to not-null="false" in 2 places? If only Sergiu's change is impacted I'd say we should revert it till 5.1 since it's not an important change for 5.0 IMO. Thanks -Vincent
On Wed, Apr 17, 2013 at 9:07 AM, Vincent Massol <[email protected]> wrote:
Hi Denis,
On Apr 16, 2013, at 8:36 PM, Denis Gervalle <[email protected]> wrote:
Hi devs,
I have filed today a blocker issue: http://jira.xwiki.org/browse/XWIKI-9041 This serious issue is the result of our current schema migration process.
Currently, managing schema changes is done at two level:
1) A automatic script is computed by hibernate, comparing the current DB schema, and the targeted schema extracted from mapping files. 2) DataMigration may change schema using liquibase (git like solution for managing schema changes)
According to the Hibernate team, 1) should only be used for development purpose. The Hibernate team clearly discourage usage of 1) in production, since it does not cover every use cases. For example, it does not: - drop column, table, etc... - change datatype - add constrainst ... However, this was the first, and until 4.x, the only way we had to manage our database schema. This is why we have introduced 2) in 4.0, to be able to change more information in our schema. Since 4.3, 2) was even be splitted in two steps, one before and one after step 1) for partially solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041.
While 1) is an uncontrollable monolithic step, 2) is like the underlying data migration, a chronologic process, that apply change in order. These two methods does not really work well join together, since parts of step 1) may need some part of step 2) to be done, and the reverse.
For example, migration of IDs from 32bits to 64bits bases its schema changes on the hibernate mapping, and need the schema to be updated by hibernate first. A new table like xwikistringlists introduced in 5.0M2 implies the creation of new constraints against existing IDs, and therefore need the change of datatype to be processed first. There is no way out of that situation, and this is the cause of http://jira.xwiki.org/browse/XWIKI-9041.
Since we do not do a lot of schema updates during a release cycle, the current situation has been manageable during the 4.x cycle. But, to avoid XWIKI-9041, it implies that user must not skip a major release during migration. This is a possible option, but it may implies some refactoring of existing migration inside a given release cycle, and probably some limitations as well.
After brainstorming about the situation with Thomas, we have concluded that it would be better to get rid of step 1), to control precisely the evolution of the schema. However, we do not want to cause a breakage for existing users. So here what we proposed:
A) For any existing database having a version lower than 50000, apply the automatic update of schema using hibernate, but based on the mapping file of the latest 4.5.x release, and followed by B) B) Any changes in mapping file done in 5.x or later (implying schema changes) should be backed by a corresponding DataMigration that provide Liquibase changelogs to reflect those changes on the database.
Actually the nice point of 1) was that Hibernate was taking care of differences between databases whereas if we have to write a migration we have to do that ourselves manually.
If we are to create migrations for any schema change it would be nice to make it as simple as possible to write a migration. I personally haven't written one and wouldn't know how to write a good one that will work on all DBs. At the very minimum we would need a migration writing guide published on xwiki.org.
C) Newly created database are setup using the latest mapping file by hibernate (since we manage versioning of our database, old liquibase change logs will never be applied to these new db) D) Advanced user will be allowed to also provide liquibase file, to manage their schema updated properly (existing hidden feature), but... E) The automatic hibernate schema update script is computed after the above steps to check that it is empty, and that the DB schema is in sync with the mapping file. Optionally we may allow this step to apply more schema updates for advanced users (at their own risk) that do not provide the liquibase change log required. (keeping the existing internal custom mapping feature, as well as the logic of our dynamic mapping).
The pros: - New database will no more be updated blindly, and liquibase will provide a complete history of changes (except advanced user changes and dynamic mapping) - Existing database will start to have a complete history of schema changes starting from 5.x (and a somewhat partial history for 4.x versions) - Migration test will only be needed from pre-5.x to 5.x version, future migration will be more isolated, and can be tested more locally between the release introducing the migration and the previous one - We optionally keep existing functionalities (step E)
The cons: - More work to be done while changing mapping file - Advanced user may need also to be more careful about their change in .hbm file - We keep using our versioning solution (we do not really use the liquibase way), maybe a advantage as well - We need some time to put that in place and we have already a slipping RC1
However, the only alternative I see to fix http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema changes until the above, or a similar solution is available.
The bad news now. Once again, I have not planned to work on this, and my planning is already really busy. It will also require to be thoroughly tested, and Sorin will not say the opposite, migration test is time consuming. So, this will not improve our schedule to release 5.0 and I will need serious support from you guys to put that in place.
WDTH ?
I'll trust Thomas and you on the details because I'm not familiar with migrations.
My only comment is that we need to release 5.0 without this big change since we're already very late.
AFAIU we could put in the Release notes of 5.0 that users upgrading from before 4.x will need first to upgrade to 4.5.x and then only upgrade to 5.0 and that this will be fixed in 5.1. Correct?
That said, after more thinking, when you talk about "reverting schema changes" what does it entail? Is it just reverting Sergiu's change that adds the new table or does it also include reverting my change for the oracle hbm file where I've changed from not-null="true" to not-null="false" in 2 places? If only Sergiu's change is impacted I'd say we should revert it till 5.1 since it's not an important change for 5.0 IMO.
1) You applied your changed to 4.5 too so it should be ok (the base hbm would be the last 4.5.x one in Denis proposal) 2) AFAIK hibernate does not know how to migration your change so you have to add a migration (nothing to do with this thread but your made me think about it)
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
On Wed, Apr 17, 2013 at 9:07 AM, Vincent Massol <[email protected]> wrote:
Hi Denis,
On Apr 16, 2013, at 8:36 PM, Denis Gervalle <[email protected]> wrote:
may need some part of step 2) to be done, and the reverse.
For example, migration of IDs from 32bits to 64bits bases its schema changes on the hibernate mapping, and need the schema to be updated by hibernate first. A new table like xwikistringlists introduced in 5.0M2 implies the creation of new constraints against existing IDs, and therefore need the change of datatype to be processed first. There is no way out of that situation, and this is the cause of http://jira.xwiki.org/browse/XWIKI-9041.
Since we do not do a lot of schema updates during a release cycle, the current situation has been manageable during the 4.x cycle. But, to avoid XWIKI-9041, it implies that user must not skip a major release during migration. This is a possible option, but it may implies some refactoring of existing migration inside a given release cycle, and probably some limitations as well.
After brainstorming about the situation with Thomas, we have concluded that it would be better to get rid of step 1), to control precisely the evolution of the schema. However, we do not want to cause a breakage for existing users. So here what we proposed:
A) For any existing database having a version lower than 50000, apply the automatic update of schema using hibernate, but based on the mapping file of the latest 4.5.x release, and followed by B) B) Any changes in mapping file done in 5.x or later (implying schema changes) should be backed by a corresponding DataMigration that provide Liquibase changelogs to reflect those changes on the database.
Hi devs,
I have filed today a blocker issue: http://jira.xwiki.org/browse/XWIKI-9041 This serious issue is the result of our current schema migration process.
Currently, managing schema changes is done at two level:
1) A automatic script is computed by hibernate, comparing the current DB schema, and the targeted schema extracted from mapping files. 2) DataMigration may change schema using liquibase (git like solution for managing schema changes)
According to the Hibernate team, 1) should only be used for development purpose. The Hibernate team clearly discourage usage of 1) in production, since it does not cover every use cases. For example, it does not: - drop column, table, etc... - change datatype - add constrainst ... However, this was the first, and until 4.x, the only way we had to manage our database schema. This is why we have introduced 2) in 4.0, to be able to change more information in our schema. Since 4.3, 2) was even be splitted in two steps, one before and one after step 1) for partially solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041.
While 1) is an uncontrollable monolithic step, 2) is like the underlying data migration, a chronologic process, that apply change in order. These two methods does not really work well join together, since parts of step
Actually the nice point of 1) was that Hibernate was taking care of differences between databases whereas if we have to write a migration we have to do that ourselves manually.
Not so nice, since hibernate do very little things. For example, your change is not applied. You will need a migration for it anyway.
If we are to create migrations for any schema change it would be nice to make it as simple as possible to write a migration. I personally haven't written one and wouldn't know how to write a good one that will work on all DBs. At the very minimum we would need a migration writing guide published on xwiki.org.
Creating new migration is as simple as writing a component, and in regards to schema, using Liquibase xml. I agree that some documentation would be nice to have. I will try to write some asap.
C) Newly created database are setup using the latest mapping file by hibernate (since we manage versioning of our database, old liquibase change logs will never be applied to these new db) D) Advanced user will be allowed to also provide liquibase file, to manage their schema updated properly (existing hidden feature), but... E) The automatic hibernate schema update script is computed after the above steps to check that it is empty, and that the DB schema is in sync with the mapping file. Optionally we may allow this step to apply more schema updates for advanced users (at their own risk) that do not provide the liquibase change log required. (keeping the existing internal custom mapping feature, as well as the logic of our dynamic mapping).
The pros: - New database will no more be updated blindly, and liquibase will provide a complete history of changes (except advanced user changes and dynamic mapping) - Existing database will start to have a complete history of schema changes starting from 5.x (and a somewhat partial history for 4.x versions) - Migration test will only be needed from pre-5.x to 5.x version, future migration will be more isolated, and can be tested more locally between the release introducing the migration and the previous one - We optionally keep existing functionalities (step E)
The cons: - More work to be done while changing mapping file - Advanced user may need also to be more careful about their change in .hbm file - We keep using our versioning solution (we do not really use the liquibase way), maybe a advantage as well - We need some time to put that in place and we have already a slipping RC1
However, the only alternative I see to fix http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema changes until the above, or a similar solution is available.
The bad news now. Once again, I have not planned to work on this, and my planning is already really busy. It will also require to be thoroughly tested, and Sorin will not say the opposite, migration test is time consuming. So, this will not improve our schedule to release 5.0 and I will need serious support from you guys to put that in place.
WDTH ?
I'll trust Thomas and you on the details because I'm not familiar with migrations.
My only comment is that we need to release 5.0 without this big change since we're already very late.
I agree.
AFAIU we could put in the Release notes of 5.0 that users upgrading from before 4.x will need first to upgrade to 4.5.x and then only upgrade to 5.0 and that this will be fixed in 5.1. Correct?
No, if we do so, we need to postpone our proposal. This means using hbm from 5.0 and starting the new solution with 5.1.
That said, after more thinking, when you talk about "reverting schema changes" what does it entail? Is it just reverting Sergiu's change that adds the new table or does it also include reverting my change for the oracle hbm file where I've changed from not-null="true" to not-null="false" in 2 places? If only Sergiu's change is impacted I'd say we should revert it till 5.1 since it's not an important change for 5.0 IMO.
Only Sergiu's change is impacted, since your changes is done in 4.5.x and not in 5.0. This is also this change that we cannot get working properly with the current solution.
Thanks -Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
participants (3)
-
Denis Gervalle -
Thomas Mortagne -
Vincent Massol