Re: [xwiki-devs] [Proposal] Our schema migration process is broken, we need to change it before 5.0

Hi devs, I have filed today a blocker issue: http://jira.xwiki.org/browse/XWIKI-9041 This serious issue is the result of our current schema migration process. Currently, managing schema changes is done at two level: 1) A automatic script is computed by hibernate, comparing the current DB schema, and the targeted schema extracted from mapping files. 2) DataMigration may change schema using liquibase (git like solution for managing schema changes) According to the Hibernate team, 1) should only be used for development purpose. The Hibernate team clearly discourage usage of 1) in production, since it does not cover every use cases. For example, it does not: - drop column, table, etc... - change datatype - add constrainst ... However, this was the first, and until 4.x, the only way we had to manage our database schema. This is why we have introduced 2) in 4.0, to be able to change more information in our schema. Since 4.3, 2) was even be splitted in two steps, one before and one after step 1) for partially solving similar issue to http://jira.xwiki.org/browse/XWIKI-9041. While 1) is an uncontrollable monolithic step, 2) is like the underlying data migration, a chronologic process, that apply change in order. These two methods does not really work well join together, since parts of step 1) may need some part of step 2) to be done, and the reverse. For example, migration of IDs from 32bits to 64bits bases its schema changes on the hibernate mapping, and need the schema to be updated by hibernate first. A new table like xwikistringlists introduced in 5.0M2 implies the creation of new constraints against existing IDs, and therefore need the change of datatype to be processed first. There is no way out of that situation, and this is the cause of http://jira.xwiki.org/browse/XWIKI-9041. Since we do not do a lot of schema updates during a release cycle, the current situation has been manageable during the 4.x cycle. But, to avoid XWIKI-9041, it implies that user must not skip a major release during migration. This is a possible option, but it may implies some refactoring of existing migration inside a given release cycle, and probably some limitations as well. After brainstorming about the situation with Thomas, we have concluded that it would be better to get rid of step 1), to control precisely the evolution of the schema. However, we do not want to cause a breakage for existing users. So here what we proposed: A) For any existing database having a version lower than 50000, apply the automatic update of schema using hibernate, but based on the mapping file of the latest 4.5.x release, and followed by B) B) Any changes in mapping file done in 5.x or later (implying schema changes) should be backed by a corresponding DataMigration that provide Liquibase changelogs to reflect those changes on the database.

C) Newly created database are setup using the latest mapping file by hibernate (since we manage versioning of our database, old liquibase change logs will never be applied to these new db) D) Advanced user will be allowed to also provide liquibase file, to manage their schema updated properly (existing hidden feature), but... E) The automatic hibernate schema update script is computed after the above steps to check that it is empty, and that the DB schema is in sync with the mapping file. Optionally we may allow this step to apply more schema updates for advanced users (at their own risk) that do not provide the liquibase change log required. (keeping the existing internal custom mapping feature, as well as the logic of our dynamic mapping). The pros: - New database will no more be updated blindly, and liquibase will provide a complete history of changes (except advanced user changes and dynamic mapping) - Existing database will start to have a complete history of schema changes starting from 5.x (and a somewhat partial history for 4.x versions) - Migration test will only be needed from pre-5.x to 5.x version, future migration will be more isolated, and can be tested more locally between the release introducing the migration and the previous one - We optionally keep existing functionalities (step E) The cons: - More work to be done while changing mapping file - Advanced user may need also to be more careful about their change in .hbm file - We keep using our versioning solution (we do not really use the liquibase way), maybe a advantage as well - We need some time to put that in place and we have already a slipping RC1 However, the only alternative I see to fix http://jira.xwiki.org/browse/XWIKI-9041 would be to revert the schema changes until the above, or a similar solution is available. The bad news now. Once again, I have not planned to work on this, and my planning is already really busy. It will also require to be thoroughly tested, and Sorin will not say the opposite, migration test is time consuming. So, this will not improve our schedule to release 5.0 and I will need serious support from you guys to put that in place. WDTH ?