[xwiki-devs] [Proposal] Fix many inconsistencies in schema updates and database migrations

Hi devs, After some analysis of XWiki startup, there is many issues in the way DB schema is updated and migrators are executed (see XWIKI-1859, XWIKI-2075 and XWIKI-2066). There is no consistency between them, which really seems not appropriate. The schema is updated in many different places (unless disabled by config): - when UpdateDatabase is called for the first time; this one being called anytime we access a sub-wiki in a farm - when CheckHibernate is called and no session factory exists, this one being called in many places and before any transaction based requests - before a migration of a sub-wiki db The update schema function itself as some issue as well: - it is a synchronized function and the config check is done after synchronization - a part from the schema update done by the generated hibernate script, it does some updates on data in the database like a migrator, but it does so even when there is no need for an updated oppposed to a migrator - these updates are written in plain SQL, not HQL, and partly MySQL oriented. - it use a different parameter than migrator in xwiki.cfg to be disabled, so it is possible to have migration without having the correct schema On the other hand, migrator are launch only on initial startup opposed to the lazy updating of schema. Since we use struts, it was no easy to have them launch before the first request, but I do see this as an issue since any admin will at least do such a request, and a request would not be accepted earlier anyway. The issue with migrators are: - migrators are executed on new XWiki database. In particular, when you create a new wiki in a farm, no version are set in the DB, and the next time the servlet container is restart, all migrator are executed on this new wiki database. - migrators may fail and the wiki start anyway (I have already a patch for that) - migrators could be disabled, but there is no guarantee that a database is at the correct version for the running core. This is particularly annoying for my migration, since old id could be kept, and document loading will fails for mandatory document, so these will be recreated, producing many duplicated document in the database, and therefore corrupting all accessed wikis that have not been migrated. So, I proposed: 1) to join together schema update and migration, using xwiki.store.migration (default to 0) && xwiki.store.hibernate.updateschema (deprecated, default to true already) && xwiki.store.migration.databases (default to main wiki only) for enabling them 2) convert migration like requests done in schema update into an very old migrator (before first one) 3) While still allowing migrators (and therefore schema update) to be disabled, keep a lazy check on the database version required by the running core. a) fails to return a given wiki if its database is outdated b) OR do a lazy update if migration is enabled 4) store the current DB version of the core in the database when creating new wikis 5) consider a DB without version and without xwikidoc table to be empty and at the current version For this to be implemented, I need to know the current core DB version at startup. I see some ways to do that: A) store it in a constant in a static in XWiki class, putting the responsibility on migration developer to update it B) compute this value by instantiating all migrators to know there resulting DB version a) and store it in the XWiki singleton b) and cache this in the migration manager, keeping it available from the current XWiki singleton, for potential lazy migration I am +1 for most of these, except 3)b) for which I am not sure it is fully safe, and which implies B)b). I am undecided regard A) or B)a), with a preference for the second, but wondering if it should consider xwiki.store.migration.ignored or not (probably not). WDYT ? -- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO