[xwiki-devs] [Proposal] Use liquibase to manage the database schema
While trying to reduce the likelihood of duplicate ids for documents, and extending my patch to provide a proper solution for objects, I fall on a really unexpected issue: the type of the object identifiers are Integer where those of documents are Long. This is completely abnormal since we have several objects per documents, and therefore we need more distinct ids for objects than documents. I have therefore upgraded the ids of objects to use Long, and provide an implementation that use the lowest 64bits of an MD5 key for object in the same way I do for documents. This implementation introduce two new serializer: UidStringEntityReferenceSerializer and LocalUidStringEntityReferenceSerializer. I have also bridged the statistics that derived from objects. The new implementation works perfectly on a new database but... ... I am unable to provide a proper migration procedure, since hibernate cannot manage changing the types of existing columns. It does not complains during schema update, it simply do nothing about them. And later the data migration breaks since my hashes cannot fit in the database properly. After thorough googling, I understand that hibernate schema updates were not made for production use and are really limited opposed to the general idea we had of it. Since I am currently stucked, I propose to move ahead and use a new tool to properly manage our database schema upgrade: http://www.liquibase.org/ Liquibase is Apache licensed and provide a database agnostic version system for migrating databases. It does not works like a diff tool, but more like a patch tool, where you provide several XML description of your wanted changes and it manage to apply or rollback there changes in an ordered manner to upgrade the database to the latest schema. There is several way to apply these changes and I would like to see if it could be integrated in our current migration procedure, or at least as an independent listener. WDYT ? -- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
Hi Denis, Just of curiosity, how did you choose liquibase from the rest? E.g. http://code.google.com/p/dbmigrate/ http://www.dbmaintain.org/overview.html http://code.google.com/p/flyway/ and etc ... What were pros and cons? Regards, Roman -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Denis Gervalle Sent: Thursday, January 26, 2012 21:29 PM To: XWiki Developers Subject: [xwiki-devs] [Proposal] Use liquibase to manage the database schema While trying to reduce the likelihood of duplicate ids for documents, and extending my patch to provide a proper solution for objects, I fall on a really unexpected issue: the type of the object identifiers are Integer where those of documents are Long. This is completely abnormal since we have several objects per documents, and therefore we need more distinct ids for objects than documents. I have therefore upgraded the ids of objects to use Long, and provide an implementation that use the lowest 64bits of an MD5 key for object in the same way I do for documents. This implementation introduce two new serializer: UidStringEntityReferenceSerializer and LocalUidStringEntityReferenceSerializer. I have also bridged the statistics that derived from objects. The new implementation works perfectly on a new database but... ... I am unable to provide a proper migration procedure, since hibernate cannot manage changing the types of existing columns. It does not complains during schema update, it simply do nothing about them. And later the data migration breaks since my hashes cannot fit in the database properly. After thorough googling, I understand that hibernate schema updates were not made for production use and are really limited opposed to the general idea we had of it. Since I am currently stucked, I propose to move ahead and use a new tool to properly manage our database schema upgrade: http://www.liquibase.org/ Liquibase is Apache licensed and provide a database agnostic version system for migrating databases. It does not works like a diff tool, but more like a patch tool, where you provide several XML description of your wanted changes and it manage to apply or rollback there changes in an ordered manner to upgrade the database to the latest schema. There is several way to apply these changes and I would like to see if it could be integrated in our current migration procedure, or at least as an independent listener. WDYT ? -- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Roman, On Thu, Jan 26, 2012 at 21:53, Roman Muntyanu <[email protected]>wrote:
Hi Denis,
Just of curiosity, how did you choose liquibase from the rest? E.g. http://code.google.com/p/dbmigrate/ http://www.dbmaintain.org/overview.html http://code.google.com/p/flyway/ and etc ... What were pros and cons?
Thanks for asking. All of the one you mentioned does not provide the most important IMO: database agnosticism. You may also add Solidbase and AutoPatch to your list. Moreover, Liquibase provide an interesting extension for hibernate, and if we drop the hibernate schema update, we will need it to produce the script to support our older databases. The Liquibase API seems also more fine grained and clearer IMO. I do not want a complete black box and I already have the migration mechanism in place for data. The best will be the tool that integrate with it and provide the same database isolation that hibernate provide for data, else I could simply written the proprietary scripts in my existing system. If you know some that really compete with liquibase, let me know.
Regards, Roman
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Denis Gervalle Sent: Thursday, January 26, 2012 21:29 PM To: XWiki Developers Subject: [xwiki-devs] [Proposal] Use liquibase to manage the database schema
While trying to reduce the likelihood of duplicate ids for documents, and extending my patch to provide a proper solution for objects, I fall on a really unexpected issue: the type of the object identifiers are Integer where those of documents are Long. This is completely abnormal since we have several objects per documents, and therefore we need more distinct ids for objects than documents.
I have therefore upgraded the ids of objects to use Long, and provide an implementation that use the lowest 64bits of an MD5 key for object in the same way I do for documents. This implementation introduce two new serializer: UidStringEntityReferenceSerializer and LocalUidStringEntityReferenceSerializer. I have also bridged the statistics that derived from objects. The new implementation works perfectly on a new database but...
... I am unable to provide a proper migration procedure, since hibernate cannot manage changing the types of existing columns. It does not complains during schema update, it simply do nothing about them. And later the data migration breaks since my hashes cannot fit in the database properly. After thorough googling, I understand that hibernate schema updates were not made for production use and are really limited opposed to the general idea we had of it.
Since I am currently stucked, I propose to move ahead and use a new tool to properly manage our database schema upgrade:
Liquibase is Apache licensed and provide a database agnostic version system for migrating databases. It does not works like a diff tool, but more like a patch tool, where you provide several XML description of your wanted changes and it manage to apply or rollback there changes in an ordered manner to upgrade the database to the latest schema. There is several way to apply these changes and I would like to see if it could be integrated in our current migration procedure, or at least as an independent listener.
WDYT ?
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
Denis, Thanks for reply :) Unfortunately I've never had a chance to adopt DB upgrade software on the project. That's why I was asking - to have a qualified reference for the future cases :) Not to simply take, I'll share one simple and obvious but efficient criterion for choosing a third-party tool (which I've learned by my own bitter experience): Number of active committers >=3 (well, at least >1) and live activity stream with regular releases (at least 1/6 months). -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Denis Gervalle Sent: Friday, January 27, 2012 00:34 AM To: XWiki Developers Subject: Re: [xwiki-devs] [Proposal] Use liquibase to manage the database schema Hi Roman, On Thu, Jan 26, 2012 at 21:53, Roman Muntyanu <[email protected]>wrote:
Hi Denis,
Just of curiosity, how did you choose liquibase from the rest? E.g. http://code.google.com/p/dbmigrate/ http://www.dbmaintain.org/overview.html http://code.google.com/p/flyway/ and etc ... What were pros and cons?
Thanks for asking. All of the one you mentioned does not provide the most important IMO: database agnosticism. You may also add Solidbase and AutoPatch to your list. Moreover, Liquibase provide an interesting extension for hibernate, and if we drop the hibernate schema update, we will need it to produce the script to support our older databases. The Liquibase API seems also more fine grained and clearer IMO. I do not want a complete black box and I already have the migration mechanism in place for data. The best will be the tool that integrate with it and provide the same database isolation that hibernate provide for data, else I could simply written the proprietary scripts in my existing system. If you know some that really compete with liquibase, let me know.
Regards, Roman
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Denis Gervalle Sent: Thursday, January 26, 2012 21:29 PM To: XWiki Developers Subject: [xwiki-devs] [Proposal] Use liquibase to manage the database schema
While trying to reduce the likelihood of duplicate ids for documents, and extending my patch to provide a proper solution for objects, I fall on a really unexpected issue: the type of the object identifiers are Integer where those of documents are Long. This is completely abnormal since we have several objects per documents, and therefore we need more distinct ids for objects than documents.
I have therefore upgraded the ids of objects to use Long, and provide an implementation that use the lowest 64bits of an MD5 key for object in the same way I do for documents. This implementation introduce two new serializer: UidStringEntityReferenceSerializer and LocalUidStringEntityReferenceSerializer. I have also bridged the statistics that derived from objects. The new implementation works perfectly on a new database but...
... I am unable to provide a proper migration procedure, since hibernate cannot manage changing the types of existing columns. It does not complains during schema update, it simply do nothing about them. And later the data migration breaks since my hashes cannot fit in the database properly. After thorough googling, I understand that hibernate schema updates were not made for production use and are really limited opposed to the general idea we had of it.
Since I am currently stucked, I propose to move ahead and use a new tool to properly manage our database schema upgrade:
Liquibase is Apache licensed and provide a database agnostic version system for migrating databases. It does not works like a diff tool, but more like a patch tool, where you provide several XML description of your wanted changes and it manage to apply or rollback there changes in an ordered manner to upgrade the database to the latest schema. There is several way to apply these changes and I would like to see if it could be integrated in our current migration procedure, or at least as an independent listener.
WDYT ?
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On 01/26/2012 02:29 PM, Denis Gervalle wrote:
While trying to reduce the likelihood of duplicate ids for documents, and extending my patch to provide a proper solution for objects, I fall on a really unexpected issue: the type of the object identifiers are Integer where those of documents are Long. This is completely abnormal since we have several objects per documents, and therefore we need more distinct ids for objects than documents.
I have therefore upgraded the ids of objects to use Long, and provide an implementation that use the lowest 64bits of an MD5 key for object in the same way I do for documents. This implementation introduce two new serializer: UidStringEntityReferenceSerializer and LocalUidStringEntityReferenceSerializer. I have also bridged the statistics that derived from objects. The new implementation works perfectly on a new database but...
... I am unable to provide a proper migration procedure, since hibernate cannot manage changing the types of existing columns. It does not complains during schema update, it simply do nothing about them. And later the data migration breaks since my hashes cannot fit in the database properly. After thorough googling, I understand that hibernate schema updates were not made for production use and are really limited opposed to the general idea we had of it.
Since I am currently stucked, I propose to move ahead and use a new tool to properly manage our database schema upgrade:
Liquibase is Apache licensed and provide a database agnostic version system for migrating databases. It does not works like a diff tool, but more like a patch tool, where you provide several XML description of your wanted changes and it manage to apply or rollback there changes in an ordered manner to upgrade the database to the latest schema. There is several way to apply these changes and I would like to see if it could be integrated in our current migration procedure, or at least as an independent listener.
WDYT ?
+1, sounds good. -- Sergiu Dumitriu http://purl.org/net/sergiu/
participants (3)
-
Denis Gervalle -
Roman Muntyanu -
Sergiu Dumitriu