[xwiki-devs] [VOTE] Change all IDs stored in the database to reduce the likelihood of duplicate id
Hi devs, I know this subject will seems to you already voted and discussed in http://xwiki.markmail.org/thread/fsd25bvft74xwgcx But following the remarks and the discussion under that thread, I had largely improved the proposed changes. This is an important matter, so I prefer to resume here to be sure we all really agree on this. To resume the current situation, we have: - document id - used in document table, rcs, attachment... - simple 32bits string hashcode of a locally serialized document reference, including the language for translated documents - stored in a 64bits field. - object id - used in object and property tables, but also in statistics tables - simple 32bits string hashcode of the concatenation of the document reference, the class reference and the object number - stored in a 32bits field (except in Oracle, where the mapping is 32bits, but the storage is larger) The vote is about: - document id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the document reference - the result is appended with the current locale for translated document, until locale are integrated in references - format for original document: 5space8document - format for translation: 5:space8:document2:fr - object id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the BaseObjectReference - current format would be: 5:space8:document12:xspace.class[0] - if my proposal in the object reference thread is adopted: 5:space8:document18:6:xspace5:class[0] Since changing document id could really not helps since document reference are used in object ids and therefore unambiguous document could receive ambiguous objects, I do not advice to split the change. Moreover, this is really sensible change in the database, so not multiplying them is better. I think the upcoming 4 is a really good time to introduce this change, so I propose to introduce this in version 40000 of the database (4.0M1 release). But I would like to use it internally earlier. So you would be pleased to settle on this thread and the previous one before. It implies the following migration for existing instances: - customer custom mapping have to be adapted before the migration, including dynamic one which could be not so easy, but this is already rarely used and very rarely require any change in fact. - change XWikiDocument to provide the key required for IDs, by the way, also use that key (non local version) for the document cache - refactor the BaseElement hierarchy to provide long IDs (no more integer) based on references (generic way to have ids for any element) - change the hibernate mapping for all object ids - provide dynamic schema updates using liquibase to fix all object id types, including those in custom mapping and collection - migrate in HQL document id for persisted class: XWikiDocument, XWikiRCSNodeInfo, XWikiLink, XWikiAttachment, DeletedAttachment - migrate in HQL object id for persisted class: BaseObject, *Property, internal custom mapped class, dynamic custom mapped class - migrate in HQL object id for custom statistics class derived form XWikiStats - migrate in SQL ids for all relational collections in the above migrated tables To provide this migration as safely as possible: - Liquibase provide a safe way to change the schema - All id conversion are gathered from the database in a first single read-only transaction, and new id are computed. - Potentially already migrated ids are detected, allowing the process to fails and be restarted. - Proceed to ids replacement using a safe algorithm that may support non-circular conflict between old and new ids (very unlikely anyway, since we move from 32 to 64bits) - Use a single transaction for each id conversion, replacing it in all related tables - Use Database independent queries (HQL) as much as possible, only bulk update on collection which are not supported by hibernate are in a minimalistic SQL update statement. Some helps for testing the migration on different environments is requested ! (I do my tests on MySQL deeply) I will commit my branch on platform soon. Here is my +1. -- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
On Wed, Feb 8, 2012 at 13:43, Denis Gervalle <[email protected]> wrote:
Hi devs,
I know this subject will seems to you already voted and discussed in http://xwiki.markmail.org/thread/fsd25bvft74xwgcx But following the remarks and the discussion under that thread, I had largely improved the proposed changes. This is an important matter, so I prefer to resume here to be sure we all really agree on this.
To resume the current situation, we have: - document id - used in document table, rcs, attachment... - simple 32bits string hashcode of a locally serialized document reference, including the language for translated documents - stored in a 64bits field. - object id - used in object and property tables, but also in statistics tables - simple 32bits string hashcode of the concatenation of the document reference, the class reference and the object number - stored in a 32bits field (except in Oracle, where the mapping is 32bits, but the storage is larger)
The vote is about: - document id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the document reference - the result is appended with the current locale for translated document, until locale are integrated in references - format for original document: 5space8document
Please read: - format for original document: 5:space8:document
- format for translation: 5:space8:document2:fr - object id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the BaseObjectReference - current format would be: 5:space8:document12:xspace.class[0] - if my proposal in the object reference thread is adopted: 5:space8:document18:6:xspace5:class[0]
Since changing document id could really not helps since document reference are used in object ids and therefore unambiguous document could receive ambiguous objects, I do not advice to split the change. Moreover, this is really sensible change in the database, so not multiplying them is better. I think the upcoming 4 is a really good time to introduce this change, so I propose to introduce this in version 40000 of the database (4.0M1 release). But I would like to use it internally earlier. So you would be pleased to settle on this thread and the previous one before.
It implies the following migration for existing instances: - customer custom mapping have to be adapted before the migration, including dynamic one which could be not so easy, but this is already rarely used and very rarely require any change in fact. - change XWikiDocument to provide the key required for IDs, by the way, also use that key (non local version) for the document cache - refactor the BaseElement hierarchy to provide long IDs (no more integer) based on references (generic way to have ids for any element) - change the hibernate mapping for all object ids - provide dynamic schema updates using liquibase to fix all object id types, including those in custom mapping and collection - migrate in HQL document id for persisted class: XWikiDocument, XWikiRCSNodeInfo, XWikiLink, XWikiAttachment, DeletedAttachment - migrate in HQL object id for persisted class: BaseObject, *Property, internal custom mapped class, dynamic custom mapped class - migrate in HQL object id for custom statistics class derived form XWikiStats - migrate in SQL ids for all relational collections in the above migrated tables
To provide this migration as safely as possible: - Liquibase provide a safe way to change the schema - All id conversion are gathered from the database in a first single read-only transaction, and new id are computed. - Potentially already migrated ids are detected, allowing the process to fails and be restarted. - Proceed to ids replacement using a safe algorithm that may support non-circular conflict between old and new ids (very unlikely anyway, since we move from 32 to 64bits) - Use a single transaction for each id conversion, replacing it in all related tables - Use Database independent queries (HQL) as much as possible, only bulk update on collection which are not supported by hibernate are in a minimalistic SQL update statement.
Some helps for testing the migration on different environments is requested ! (I do my tests on MySQL deeply) I will commit my branch on platform soon.
Here is my +1.
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
9 days, nobody care, most of this was voted previously, I will go ahead and commit. Document base key format will be like: 5:space8:document Object base key format will be like: 5:space8:document15:xspace.class[0] Denis On Wed, Feb 8, 2012 at 18:08, Denis Gervalle <[email protected]> wrote:
On Wed, Feb 8, 2012 at 13:43, Denis Gervalle <[email protected]> wrote:
Hi devs,
I know this subject will seems to you already voted and discussed in http://xwiki.markmail.org/thread/fsd25bvft74xwgcx But following the remarks and the discussion under that thread, I had largely improved the proposed changes. This is an important matter, so I prefer to resume here to be sure we all really agree on this.
To resume the current situation, we have: - document id - used in document table, rcs, attachment... - simple 32bits string hashcode of a locally serialized document reference, including the language for translated documents - stored in a 64bits field. - object id - used in object and property tables, but also in statistics tables - simple 32bits string hashcode of the concatenation of the document reference, the class reference and the object number - stored in a 32bits field (except in Oracle, where the mapping is 32bits, but the storage is larger)
The vote is about: - document id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the document reference - the result is appended with the current locale for translated document, until locale are integrated in references - format for original document: 5space8document
Please read: - format for original document: 5:space8:document
- format for translation: 5:space8:document2:fr - object id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the BaseObjectReference - current format would be: 5:space8:document12:xspace.class[0] - if my proposal in the object reference thread is adopted: 5:space8:document18:6:xspace5:class[0]
Since changing document id could really not helps since document reference are used in object ids and therefore unambiguous document could receive ambiguous objects, I do not advice to split the change. Moreover, this is really sensible change in the database, so not multiplying them is better. I think the upcoming 4 is a really good time to introduce this change, so I propose to introduce this in version 40000 of the database (4.0M1 release). But I would like to use it internally earlier. So you would be pleased to settle on this thread and the previous one before.
It implies the following migration for existing instances: - customer custom mapping have to be adapted before the migration, including dynamic one which could be not so easy, but this is already rarely used and very rarely require any change in fact. - change XWikiDocument to provide the key required for IDs, by the way, also use that key (non local version) for the document cache - refactor the BaseElement hierarchy to provide long IDs (no more integer) based on references (generic way to have ids for any element) - change the hibernate mapping for all object ids - provide dynamic schema updates using liquibase to fix all object id types, including those in custom mapping and collection - migrate in HQL document id for persisted class: XWikiDocument, XWikiRCSNodeInfo, XWikiLink, XWikiAttachment, DeletedAttachment - migrate in HQL object id for persisted class: BaseObject, *Property, internal custom mapped class, dynamic custom mapped class - migrate in HQL object id for custom statistics class derived form XWikiStats - migrate in SQL ids for all relational collections in the above migrated tables
To provide this migration as safely as possible: - Liquibase provide a safe way to change the schema - All id conversion are gathered from the database in a first single read-only transaction, and new id are computed. - Potentially already migrated ids are detected, allowing the process to fails and be restarted. - Proceed to ids replacement using a safe algorithm that may support non-circular conflict between old and new ids (very unlikely anyway, since we move from 32 to 64bits) - Use a single transaction for each id conversion, replacing it in all related tables - Use Database independent queries (HQL) as much as possible, only bulk update on collection which are not supported by hibernate are in a minimalistic SQL update statement.
Some helps for testing the migration on different environments is requested ! (I do my tests on MySQL deeply) I will commit my branch on platform soon.
Here is my +1.
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
If I understand properly the string length is there to assure easier migration. Are we sure this will work with Unicode strings which can use 2 bytes (if we were ever to use Unicode) ? I give my +1 but I think we need to put in the M1 release notes that this Milestone is really for people that know what they are doing. Also we need to have a good testing process on existing databases. Let us know when this is commited, XWiki SAS can put some people to do a migration test of some of our wikis. Ludovic 2012/2/17 Denis Gervalle <[email protected]>
9 days, nobody care, most of this was voted previously, I will go ahead and commit. Document base key format will be like: 5:space8:document Object base key format will be like: 5:space8:document15:xspace.class[0]
Denis
On Wed, Feb 8, 2012 at 18:08, Denis Gervalle <[email protected]> wrote:
On Wed, Feb 8, 2012 at 13:43, Denis Gervalle <[email protected]> wrote:
Hi devs,
I know this subject will seems to you already voted and discussed in http://xwiki.markmail.org/thread/fsd25bvft74xwgcx But following the remarks and the discussion under that thread, I had largely improved the proposed changes. This is an important matter, so I prefer to resume here to be sure we
all
really agree on this.
To resume the current situation, we have: - document id - used in document table, rcs, attachment... - simple 32bits string hashcode of a locally serialized document reference, including the language for translated documents - stored in a 64bits field. - object id - used in object and property tables, but also in statistics tables - simple 32bits string hashcode of the concatenation of the document reference, the class reference and the object number - stored in a 32bits field (except in Oracle, where the mapping is 32bits, but the storage is larger)
The vote is about: - document id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the document reference - the result is appended with the current locale for translated document, until locale are integrated in references - format for original document: 5space8document
Please read: - format for original document: 5:space8:document
- format for translation: 5:space8:document2:fr - object id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the BaseObjectReference - current format would be: 5:space8:document12:xspace.class[0] - if my proposal in the object reference thread is adopted: 5:space8:document18:6:xspace5:class[0]
Since changing document id could really not helps since document reference are used in object ids and therefore unambiguous document
could
receive ambiguous objects, I do not advice to split the change. Moreover, this is really sensible change in the database, so not multiplying them is better. I think the upcoming 4 is a really good time to introduce this change, so I propose to introduce this in version 40000 of the database (4.0M1 release). But I would like to use it internally earlier. So you would be pleased to settle on this thread and the previous one before.
It implies the following migration for existing instances: - customer custom mapping have to be adapted before the migration, including dynamic one which could be not so easy, but this is already rarely used and very rarely require any change in fact. - change XWikiDocument to provide the key required for IDs, by the way, also use that key (non local version) for the document cache - refactor the BaseElement hierarchy to provide long IDs (no more integer) based on references (generic way to have ids for any element) - change the hibernate mapping for all object ids - provide dynamic schema updates using liquibase to fix all object id types, including those in custom mapping and collection - migrate in HQL document id for persisted class: XWikiDocument, XWikiRCSNodeInfo, XWikiLink, XWikiAttachment, DeletedAttachment - migrate in HQL object id for persisted class: BaseObject, *Property, internal custom mapped class, dynamic custom mapped class - migrate in HQL object id for custom statistics class derived form XWikiStats - migrate in SQL ids for all relational collections in the above migrated tables
To provide this migration as safely as possible: - Liquibase provide a safe way to change the schema - All id conversion are gathered from the database in a first single read-only transaction, and new id are computed. - Potentially already migrated ids are detected, allowing the process to fails and be restarted. - Proceed to ids replacement using a safe algorithm that may support non-circular conflict between old and new ids (very unlikely anyway, since we move from 32 to 64bits) - Use a single transaction for each id conversion, replacing it in all related tables - Use Database independent queries (HQL) as much as possible, only bulk update on collection which are not supported by hibernate are in a minimalistic SQL update statement.
Some helps for testing the migration on different environments is requested ! (I do my tests on MySQL deeply) I will commit my branch on platform soon.
Here is my +1.
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Ludovic Dubost Founder and CEO Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost
On Fri, Feb 17, 2012 at 09:29, Ludovic Dubost <[email protected]> wrote:
If I understand properly the string length is there to assure easier migration.
The length are there to provide truly unique ids, whatever the content, without escaping which would be longer and useless since these key are only internal.
Are we sure this will work with Unicode strings which can use 2 bytes (if we were ever to use Unicode) ?
This is already unicode, since this is the length of java string.
I give my +1 but I think we need to put in the M1 release notes that this Milestone is really for people that know what they are doing.
I fully agree, this is why we have waited for 4.0M1, since most of this is ready since 3.4.
Also we need to have a good testing process on existing databases. Let us know when this is commited, XWiki SAS can put some people to do a migration test of some of our wikis.
This is great news, I have tested carefully on MySQL but I cannot afford testing other infrastructure. On the other hand, I use mostly HQL in the migration process or liquibase, to avoir any database dependent access. I will commit this morning. Denis
Ludovic
2012/2/17 Denis Gervalle <[email protected]>
9 days, nobody care, most of this was voted previously, I will go ahead and commit. Document base key format will be like: 5:space8:document Object base key format will be like: 5:space8:document15:xspace.class[0]
Denis
On Wed, Feb 8, 2012 at 18:08, Denis Gervalle <[email protected]> wrote:
On Wed, Feb 8, 2012 at 13:43, Denis Gervalle <[email protected]> wrote:
Hi devs,
I know this subject will seems to you already voted and discussed in http://xwiki.markmail.org/thread/fsd25bvft74xwgcx But following the remarks and the discussion under that thread, I had largely improved the proposed changes. This is an important matter, so I prefer to resume here to be sure we
all
really agree on this.
To resume the current situation, we have: - document id - used in document table, rcs, attachment... - simple 32bits string hashcode of a locally serialized document reference, including the language for translated documents - stored in a 64bits field. - object id - used in object and property tables, but also in statistics tables - simple 32bits string hashcode of the concatenation of the document reference, the class reference and the object number - stored in a 32bits field (except in Oracle, where the mapping is 32bits, but the storage is larger)
The vote is about: - document id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the document reference - the result is appended with the current locale for translated document, until locale are integrated in references - format for original document: 5space8document
Please read: - format for original document: 5:space8:document
- format for translation: 5:space8:document2:fr - object id - use the lower 64bits of an MD5 hashcode - the base key for the hashcode is serialized using a LocalUidEntityReferenceSerializer of the BaseObjectReference - current format would be: 5:space8:document12:xspace.class[0] - if my proposal in the object reference thread is adopted: 5:space8:document18:6:xspace5:class[0]
Since changing document id could really not helps since document reference are used in object ids and therefore unambiguous document
could
receive ambiguous objects, I do not advice to split the change. Moreover, this is really sensible change in the database, so not multiplying them is better. I think the upcoming 4 is a really good time to introduce this change, so I propose to introduce this in version 40000 of the database (4.0M1 release). But I would like to use it internally earlier. So you would be pleased to settle on this thread and the previous one before.
It implies the following migration for existing instances: - customer custom mapping have to be adapted before the migration, including dynamic one which could be not so easy, but this is already rarely used and very rarely require any change in fact. - change XWikiDocument to provide the key required for IDs, by the way, also use that key (non local version) for the document cache - refactor the BaseElement hierarchy to provide long IDs (no more integer) based on references (generic way to have ids for any element) - change the hibernate mapping for all object ids - provide dynamic schema updates using liquibase to fix all object id types, including those in custom mapping and collection - migrate in HQL document id for persisted class: XWikiDocument, XWikiRCSNodeInfo, XWikiLink, XWikiAttachment, DeletedAttachment - migrate in HQL object id for persisted class: BaseObject, *Property, internal custom mapped class, dynamic custom mapped class - migrate in HQL object id for custom statistics class derived form XWikiStats - migrate in SQL ids for all relational collections in the above migrated tables
To provide this migration as safely as possible: - Liquibase provide a safe way to change the schema - All id conversion are gathered from the database in a first single read-only transaction, and new id are computed. - Potentially already migrated ids are detected, allowing the process to fails and be restarted. - Proceed to ids replacement using a safe algorithm that may support non-circular conflict between old and new ids (very unlikely anyway, since we move from 32 to 64bits) - Use a single transaction for each id conversion, replacing it in all related tables - Use Database independent queries (HQL) as much as possible, only bulk update on collection which are not supported by hibernate are in a minimalistic SQL update statement.
Some helps for testing the migration on different environments is requested ! (I do my tests on MySQL deeply) I will commit my branch on platform soon.
Here is my +1.
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Ludovic Dubost Founder and CEO Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Denis Gervalle SOFTEC sa - CEO eGuilde sarl - CTO
participants (2)
-
Denis Gervalle -
Ludovic Dubost