[xwiki-devs] [VOTE] Change document id stored in the database to reduce the likelihood of duplicate id
7 Jan
2012
7 Jan
'12
10:39 p.m.
Now that the database migration mechanism has been improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized document
reference, including the language for translated documents. The likelihood
of having duplicates with the string hashing algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed, and
this vote is in particular about the string used for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for the default document, so it
would look like 7:mySpace5:myDoc0: and its french translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the same
reference, but prefixed by <lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved independently,
so it will be stable over time which is really important when it is used as
a base for the hashing algorithm used for document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones, the
risk of collision is really low, and to be careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more risky
and the migration difficult to implements, so it is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
8 Jan
8 Jan
2:44 a.m.
+1
Caleb
On 01/07/2012 04:39 PM, Denis Gervalle wrote:
Now that the database migration mechanism has been
improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized document
reference, including the language for translated documents. The likelihood
of having duplicates with the string hashing algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed, and
this vote is in particular about the string used for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for the default document, so it
would look like 7:mySpace5:myDoc0: and its french translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the same
reference, but prefixed by <lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved independently,
so it will be stable over time which is really important when it is used as
a base for the hashing algorithm used for document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones, the
risk of collision is really low, and to be careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more risky
and the migration difficult to implements, so it is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
8:29 a.m.
+1
On Sun, Jan 8, 2012 at 2:44 AM, Caleb James DeLisle
<calebdelisle(a)lavabit.com> wrote:
+1
Caleb
On 01/07/2012 04:39 PM, Denis Gervalle wrote:
--
Thomas Mortagne
Now that the database migration mechanism has
been improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized document
reference, including the language for translated documents. The likelihood
of having duplicates with the string hashing algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed, and
this vote is in particular about the string used for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for the default document, so it
would look like 7:mySpace5:myDoc0: and its french translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the same
reference, but prefixed by <lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved independently,
so it will be stable over time which is really important when it is used as
a base for the hashing algorithm used for document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones, the
risk of collision is really low, and to be careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more risky
and the migration difficult to implements, so it is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs 
9 Jan
9 Jan
8:54 a.m.
+1
Thanks,
Marius
On Sat, Jan 7, 2012 at 11:39 PM, Denis Gervalle <dgl(a)softec.lu> wrote:
Now that the database migration mechanism has been
improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized document
reference, including the language for translated documents. The likelihood
of having duplicates with the string hashing algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed, and
this vote is in particular about the string used for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for the default document, so it
would look like 7:mySpace5:myDoc0: and its french translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the same
reference, but prefixed by <lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved independently,
so it will be stable over time which is really important when it is used as
a base for the hashing algorithm used for document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones, the
risk of collision is really low, and to be careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more risky
and the migration difficult to implements, so it is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
10:07 a.m.
New subject: [xwiki-devs] [VOTE] Change document id stored in the database to reduce the likelihood of duplicate id
+1 with the following caveats:
* We need to guarantee that a migration cannot corrupt the DB. For example imagine that we
change a document id but this id is also used in some other tables and the migration stops
before it's changed in the other tables. The change needs to be done in transactions
for each doc being changed across all tables. Said differently the migrator should be
allowed to be ctrl-c-ed at any time and you safely restart xwiki and the migrator will
just carry on from where it was.
* OR we need to have a configuration parameter for deciding to run this migration or not
so that users run it only when they decide thus ensuring that they've done the proper
backups and saving of DBs.
I prefer the first option but we need to guarantee it.
Thanks
-Vincent
On Jan 7, 2012, at 10:39 PM, Denis Gervalle wrote:
Now that the database migration mechanism has been
improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized document
reference, including the language for translated documents. The likelihood
of having duplicates with the string hashing algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed, and
this vote is in particular about the string used for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for the default document, so it
would look like 7:mySpace5:myDoc0: and its french translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the same
reference, but prefixed by <lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved independently,
so it will be stable over time which is really important when it is used as
a base for the hashing algorithm used for document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones, the
risk of collision is really low, and to be careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more risky
and the migration difficult to implements, so it is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
--
Denis Gervalle
11:09 a.m.
On Mon, Jan 9, 2012 at 10:07, Vincent Massol <vincent(a)massol.net> wrote:
+1 with the following caveats:
* We need to guarantee that a migration cannot corrupt the DB.
The evolution of the migration was the first steps in that procedure, since
accessing a DB with an inappropriate XWiki core could have corrupt it.
For example imagine that we change a document id but
this id is also used
in some other tables and the migration stops before it's changed in the
other tables. The change needs to be done in transactions for each doc
being changed across all tables.
That would be nice, but MySQL does not support transaction on ISAM table.
I use a single transaction for the whole migration process, so on systems
that support it (Oracle ?), there will be migration or not. But I could not
secure MySQL better that it is possible to.
Said differently the migrator should be allowed to be
ctrl-c-ed at any
time and you safely restart xwiki and the migrator will just carry on from
where it was.
The migrator will restart were it left-off, but the granularity is the
document. I proceed the updates by documents, updating all tables for each
one. If there is some issue during the migration let say on MySQL, and it
is restarted, it will start again skipping documents that have been
converted previously. So the corruption could be limited to a single
document.
* OR we need to have a configuration parameter for
deciding to run this
migration or not so that users run it only when they decide thus ensuring
that they've done the proper backups and saving of DBs.
This is true using the new migration procedure, but not as flexible as you
seems to expect. Supporting two hashing algorithm is not a feature, but an
augmented risk of causing corruption for me.
Now, if you use a recent core, that use new id, and on the other side, you
have not activated migrations and access an old db, you will simply be
unable to access the database. You will receive a "db require migration"
exception.
Anyway, migration are disable by default, and should be enabled by an
administrator in xwiki.cfg. The release notes will mention the needs to
proceed to it, and of course, to make a backup before. And you are always
supposed to have backup when you upgrade, or you are not a system admin ;)
I prefer the first option but we need to guarantee
it.
We will never be able to guarantee it, but I have done my best to have it
the most secure.
Thanks
-Vincent
On Jan 7, 2012, at 10:39 PM, Denis Gervalle wrote:
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
Now that the database migration mechanism has
been improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized
document
reference, including the language for translated
documents. The
likelihood
of having duplicates with the string hashing
algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed,
and
this vote is in particular about the string used
for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for
the default document, so
it
would look like 7:mySpace5:myDoc0: and its french
translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the
same
reference, but prefixed by
<lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved
independently,
so it will be stable over time which is really
important when it is used
as
a base for the hashing algorithm used for
document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones,
the
risk of collision is really low, and to be
careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more
risky
and the migration difficult to implements, so it
is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
--
Denis Gervalle
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
11:23 a.m.
New subject: [xwiki-devs] [VOTE] Change document id stored in the database to reduce the likelihood of duplicate id
On Jan 9, 2012, at 11:09 AM, Denis Gervalle wrote:
On Mon, Jan 9, 2012 at 10:07, Vincent Massol
<vincent(a)massol.net> wrote:
I think we should have one transaction per document update instead. We've had this
problem in the past when upgrading very large systems. The migration was never going
through in one go for some reason which I have forgotten so we had needed to use several
tx so that the migrations could be restarted when it failed and so that it could
complete.
+1 with the following caveats:
* We need to guarantee that a migration cannot corrupt the DB.
The evolution of the migration was the first steps in that procedure, since
accessing a DB with an inappropriate XWiki core could have corrupt it.
For example imagine that we change a document id
but this id is also used
in some other tables and the migration stops before it's changed in the
other tables. The change needs to be done in transactions for each doc
being changed across all tables.
That would be nice, but MySQL does not support transaction on ISAM table.
I use a single transaction for the whole migration process, so on systems
that support it (Oracle ?), there will be migration or not. But I could not
secure MySQL better that it is possible to.
It should work fine on MySQL with InnoDB which recommend (see
http://platform.xwiki.org/xwiki/bin/view/AdminGuide/InstallationMySQL).
Thanks
-Vincent
Said
differently the migrator should be allowed to be ctrl-c-ed at any
time and you safely restart xwiki and the migrator will just carry on from
where it was.
The migrator will restart were it left-off, but the granularity is the
document. I proceed the updates by documents, updating all tables for each
one. If there is some issue during the migration let say on MySQL, and it
is restarted, it will start again skipping documents that have been
converted previously. So the corruption could be limited to a single
document.
* OR we need to have a configuration parameter
for deciding to run this
migration or not so that users run it only when they decide thus ensuring
that they've done the proper backups and saving of DBs.
This is true using the new migration procedure, but not as flexible as you
seems to expect. Supporting two hashing algorithm is not a feature, but an
augmented risk of causing corruption for me.
Now, if you use a recent core, that use new id, and on the other side, you
have not activated migrations and access an old db, you will simply be
unable to access the database. You will receive a "db require migration"
exception.
Anyway, migration are disable by default, and should be enabled by an
administrator in xwiki.cfg. The release notes will mention the needs to
proceed to it, and of course, to make a backup before. And you are always
supposed to have backup when you upgrade, or you are not a system admin ;)
I prefer the first option but we need to
guarantee it.
We will never be able to guarantee it, but I have done my best to have it
the most secure.
Thanks
-Vincent
On Jan 7, 2012, at 10:39 PM, Denis Gervalle wrote:
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs Now that the database migration mechanism has
been improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized
document
reference, including the language for translated
documents. The
likelihood
of having duplicates with the string hashing
algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed,
and
this vote is in particular about the string used
for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for the
default document, so
it
would look like 7:mySpace5:myDoc0: and its french
translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the
same
reference, but prefixed by
<lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved
independently,
so it will be stable over time which is really
important when it is used
as
a base for the hashing algorithm used for
document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones,
the
risk of collision is really low, and to be
careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more
risky
and the migration difficult to implements, so it
is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
--
Denis Gervalle
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
11:36 a.m.
On Mon, Jan 9, 2012 at 11:23, Vincent Massol <vincent(a)massol.net> wrote:
I am myself on MyISAM since long, since there is other drawback using
InnoDB.
I do not experience much issue with corruption up to now. So you could
expect other to have similar setup.
On Jan 9, 2012, at 11:09 AM, Denis Gervalle wrote:
I think we should have one transaction per document update instead. We've
had this problem in the past when upgrading very large systems. The
migration was never going through in one go for some reason which I have
forgotten so we had needed to use several tx so that the migrations could
be restarted when it failed and so that it could complete.
This could be done easily if you want it so. Just note that all other
migration are single transaction based AFAICS.
On Mon, Jan 9, 2012 at 10:07, Vincent Massol
<vincent(a)massol.net> wrote:
since
+1 with the following caveats:
* We need to guarantee that a migration cannot corrupt the DB.
The evolution of the migration was the first steps in that procedure, accessing a DB with an inappropriate XWiki core
could have corrupt it.
> For example imagine that we change a document id but this id is also
used
in some
other tables and the migration stops before it's changed in the
other tables. The change needs to be done in transactions for each doc
being changed across all tables.
That would be nice, but MySQL does not support transaction on ISAM table.
I use a single transaction for the whole migration process, so on systems
that support it (Oracle ?), there will be migration or not. But I could
not
secure MySQL better that it is possible to.
It should work fine on MySQL with InnoDB which recommend (see
http://platform.xwiki.org/xwiki/bin/view/AdminGuide/InstallationMySQL).
Thanks
-Vincent
each
you
like
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
> Said differently the migrator should be
allowed to be ctrl-c-ed at any
> time and you safely restart xwiki and the migrator will just carry on
from
where it
was.
The migrator will restart were it left-off, but the granularity is the
document. I proceed the updates by documents, updating all tables for one. If there is some issue during the migration
let say on MySQL, and it
is restarted, it will start again skipping documents that have been
converted previously. So the corruption could be limited to a single
document.
> * OR we need to have a configuration parameter for deciding to run this
> migration or not so that users run it only when they decide thus
ensuring
that
they've done the proper backups and saving of DBs.
This is true using the new migration procedure, but not as flexible as seems to expect. Supporting two hashing algorithm
is not a feature, but
an
augmented risk of causing corruption for me.
Now, if you use a recent core, that use new id, and on the other side,
you
have not activated migrations and access an old
db, you will simply be
unable to access the database. You will receive a "db require migration"
exception.
Anyway, migration are disable by default, and should be enabled by an
administrator in xwiki.cfg. The release notes will mention the needs to
proceed to it, and of course, to make a backup before. And you are always
supposed to have backup when you upgrade, or you are not a system admin
;)
I prefer the first option but we need to
guarantee it.
We will never be able to guarantee it, but I have done my best to have it
the most secure.
>
> Thanks
> -Vincent
>
> On Jan 7, 2012, at 10:39 PM, Denis Gervalle wrote:
>
>> Now that the database migration mechanism has been improved, I would >> to go ahead with my patch to improve
document ids.
>>
>> Currently, ids are simple string hashcode of a locally serialized
> document
>> reference, including the language for translated documents. The
> likelihood
>> of having duplicates with the string hashing algorithm of java is
really
>> high.
>>
>> What I propose is:
>>
>> 1) use an MD5 hashing which is particularly good at distributing.
>> 2) truncate the hash to the first 64bits, since the XWD_ID column is a
>> 64bit long.
>> 3) use a better string representation as the source of hashing
>>
>> Based on previous discussion, point 1) and 2) has already been agreed,
> and
>> this vote is in particular about the string used for 3).
>> I propose it in 2 steps:
>>
>> 1) before locale are fully supported in document reference, use this
>> format:
>>
>>
>
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
>> where language would be an empty string
for the default document, so
> it
>> would look like 7:mySpace5:myDoc0: and its french translation could be
>> 7:mySpace5:myDoc2:fr
>> 2) when locale are included in reference, we will replace the
>> implementation by a reference serializer that would produce the same
kind
>> of representation, but that will include
all spaces (not only the last
>> one), to be prepared for the future.
>>
>> While doing so, I also propose to fix the cache key issue by using the
> same
>> reference, but prefixed by <lengthOfWikiName>:<wikiName>, so the
previous
>> examples will have the following key in
the document cache:
>> 5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
>>
>> Using such a key (compared to the usual serialization) has the
following
>> advantages:
>> - ensure uniqueness of the reference without requiring a complex
escaping
>> algorithm, which is unneeded here.
>> - potentially reversible
>> - faster than the usual serialization
>> - support language
>> - independent of the current serialization that may evolved
> independently,
>> so it will be stable over time which is really important when it is
used
> as
>> a base for the hashing algorithm used for document ids stored in the
>> database.
>>
>> I would like to introduce this as early as possible, which means has
soon
>> has we are confident with the migration
mechanism recently introduced.
>> Since the migration of ids will convert 32bits hashes into 64bits ones,
> the
>> risk of collision is really low, and to be careful, I have written a
>> migration algorithm that would support such collision (unless it cause
a
>> circular reference collision, but this is
really unexpected). However,
>> changing ids again later, if we change our mind, will be really more
> risky
>> and the migration difficult to implements, so it is really important
that
we agree on the way we compute these ids, once for
all.
Here is my +1,
--
Denis Gervalle
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
11:44 a.m.
New subject: [xwiki-devs] [VOTE] Change document id stored in the database to reduce the likelihood of duplicate id
On Jan 9, 2012, at 11:36 AM, Denis Gervalle wrote:
I am myself on MyISAM since long, since there is other drawback using
InnoDB.
I do not experience much issue with corruption up to now. So you could
expect other to have similar setup.
>
> Thanks
> -Vincent
>
>>> Said differently the migrator should be allowed to be ctrl-c-ed at any
>>> time and you safely restart xwiki and the migrator will just carry on
> from
>>> where it was.
>>>
>>
>> The migrator will restart were it left-off, but the granularity is the
>> document. I proceed the updates by documents, updating all tables for
> each
>> one. If there is some issue during the migration let say on MySQL, and it
>> is restarted, it will start again skipping documents that have been
>> converted previously. So the corruption could be limited to a single
>> document.
>>
>>
>>> * OR we need to have a configuration parameter for deciding to run this
>>> migration or not so that users run it only when they decide thus
> ensuring
>>> that they've done the proper backups and saving of DBs.
>>>
>>
>> This is true using the new migration procedure, but not as flexible as
> you
>> seems to expect. Supporting two hashing algorithm is not a feature, but
> an
>> augmented risk of causing corruption for me.
>> Now, if you use a recent core, that use new id, and on the other side,
> you
>> have not activated migrations and access an old db, you will simply be
>> unable to access the database. You will receive a "db require
migration"
>> exception.
>>
>> Anyway, migration are disable by default, and should be enabled by an
>> administrator in xwiki.cfg. The release notes will mention the needs to
>> proceed to it, and of course, to make a backup before. And you are always
>> supposed to have backup when you upgrade, or you are not a system admin
> ;)
>>
>>
>>> I prefer the first option but we need to guarantee it.
>>>
>>
>> We will never be able to guarantee it, but I have done my best to have it
>> the most secure.
>>
>>
>>>
>>> Thanks
>>> -Vincent
>>>
>>> On Jan 7, 2012, at 10:39 PM, Denis Gervalle wrote:
>>>
>>>> Now that the database migration mechanism has been improved, I would
> like
>>>> to go ahead with my patch to improve document ids.
>>>>
>>>> Currently, ids are simple string hashcode of a locally serialized
>>> document
>>>> reference, including the language for translated documents. The
>>> likelihood
>>>> of having duplicates with the string hashing algorithm of java is
> really
>>>> high.
>>>>
>>>> What I propose is:
>>>>
>>>> 1) use an MD5 hashing which is particularly good at distributing.
>>>> 2) truncate the hash to the first 64bits, since the XWD_ID column is a
>>>> 64bit long.
>>>> 3) use a better string representation as the source of hashing
>>>>
>>>> Based on previous discussion, point 1) and 2) has already been agreed,
>>> and
>>>> this vote is in particular about the string used for 3).
>>>> I propose it in 2 steps:
>>>>
>>>> 1) before locale are fully supported in document reference, use this
>>>> format:
>>>>
>>>>
>>>
>
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
>>>> where language would be an empty string for the default document, so
>>> it
>>>> would look like 7:mySpace5:myDoc0: and its french translation could be
>>>> 7:mySpace5:myDoc2:fr
>>>> 2) when locale are included in reference, we will replace the
>>>> implementation by a reference serializer that would produce the same
> kind
>>>> of representation, but that will include all spaces (not only the last
>>>> one), to be prepared for the future.
>>>>
>>>> While doing so, I also propose to fix the cache key issue by using the
>>> same
>>>> reference, but prefixed by <lengthOfWikiName>:<wikiName>, so
the
> previous
>>>> examples will have the following key in the document cache:
>>>> 5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
>>>>
>>>> Using such a key (compared to the usual serialization) has the
> following
>>>> advantages:
>>>> - ensure uniqueness of the reference without requiring a complex
> escaping
>>>> algorithm, which is unneeded here.
>>>> - potentially reversible
>>>> - faster than the usual serialization
>>>> - support language
>>>> - independent of the current serialization that may evolved
>>> independently,
>>>> so it will be stable over time which is really important when it is
> used
>>> as
>>>> a base for the hashing algorithm used for document ids stored in the
>>>> database.
>>>>
>>>> I would like to introduce this as early as possible, which means has
> soon
>>>> has we are confident with the migration mechanism recently introduced.
>>>> Since the migration of ids will convert 32bits hashes into 64bits ones,
>>> the
>>>> risk of collision is really low, and to be careful, I have written a
>>>> migration algorithm that would support such collision (unless it cause
> a
>>>> circular reference collision, but this is really unexpected). However,
>>>> changing ids again later, if we change our mind, will be really more
>>> risky
>>>> and the migration difficult to implements, so it is really important
> that
>>>> we agree on the way we compute these ids, once for all.
>>>>
>>>> Here is my +1,
>>>>
>>>> --
>>>> Denis Gervalle
On Mon, Jan 9, 2012 at 11:23, Vincent Massol
<vincent(a)massol.net> wrote:
I'm pretty sure this isn't the case.
See R4359XWIKI1459DataMigration and R6079XWIKI1878DataMigration for example.
Thanks
-Vincent
On Jan 9, 2012, at 11:09 AM, Denis Gervalle wrote:
I think we should have one transaction per document update instead. We've
had this problem in the past when upgrading very large systems. The
migration was never going through in one go for some reason which I have
forgotten so we had needed to use several tx so that the migrations could
be restarted when it failed and so that it could complete.
This could be done easily if you want it so. Just note that all other
migration are single transaction based AFAICS. On Mon, Jan 9, 2012 at 10:07, Vincent Massol
<vincent(a)massol.net> wrote:
since
+1 with the following caveats:
* We need to guarantee that a migration cannot corrupt the DB.
The evolution of the migration was the first steps in that procedure, accessing a DB with an inappropriate XWiki core
could have corrupt it.
> For example imagine that we change a document id but this id is also
used
in some
other tables and the migration stops before it's changed in the
other tables. The change needs to be done in transactions for each doc
being changed across all tables.
That would be nice, but MySQL does not support transaction on ISAM table.
I use a single transaction for the whole migration process, so on systems
that support it (Oracle ?), there will be migration or not. But I could
not
secure MySQL better that it is possible to.
It should work fine on MySQL with InnoDB which recommend (see
http://platform.xwiki.org/xwiki/bin/view/AdminGuide/InstallationMySQL).
3:35 p.m.
On Mon, Jan 9, 2012 at 11:44, Vincent Massol <vincent(a)massol.net> wrote:
On Jan 9, 2012, at 11:36 AM, Denis Gervalle wrote:
I'm pretty sure this isn't the case.
See R4359XWIKI1459DataMigration and R6079XWIKI1878DataMigration for
example.
On Mon, Jan 9, 2012 at 11:23, Vincent Massol
<vincent(a)massol.net> wrote:
>
> On Jan 9, 2012, at 11:09 AM, Denis Gervalle wrote:
>
>> On Mon, Jan 9, 2012 at 10:07, Vincent Massol <vincent(a)massol.net>
wrote:
>>
>>> +1 with the following caveats:
>>>
>>> * We need to guarantee that a migration cannot corrupt the DB.
>>
>>
>> The evolution of the migration was the first steps in that procedure,
> since
>> accessing a DB with an inappropriate XWiki core could have corrupt it.
>>
>>
>>> For example imagine that we change a document id but this id is also
> used
>>> in some other tables and the migration stops before it's changed in
the
>>> other tables. The change needs to be
done in transactions for each doc
>>> being changed across all tables.
>>
>>
>> That would be nice, but MySQL does not support transaction on ISAM
table.
>> I use a single transaction for the whole
migration process,
>
> I think we should have one transaction per document update instead.
We've
> had this problem in the past when upgrading
very large systems. The
> migration was never going through in one go for some reason which I have
> forgotten so we had needed to use several tx so that the migrations
could
be
restarted when it failed and so that it could complete.
This could be done easily if you want it so. Just note that all other
migration are single transaction based AFAICS. From what I see in those, we use a single separate
session and transaction
for executing some stuffs, I really do not see a partial
commit there
between "rows".
Thanks
-Vincent
I am myself on MyISAM since long, since there is other drawback using
InnoDB.
I do not experience much issue with corruption up to now. So you could
expect other to have similar setup.
>
> Thanks
> -Vincent
>
>>> Said differently the migrator should be allowed to be ctrl-c-ed at any
>>> time and you safely restart xwiki and the migrator will just carry on
> from
>>> where it was.
>>>
>>
>> The migrator will restart were it left-off, but the granularity is the
>> document. I proceed the updates by documents, updating all tables for
> each
>> one. If there is some issue during the migration let say on MySQL, and it
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
so on systems
that support it (Oracle ?), there will be migration or not. But I could
not
secure MySQL better that it is possible to.
It should work fine on MySQL with InnoDB which recommend (see
http://platform.xwiki.org/xwiki/bin/view/AdminGuide/InstallationMySQL).
>> is restarted, it will start again
skipping documents that have been
>> converted previously. So the corruption could be limited to a single
>> document.
>>
>>
>>> * OR we need to have a configuration parameter for deciding to run
this
>>> migration or not so that users run it
only when they decide thus
> ensuring
>>> that they've done the proper backups and saving of DBs.
>>>
>>
>> This is true using the new migration procedure, but not as flexible as
> you
>> seems to expect. Supporting two hashing algorithm is not a feature, but
> an
>> augmented risk of causing corruption for me.
>> Now, if you use a recent core, that use new id, and on the other side,
> you
>> have not activated migrations and access an old db, you will simply be
>> unable to access the database. You will receive a "db require
migration"
>> exception.
>>
>> Anyway, migration are disable by default, and should be enabled by an
>> administrator in xwiki.cfg. The release notes will mention the needs to
>> proceed to it, and of course, to make a backup before. And you are
always
>> supposed to have backup when you upgrade,
or you are not a system admin
> ;)
>>
>>
>>> I prefer the first option but we need to guarantee it.
>>>
>>
>> We will never be able to guarantee it, but I have done my best to have
it
>> the most secure.
>>
>>
>>>
>>> Thanks
>>> -Vincent
>>>
>>> On Jan 7, 2012, at 10:39 PM, Denis Gervalle wrote:
>>>
>>>> Now that the database migration mechanism has been improved, I would
> like
>>>> to go ahead with my patch to improve document ids.
>>>>
>>>> Currently, ids are simple string hashcode of a locally serialized
>>> document
>>>> reference, including the language for translated documents. The
>>> likelihood
>>>> of having duplicates with the string hashing algorithm of java is
> really
>>>> high.
>>>>
>>>> What I propose is:
>>>>
>>>> 1) use an MD5 hashing which is particularly good at distributing.
>>>> 2) truncate the hash to the first 64bits, since the XWD_ID column is
a
>>>> 64bit long.
>>>> 3) use a better string representation as the source of hashing
>>>>
>>>> Based on previous discussion, point 1) and 2) has already been
agreed,
>>> and
>>>> this vote is in particular about the string used for 3).
>>>> I propose it in 2 steps:
>>>>
>>>> 1) before locale are fully supported in document reference, use this
>>>> format:
>>>>
>>>>
>>>
>
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
>>>> where language would be an empty
string for the default document, so
>>> it
>>>> would look like 7:mySpace5:myDoc0: and its french translation could
be
>>>> 7:mySpace5:myDoc2:fr
>>>> 2) when locale are included in reference, we will replace the
>>>> implementation by a reference serializer that would produce the same
> kind
>>>> of representation, but that will include all spaces (not only the
last
>>>> one), to be prepared for the
future.
>>>>
>>>> While doing so, I also propose to fix the cache key issue by using
the
>>> same
>>>> reference, but prefixed by <lengthOfWikiName>:<wikiName>, so
the
> previous
>>>> examples will have the following key in the document cache:
>>>> 5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
>>>>
>>>> Using such a key (compared to the usual serialization) has the
> following
>>>> advantages:
>>>> - ensure uniqueness of the reference without requiring a complex
> escaping
>>>> algorithm, which is unneeded here.
>>>> - potentially reversible
>>>> - faster than the usual serialization
>>>> - support language
>>>> - independent of the current serialization that may evolved
>>> independently,
>>>> so it will be stable over time which is really important when it is
> used
>>> as
>>>> a base for the hashing algorithm used for document ids stored in the
>>>> database.
>>>>
>>>> I would like to introduce this as early as possible, which means has
> soon
>>>> has we are confident with the migration mechanism recently
introduced.
>>>> Since the migration of ids will
convert 32bits hashes into 64bits
ones,
>>> the
>>>> risk of collision is really low, and to be careful, I have written a
>>>> migration algorithm that would support such collision (unless it
cause
> a
>>>> circular reference collision, but this is really unexpected).
However,
>>>> changing ids again later, if we
change our mind, will be really more
>>> risky
>>>> and the migration difficult to implements, so it is really important
> that
>>>> we agree on the way we compute these ids, once for all.
>>>>
>>>> Here is my +1,
>>>>
>>>> --
>>>> Denis Gervalle
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
12:20 p.m.
Hi,
I have one small concern which leads to a big concern, and I was wondering
about something.
1/ Small concern: what did we do to verify the potential level of
collisions and if there is a chance they happen in our case
I see we want to truncate the MD5 hash to 64 bits. I was wondering if there
is a not a risk of having more collisions.
My question here is what did we do to verify the level of collisions on
real data.
We could provide some XWiki SAS client DBs, including our Intranet which is
quite big for testing if there was a testing program.
2/ Bigger concern: wouldn't it be better to have a way to
activate/deactivate the new feature. This would allow to still upgrade and
make tests on real life data without risking being in a corner
3/ Wondering: wouldn't it be better to use the real reference as the ID and
move to strings for it
Give that in an XWiki database, this part is really small (compared to
attachments and the data itself), are there really any reasons to use IDs
for this reference. Wouldn't the use of a String be better in the end ? We
already use this for the join between xwikidoc and xwikiobjects and haven't
seen any big problem with that did we ?
If we used that method wouldn't it means ZERO collision ?
4/ Small additional stuff
There is also the migration of Object IDs right ? The object IDs use the
same system and also have a risk of collision (which would lead to property
data being shared with completely unrelevant documents)
Ludovic
2012/1/7 Denis Gervalle <dgl(a)softec.lu>
Now that the database migration mechanism has been
improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized document
reference, including the language for translated documents. The likelihood
of having duplicates with the string hashing algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed, and
this vote is in particular about the string used for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for the default document, so it
would look like 7:mySpace5:myDoc0: and its french translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the same
reference, but prefixed by <lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved independently,
so it will be stable over time which is really important when it is used as
a base for the hashing algorithm used for document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones, the
risk of collision is really low, and to be careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more risky
and the migration difficult to implements, so it is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
5:01 p.m.
On Mon, Jan 9, 2012 at 12:20, Ludovic Dubost <ludovic(a)xwiki.com> wrote:
Hi,
I have one small concern which leads to a big concern, and I was wondering
about something.
1/ Small concern: what did we do to verify the potential level of
collisions and if there is a chance they happen in our case
In theory, there is a risk, the same than the one you have with current
ids, but since the risk is reduced, it would be really bad luck to felt on
it. Note that we move from 32bits badly formed hash, to 64bits well suited
one. So the risk is not zero, but really unexpected since we double the
hash space and use a better repartition algorithm.
I see we want to truncate the MD5 hash to 64 bits. I
was wondering if there
is a not a risk of having more collisions.
Sure, using the lower 64bits is not as good as using the full 128bits.
Using more bits require a change in the mapping and schema (see later).
My question here is what did we do to verify the level
of collisions on
real data.
We could provide some XWiki SAS client DBs, including our Intranet which is
quite big for testing if there was a testing program.
That will be the purpose of the non final release, have some of you check
that their large DB works well with it. In particular, I would appreciate
some tests in non MySQL environment, especially Oracle...
2/ Bigger concern: wouldn't it be better to have a
way to
activate/deactivate the new feature. This would allow to still upgrade and
make tests on real life data without risking being in a corner
We are already in a corner since we have fallen on id collision, so I only
try to put that corner further until we can fully change this to a fully
unique id. I know this is not easy, but we cannot stay stucked.
Providing both would require a two way migrator, and this would also
introduce more risk of mistakes that would cause database corruptions. I
have built a solid migrator that ensure you will migrate properly and
voluntarily before using the new core.
Note that if you plug an old core on the new DB, this one will corrupt it
somewhat, by not seing any documents, and recreating some default initial
documents using the old ids. This is more concerning since event if
document will not mixup, there object will partly, which will cause really
annoying issue. But what can we do, we cannot change old core retroactively.
So even if we provide rock solid solution, an old core will still corrupt
data. We cannot prevent all administrator mistakes.
Also providing both would means that we do not trust our 64bits hash to be
better than the 32bits one. As I said, there is no zero risk, but the
probability is really near zero.
3/ Wondering: wouldn't it be better to use the real reference as the ID and
move to strings for it
Give that in an XWiki database, this part is really
small (compared to
attachments and the data itself), are there really any reasons to use IDs
for this reference. Wouldn't the use of a String be better in the end ? We
already use this for the join between xwikidoc and xwikiobjects and haven't
seen any big problem with that did we ?
But objects still use an id (for hibernate and) to link properties to
objects as you mention later. Maybe you want to means that we use a mix of
objects for properties.
If we used that method wouldn't it means ZERO
collision ?
Sure, it would have been from the beginning, but it was not, and this is
now really difficult to change. The best would have been not to use
significant ID. Changing to string now would require an external migration
process, that use the old mapping to create a new id, than another process
that use the new mapping and remove the old ids. This is really another
job, that would be best done when we fully review the model.
4/ Small additional stuff
There is also the migration of Object IDs right ? The object IDs use the
same system and also have a risk of collision (which would lead to property
data being shared with completely unrelevant documents)
Good catch ! I have not seen it, since the object does not directly depends
on document ids, but indirectly depends on similarly calculated ones. I
need to look at these as well since this could mean two object of a
previously colliding documents will collide.
Thanks,
Ludovic
2012/1/7 Denis Gervalle <dgl(a)softec.lu>
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
Now that the database migration mechanism has
been improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized
document
reference, including the language for translated
documents. The
likelihood
of having duplicates with the string hashing
algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed,
and
this vote is in particular about the string used
for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for
the default document, so
it
would look like 7:mySpace5:myDoc0: and its french
translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the
same
reference, but prefixed by
<lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex
escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved
independently,
so it will be stable over time which is really
important when it is used
as
a base for the hashing algorithm used for
document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones,
the
risk of collision is really low, and to be
careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more
risky
and the migration difficult to implements, so it
is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
--
Denis Gervalle
SOFTEC sa - CEO
eGuilde sarl - CTO
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
12 Jan
12 Jan
5:38 a.m.
+1:
- there's really no need to support older hashcode IDs (Ludo's question)
- changing object IDs should be done in a different migrator, but it
should be done soon, nevertheless (in most instances there are more
objects than documents in a database)
- +1 for smaller transactions, one per document or at least one per
smaller batch of, let's say, 1000 documents
On 01/07/2012 04:39 PM, Denis Gervalle wrote:
Now that the database migration mechanism has been
improved, I would like
to go ahead with my patch to improve document ids.
Currently, ids are simple string hashcode of a locally serialized document
reference, including the language for translated documents. The likelihood
of having duplicates with the string hashing algorithm of java is really
high.
What I propose is:
1) use an MD5 hashing which is particularly good at distributing.
2) truncate the hash to the first 64bits, since the XWD_ID column is a
64bit long.
3) use a better string representation as the source of hashing
Based on previous discussion, point 1) and 2) has already been agreed, and
this vote is in particular about the string used for 3).
I propose it in 2 steps:
1) before locale are fully supported in document reference, use this
format:
<lengthOfLastSpaceName>:<lastSpaceName><lengthOfDocumentName>:<documentName><lengthOfLanguage>:<language>
where language would be an empty string for the default document, so it
would look like 7:mySpace5:myDoc0: and its french translation could be
7:mySpace5:myDoc2:fr
2) when locale are included in reference, we will replace the
implementation by a reference serializer that would produce the same kind
of representation, but that will include all spaces (not only the last
one), to be prepared for the future.
While doing so, I also propose to fix the cache key issue by using the same
reference, but prefixed by<lengthOfWikiName>:<wikiName>, so the previous
examples will have the following key in the document cache:
5:xwiki7:mySpace5:myDoc0: and 5:xwiki7:mySpace5:myDoc2:fr
Using such a key (compared to the usual serialization) has the following
advantages:
- ensure uniqueness of the reference without requiring a complex escaping
algorithm, which is unneeded here.
- potentially reversible
- faster than the usual serialization
- support language
- independent of the current serialization that may evolved independently,
so it will be stable over time which is really important when it is used as
a base for the hashing algorithm used for document ids stored in the
database.
I would like to introduce this as early as possible, which means has soon
has we are confident with the migration mechanism recently introduced.
Since the migration of ids will convert 32bits hashes into 64bits ones, the
risk of collision is really low, and to be careful, I have written a
migration algorithm that would support such collision (unless it cause a
circular reference collision, but this is really unexpected). However,
changing ids again later, if we change our mind, will be really more risky
and the migration difficult to implements, so it is really important that
we agree on the way we compute these ids, once for all.
Here is my +1,
--
Sergiu Dumitriu
http://purl.org/net/sergiu/
4631
days inactive
4636
days old
12 comments
7 participants
participants (7)
-
Caleb James DeLisle -
Denis Gervalle -
Ludovic Dubost -
Marius Dumitru Florea -
Sergiu Dumitriu -
Thomas Mortagne -
Vincent Massol