[xwiki-devs] [VOTE] How to handle large content inside string properties
Hi devs, While trying to fix XWIKI-1480, the following question was raised: What to do with large values placed inside a StringProperty? StringProperties are defined to hold at most 255 characters (which are actually bytes in mysql). Some databases automatically ignore this limit (this is the case with hsql, which is the default database for our standalone package), while others throw exceptions when trying to commit the transaction (mysql does this). Currently, we don't handle this exception at all, so it bubbles up to the UI in the form of a stack trace. XWIKI-1480 is about fixing this error. What can be done: 1. Always trim the content to 255 bytes. This is a regression from the point of view of hsql users, which will suddenly notice that their data gets truncated. 2. If the data is larger than 255 bytes, silently ignore this new value, and continue using the previous one. This is also bad, since users won't understand why doesn't the value get saved, and is a regression on hsql. 3. Try to use this value, but if an exception is thrown, truncate the data and commit again. This fixes the regression part, but might still be surprising to users. 4. Continue as before, but catch the exception and display a nice warning message to the user. This looks like the safest approach, but it fails when manipulating data from outside the UI, like scripts and XmlRpc connections. So, what is the best approach? -- Sergiu Dumitriu http://purl.org/net/sergiu/
Hi, On Thu, Apr 9, 2009 at 1:52 AM, Sergiu Dumitriu <[email protected]> wrote:
Hi devs,
While trying to fix XWIKI-1480, the following question was raised: What to do with large values placed inside a StringProperty?
StringProperties are defined to hold at most 255 characters (which are actually bytes in mysql). Some databases automatically ignore this limit (this is the case with hsql, which is the default database for our standalone package), while others throw exceptions when trying to commit the transaction (mysql does this).
Currently, we don't handle this exception at all, so it bubbles up to the UI in the form of a stack trace. XWIKI-1480 is about fixing this error.
What can be done:
1. Always trim the content to 255 bytes. This is a regression from the point of view of hsql users, which will suddenly notice that their data gets truncated.
2. If the data is larger than 255 bytes, silently ignore this new value, and continue using the previous one. This is also bad, since users won't understand why doesn't the value get saved, and is a regression on hsql.
3. Try to use this value, but if an exception is thrown, truncate the data and commit again. This fixes the regression part, but might still be surprising to users.
4. Continue as before, but catch the exception and display a nice warning message to the user. This looks like the safest approach, but it fails when manipulating data from outside the UI, like scripts and XmlRpc connections.
So, what is the best approach?
What about a mix of 3) & 4) ? Try using the value, but if an exception is thrown truncate the data and commit again. Then display a warning message to the user saying that his content has been truncated because it was too long and that only 255 chars are allowed for the property that threw the exception. The message wouldn't be displayed when manipulating data from script, but I don't think that's much of the problem -> it becomes the script's author problem at that point. WDYT? Guillaume
-- Sergiu Dumitriu http://purl.org/net/sergiu/ _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Guillaume Lerouge Product Manager - XWiki Skype ID : wikibc http://guillaumelerouge.com/
On Apr 9, 2009, at 1:52 AM, Sergiu Dumitriu wrote:
Hi devs,
While trying to fix XWIKI-1480, the following question was raised: What to do with large values placed inside a StringProperty?
StringProperties are defined to hold at most 255 characters (which are actually bytes in mysql). Some databases automatically ignore this limit (this is the case with hsql, which is the default database for our standalone package), while others throw exceptions when trying to commit the transaction (mysql does this).
Currently, we don't handle this exception at all, so it bubbles up to the UI in the form of a stack trace. XWIKI-1480 is about fixing this error.
What can be done:
1. Always trim the content to 255 bytes. This is a regression from the point of view of hsql users, which will suddenly notice that their data gets truncated.
2. If the data is larger than 255 bytes, silently ignore this new value, and continue using the previous one. This is also bad, since users won't understand why doesn't the value get saved, and is a regression on hsql.
3. Try to use this value, but if an exception is thrown, truncate the data and commit again. This fixes the regression part, but might still be surprising to users.
4. Continue as before, but catch the exception and display a nice warning message to the user. This looks like the safest approach, but it fails when manipulating data from outside the UI, like scripts and XmlRpc connections.
So, what is the best approach?
Do we have to store string properties in such small types? Would it be detrimental to performances a lot to use a large DB types to store string properties? Thanks -Vincent
Vincent Massol wrote:
On Apr 9, 2009, at 1:52 AM, Sergiu Dumitriu wrote:
Hi devs,
While trying to fix XWIKI-1480, the following question was raised: What to do with large values placed inside a StringProperty?
StringProperties are defined to hold at most 255 characters (which are actually bytes in mysql). Some databases automatically ignore this limit (this is the case with hsql, which is the default database for our standalone package), while others throw exceptions when trying to commit the transaction (mysql does this).
Currently, we don't handle this exception at all, so it bubbles up to the UI in the form of a stack trace. XWIKI-1480 is about fixing this error.
What can be done:
1. Always trim the content to 255 bytes. This is a regression from the point of view of hsql users, which will suddenly notice that their data gets truncated.
2. If the data is larger than 255 bytes, silently ignore this new value, and continue using the previous one. This is also bad, since users won't understand why doesn't the value get saved, and is a regression on hsql.
3. Try to use this value, but if an exception is thrown, truncate the data and commit again. This fixes the regression part, but might still be surprising to users.
4. Continue as before, but catch the exception and display a nice warning message to the user. This looks like the safest approach, but it fails when manipulating data from outside the UI, like scripts and XmlRpc connections.
So, what is the best approach?
Do we have to store string properties in such small types? Would it be detrimental to performances a lot to use a large DB types to store string properties?
I for one don't think that making the field larger is a problem. The only downside I see is that it will be harder to make an index on it. Maybe our sysadmins can say more about how they optimize databases. -- Sergiu Dumitriu http://purl.org/net/sergiu/
Sergiu Dumitriu wrote:
Vincent Massol wrote:
On Apr 9, 2009, at 1:52 AM, Sergiu Dumitriu wrote:
Hi devs,
While trying to fix XWIKI-1480, the following question was raised: What to do with large values placed inside a StringProperty?
StringProperties are defined to hold at most 255 characters (which are actually bytes in mysql). Some databases automatically ignore this limit (this is the case with hsql, which is the default database for our standalone package), while others throw exceptions when trying to commit the transaction (mysql does this).
Currently, we don't handle this exception at all, so it bubbles up to the UI in the form of a stack trace. XWIKI-1480 is about fixing this error.
What can be done:
1. Always trim the content to 255 bytes. This is a regression from the point of view of hsql users, which will suddenly notice that their data gets truncated.
2. If the data is larger than 255 bytes, silently ignore this new value, and continue using the previous one. This is also bad, since users won't understand why doesn't the value get saved, and is a regression on hsql.
3. Try to use this value, but if an exception is thrown, truncate the data and commit again. This fixes the regression part, but might still be surprising to users.
4. Continue as before, but catch the exception and display a nice warning message to the user. This looks like the safest approach, but it fails when manipulating data from outside the UI, like scripts and XmlRpc connections.
So, what is the best approach?
Do we have to store string properties in such small types? Would it be detrimental to performances a lot to use a large DB types to store string properties?
I for one don't think that making the field larger is a problem. The only downside I see is that it will be harder to make an index on it. Maybe our sysadmins can say more about how they optimize databases.
Still, any limit is a tangible limit that can be crossed. 255 bytes is easier to cross than 65k, but what if somebody wants to put large data in a string property? Do we agree on option 4? -- Sergiu Dumitriu http://purl.org/net/sergiu/
I vote for 4 but also extending the DB field to mediumtext which would make this actually never happen Ludovic Sergiu Dumitriu a écrit :
Hi devs,
While trying to fix XWIKI-1480, the following question was raised: What to do with large values placed inside a StringProperty?
StringProperties are defined to hold at most 255 characters (which are actually bytes in mysql). Some databases automatically ignore this limit (this is the case with hsql, which is the default database for our standalone package), while others throw exceptions when trying to commit the transaction (mysql does this).
Currently, we don't handle this exception at all, so it bubbles up to the UI in the form of a stack trace. XWIKI-1480 is about fixing this error.
What can be done:
1. Always trim the content to 255 bytes. This is a regression from the point of view of hsql users, which will suddenly notice that their data gets truncated.
2. If the data is larger than 255 bytes, silently ignore this new value, and continue using the previous one. This is also bad, since users won't understand why doesn't the value get saved, and is a regression on hsql.
3. Try to use this value, but if an exception is thrown, truncate the data and commit again. This fixes the regression part, but might still be surprising to users.
4. Continue as before, but catch the exception and display a nice warning message to the user. This looks like the safest approach, but it fails when manipulating data from outside the UI, like scripts and XmlRpc connections.
So, what is the best approach?
-- Ludovic Dubost Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost
participants (4)
-
Guillaume Lerouge -
Ludovic Dubost -
Sergiu Dumitriu -
Vincent Massol