[xwiki-devs] [proposal] Filesystem based attachment storage.
I now have a working model of attachment storage in the sandbox. http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg. This introduces a 2 major changes to the storage direction which I want to go over: 1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature. 2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path: work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content. WDYT? Caleb PS. This code currently contains no tests, I do not want to go any further until I know this path is agreed upon.
On 12/03/2010 12:45 AM, Caleb James DeLisle wrote:
I now have a working model of attachment storage in the sandbox. http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg.
This introduces a 2 major changes to the storage direction which I want to go over:
1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f...
Maybe you should make the start method final.
NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature.
2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path:
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg
The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content.
WDYT?
Looks good to me, +1. Thanks, Marius
Caleb
PS. This code currently contains no tests, I do not want to go any further until I know this path is agreed upon.
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi, On Fri, Dec 3, 2010 at 08:55, Marius Dumitru Florea < [email protected]> wrote:
On 12/03/2010 12:45 AM, Caleb James DeLisle wrote:
I now have a working model of attachment storage in the sandbox.
http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f...
I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg.
This introduces a 2 major changes to the storage direction which I want to go over:
1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of:
http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f...
Maybe you should make the start method final.
NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature.
2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path:
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg
The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content.
Right now, where are all the versions of a given file stored and under which name? Is there work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579_1.jpg work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579_2.jpg work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579_3.jpg and so on? If not, where are previous versions of the file stored?
WDYT?
Looks good to me, +1.
Looks promising to me as well... Is there a way I could get a version to test without having to build XWiki? Guillaume
Thanks, Marius
Caleb
PS. This code currently contains no tests, I do not want to go any
further until I know this path is
agreed upon.
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On 12/03/2010 04:13 AM, Guillaume Lerouge wrote:
Hi,
On Fri, Dec 3, 2010 at 08:55, Marius Dumitru Florea < [email protected]> wrote:
On 12/03/2010 12:45 AM, Caleb James DeLisle wrote:
I now have a working model of attachment storage in the sandbox.
http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f...
I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg.
This introduces a 2 major changes to the storage direction which I want to go over:
1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of:
http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f...
Maybe you should make the start method final.
NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature.
2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path:
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg
The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content.
Right now, where are all the versions of a given file stored and under which name? Is there
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579_1.jpg work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579_2.jpg work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579_3.jpg
and so on?
If not, where are previous versions of the file stored?
The versioning store is not written yet but that's about what I plan to do. My plan is to use: work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579~1.1.jpg because the user might attach a file called 30579_1.jpg which would collide with 30579.jpg (version 1) "~" will be escaped by URL encoding which happens to the file name which ensures no collisions. Caleb
WDYT?
Looks good to me, +1.
Looks promising to me as well... Is there a way I could get a version to test without having to build XWiki?
Guillaume
Thanks, Marius
Caleb
PS. This code currently contains no tests, I do not want to go any
further until I know this path is
agreed upon.
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Dec 2, 2010, at 11:45 PM, Caleb James DeLisle wrote:
I now have a working model of attachment storage in the sandbox. http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg.
Sounds cool. Note that I haven't read the code.
This introduces a 2 major changes to the storage direction which I want to go over:
1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature.
2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path:
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg
The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content.
I don't understand why we need ~this. Why can't we count the number of directories from the end? ..../attachments/<attachmentname>/<attachmentversion> So to know the attachment name it's simply 2 "/" from the end.
WDYT?
Caleb
PS. This code currently contains no tests, I do not want to go any further until I know this path is agreed upon.
bad bad :) Are you saying this wasn't TDD-developed? Oh my gaaawwwd..... -Vincent
On 12/03/2010 05:24 AM, Vincent Massol wrote:
On Dec 2, 2010, at 11:45 PM, Caleb James DeLisle wrote:
I now have a working model of attachment storage in the sandbox. http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg.
Sounds cool. Note that I haven't read the code.
This introduces a 2 major changes to the storage direction which I want to go over:
1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature.
2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path:
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg
The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content.
I don't understand why we need ~this. Why can't we count the number of directories from the end? ..../attachments/<attachmentname>/<attachmentversion>
So to know the attachment name it's simply 2 "/" from the end.
If we are never going to implement nested spaces then that plan would work fine. The problem with nested spaces comes when you have a nested space called "attachments" and then you will have a collision. Caleb
WDYT?
Caleb
PS. This code currently contains no tests, I do not want to go any further until I know this path is agreed upon.
bad bad :)
Are you saying this wasn't TDD-developed? Oh my gaaawwwd.....
-Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Dec 3, 2010, at 2:20 PM, Caleb James DeLisle wrote:
On 12/03/2010 05:24 AM, Vincent Massol wrote:
On Dec 2, 2010, at 11:45 PM, Caleb James DeLisle wrote:
I now have a working model of attachment storage in the sandbox. http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg.
Sounds cool. Note that I haven't read the code.
This introduces a 2 major changes to the storage direction which I want to go over:
1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature.
2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path:
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg
The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content.
I don't understand why we need ~this. Why can't we count the number of directories from the end? ..../attachments/<attachmentname>/<attachmentversion>
So to know the attachment name it's simply 2 "/" from the end.
If we are never going to implement nested spaces then that plan would work fine. The problem with nested spaces comes when you have a nested space called "attachments" and then you will have a collision.
Why? I don't see any collision. Thanks -Vincent
Caleb
WDYT?
Caleb
PS. This code currently contains no tests, I do not want to go any further until I know this path is agreed upon.
bad bad :)
Are you saying this wasn't TDD-developed? Oh my gaaawwwd.....
-Vincent
On 12/03/2010 09:07 AM, Vincent Massol wrote:
On Dec 3, 2010, at 2:20 PM, Caleb James DeLisle wrote:
On 12/03/2010 05:24 AM, Vincent Massol wrote:
On Dec 2, 2010, at 11:45 PM, Caleb James DeLisle wrote:
I now have a working model of attachment storage in the sandbox. http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg.
Sounds cool. Note that I haven't read the code.
This introduces a 2 major changes to the storage direction which I want to go over:
1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature.
2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path:
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg
The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content.
I don't understand why we need ~this. Why can't we count the number of directories from the end? ..../attachments/<attachmentname>/<attachmentversion>
So to know the attachment name it's simply 2 "/" from the end.
If we are never going to implement nested spaces then that plan would work fine. The problem with nested spaces comes when you have a nested space called "attachments" and then you will have a collision.
Why? I don't see any collision.
"..../attachments/<attachmentname>/<attachmentversion>" You have a document called xwiki:Main.WebHome and you attach a file called myAttach. That goes in xwiki/Main/WebHome/attachments/myAttach/1.0 Now you add a document called xwiki:Main.WebHome.attachments.myAttach.1\.0 and you attach a file called anotherAttach to this document. That attachment would go in: xwiki/Main/WebHome/attachments/myAttach/1.0/attachments/anotherAttach/1.0 xwiki/Main/WebHome/attachments/myAttach/1.0 already exists and it is not a directory. Caleb
Thanks -Vincent
Caleb
WDYT?
Caleb
PS. This code currently contains no tests, I do not want to go any further until I know this path is agreed upon.
bad bad :)
Are you saying this wasn't TDD-developed? Oh my gaaawwwd.....
-Vincent
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On 12/02/2010 11:45 PM, Caleb James DeLisle wrote:
I now have a working model of attachment storage in the sandbox. http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... I can upload and download attachments with it simply by adding the .jar file to the /lib/ dir and changing the attachment store in the xwiki.cfg.
This introduces a 2 major changes to the storage direction which I want to go over:
1. TransactionRunnable. This is a closure (like java.lang.Runnable) which does a job inside of a transaction, it also provides hooks for rollback, commit and complete events. The underlying concept is that code which is to run inside of a transaction is passed as an object to the storage engine where the transaction is opened and it is run. This will make the exception catching generic for all storage and Transaction is an interface independent of Hibernate or SQL. See the bottom part of: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-store/xwiki-store-f... NOTE: I have TransactionRunnable as a nested class while it is in development and I intend to move it out when it becomes more mature.
+0.5 (I need to take a closer look at this, but my initial feeling is that this is a good approach).
2. Filesystem hierarchical storage: An attachment called 30579.jpg a document called xwiki:Main.WebHome will be stored in a file at the path:
work/storage/xwiki/Main/WebHome/~this/attachments/30579.jpg/30579.jpg
+1 (I already talked abut this on IRC with Caleb).
The names are URL encoded for security. The directory ~this contains a character which is URL invalid and thus insures that if nested spaces are implemented, a document called xwiki:Main.WebHome.attachments will not cause a collision. The "attachments" directory is a directory dedicated to attachments for the given document. The "30579.jpg" directory under the attachments directory is dedicated to all information related to that directory. This opens the possibility for attachment history to be stored along side the attachment content.
WDYT?
Caleb
PS. This code currently contains no tests, I do not want to go any further until I know this path is agreed upon.
This is bad. Tests should drive development, since they are the specification of what needs to be done, and they are also the validation that what was done is correct. -- Sergiu Dumitriu http://purl.org/net/sergiu/
participants (5)
-
Caleb James DeLisle -
Guillaume Lerouge -
Marius Dumitru Florea -
Sergiu Dumitriu -
Vincent Massol