[xwiki-devs] [proposal] New module to handle large binary objects in a generic way.
In light of the need for better attachment handling, I spent the past few days working on the basic framework for a new module to support large objects. The code can be viewed here: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-blob/src/main/java/... Use case #1: Multiple methods of storing content and the ability to define more. Use case #2: Thread safe locking to prevent concurrent modifications from damaging content. Use case #3: ACID commit/rollback protection. The API user has 2 access points. BinaryObject is a thread safe interface with ACID and option of using OutputStream or InputStream to set and get data. StorageItem is a simplistic interface modeled after a small subset of the methods in java.io.File. If you create a component which implements StorageItem, BinaryObjectProvider will allow you to access a BinaryObject which uses your StorageItem for persistent storage. Example: @Requirement private BinaryObjectProvider provider; private BinaryObject binaryObj; public void initialize() { this.binaryObj = this.provider.get("myImpl"); } "myImpl" is the hint for the StorageItem which you may define. The next step is of course to add versioning support but I think this will provide a solid foundation to build that on. WDYT? Caleb
Hi Caleb, I haven't looked at the code yet but could you provide the architectural vision going forward and how this module would relate to the future storage module. Also imagining we use JCR as the storage engine and/or API, how would this module integrate with it? Thanks -Vincent On Oct 28, 2010, at 1:45 PM, Caleb James DeLisle wrote:
In light of the need for better attachment handling, I spent the past few days working on the basic framework for a new module to support large objects. The code can be viewed here: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-blob/src/main/java/...
Use case #1: Multiple methods of storing content and the ability to define more.
Use case #2: Thread safe locking to prevent concurrent modifications from damaging content.
Use case #3: ACID commit/rollback protection.
The API user has 2 access points. BinaryObject is a thread safe interface with ACID and option of using OutputStream or InputStream to set and get data.
StorageItem is a simplistic interface modeled after a small subset of the methods in java.io.File.
If you create a component which implements StorageItem, BinaryObjectProvider will allow you to access a BinaryObject which uses your StorageItem for persistent storage.
Example:
@Requirement private BinaryObjectProvider provider;
private BinaryObject binaryObj;
public void initialize() { this.binaryObj = this.provider.get("myImpl"); }
"myImpl" is the hint for the StorageItem which you may define.
The next step is of course to add versioning support but I think this will provide a solid foundation to build that on.
WDYT?
Caleb
``this module would relate to the future storage module.'' Is there an agreed upon future path for XWiki storage? If so I wasn't aware of it. I designed the module to be as flexible and easy to use as possible. Storage in an SQL or JCR database (or anywhere else) is possible by implementing StorageItem. This uses UUID as the key for loading the content simply because it's easy to manage and binary data won't be subject to the types of queries with which UUIDs perform poorly. How content is ID tagged at the low level is something which deserves discussion though. I see large content such as attachments moving away from the database and toward the filesystem in the future because of better performance so I think persistance of large binary objects will remain a seperate concern for some time to come. I understand that versioning and large item support is promised by JCR and although I am not against JCR if it performs well, I see a tremendous amount of momentum for hibernate+SQL in the code and among the users and I think writing a drop in replacement for hibernate would be a task of such difficulty and expense as to be paralleled only by that of building a community around it. I designed xwiki-blob to be extensible to new and different storage platforms but to implement the features which we need now rather than waiting for a new storage engine. This answers the question of how this module would relate to how I expect the future storage will turn out. Does that answer your question? Caleb On 10/28/2010 08:03 AM, Vincent Massol wrote:
Hi Caleb,
I haven't looked at the code yet but could you provide the architectural vision going forward and how this module would relate to the future storage module. Also imagining we use JCR as the storage engine and/or API, how would this module integrate with it?
Thanks -Vincent
On Oct 28, 2010, at 1:45 PM, Caleb James DeLisle wrote:
In light of the need for better attachment handling, I spent the past few days working on the basic framework for a new module to support large objects. The code can be viewed here: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-blob/src/main/java/...
Use case #1: Multiple methods of storing content and the ability to define more.
Use case #2: Thread safe locking to prevent concurrent modifications from damaging content.
Use case #3: ACID commit/rollback protection.
The API user has 2 access points. BinaryObject is a thread safe interface with ACID and option of using OutputStream or InputStream to set and get data.
StorageItem is a simplistic interface modeled after a small subset of the methods in java.io.File.
If you create a component which implements StorageItem, BinaryObjectProvider will allow you to access a BinaryObject which uses your StorageItem for persistent storage.
Example:
@Requirement private BinaryObjectProvider provider;
private BinaryObject binaryObj;
public void initialize() { this.binaryObj = this.provider.get("myImpl"); }
"myImpl" is the hint for the StorageItem which you may define.
The next step is of course to add versioning support but I think this will provide a solid foundation to build that on.
WDYT?
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
What I'm planning for now is to continue developing xwiki-blob and begin developing xwiki-blob-attachment, a module which depends on xwiki-blob and xwiki-core and provides an AttachmentStore based on xwiki-blob so then we may choose to: 1. Integrate xwiki-blob-attachment into xwiki-core 2. Distribute xwiki-blob and xwiki-blob-attachment as an extension. 3. Integrate xwiki-blob and xwiki-blob-attachment as they are. WDYT? Caleb On 10/28/2010 09:54 AM, Caleb James DeLisle wrote:
``this module would relate to the future storage module.'' Is there an agreed upon future path for XWiki storage? If so I wasn't aware of it.
I designed the module to be as flexible and easy to use as possible. Storage in an SQL or JCR database (or anywhere else) is possible by implementing StorageItem.
This uses UUID as the key for loading the content simply because it's easy to manage and binary data won't be subject to the types of queries with which UUIDs perform poorly. How content is ID tagged at the low level is something which deserves discussion though.
I see large content such as attachments moving away from the database and toward the filesystem in the future because of better performance so I think persistance of large binary objects will remain a seperate concern for some time to come.
I understand that versioning and large item support is promised by JCR and although I am not against JCR if it performs well, I see a tremendous amount of momentum for hibernate+SQL in the code and among the users and I think writing a drop in replacement for hibernate would be a task of such difficulty and expense as to be paralleled only by that of building a community around it. I designed xwiki-blob to be extensible to new and different storage platforms but to implement the features which we need now rather than waiting for a new storage engine.
This answers the question of how this module would relate to how I expect the future storage will turn out. Does that answer your question?
Caleb
On 10/28/2010 08:03 AM, Vincent Massol wrote:
Hi Caleb,
I haven't looked at the code yet but could you provide the architectural vision going forward and how this module would relate to the future storage module. Also imagining we use JCR as the storage engine and/or API, how would this module integrate with it?
Thanks -Vincent
On Oct 28, 2010, at 1:45 PM, Caleb James DeLisle wrote:
In light of the need for better attachment handling, I spent the past few days working on the basic framework for a new module to support large objects. The code can be viewed here: http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-blob/src/main/java/...
Use case #1: Multiple methods of storing content and the ability to define more.
Use case #2: Thread safe locking to prevent concurrent modifications from damaging content.
Use case #3: ACID commit/rollback protection.
The API user has 2 access points. BinaryObject is a thread safe interface with ACID and option of using OutputStream or InputStream to set and get data.
StorageItem is a simplistic interface modeled after a small subset of the methods in java.io.File.
If you create a component which implements StorageItem, BinaryObjectProvider will allow you to access a BinaryObject which uses your StorageItem for persistent storage.
Example:
@Requirement private BinaryObjectProvider provider;
private BinaryObject binaryObj;
public void initialize() { this.binaryObj = this.provider.get("myImpl"); }
"myImpl" is the hint for the StorageItem which you may define.
The next step is of course to add versioning support but I think this will provide a solid foundation to build that on.
WDYT?
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
participants (2)
-
Caleb James DeLisle -
Vincent Massol