[xwiki-devs] State of filesystem attachment storage.
Hello, Here's an update on the current state of new filesystem attachment storage: There are a couple platforms on which attachment storage sits: * Transaction handling - This is finished to my satisfaction. We may decide later that it is inadequate or needs changing but it serves this purpose now. * Transaction safe file save/delete - Finished (based on transaction handling) * Serialization - Finished refactoring of XMLWriter and generic Serializer and XMLSerializer. * Providing of files - Mostly finished, needs review to see if there is a cleaner way. Barring any major changes, this should not take longer than 1 day to complete. * Locking - Needed, lock management exists but does not handle deadlock nor "greedy thread" situations. 3 days needed to do code review on Apache commons LockManager to determine if it handles these situations, I'm guessing 2 days are needed to integrate LockManager and in place of the existing lock mechanism and a further 3 days to implement one of the features on top of LockManager if need be. "Consumer" code: * FilesystemAttachmentStore - Complete, enabling requires adding the .jar file to WEB-INF and changing a line in xwiki.cfg. Suffers from deadlock because the locking mechanism, is inadequate. * FilesystemAttachmentVersioningStore - Complete but needs tests, enabling is same as for FilesystemAttachmentStore. 1-2 days should be adequate for testing. * FilesystemAttachmentRecycleBinStore - in progress, 3 days to complete and test (?). Finished DeletedFilesystemAttachment, need TransactionRunnables for saving and deleting and serializer for deleted attachment meta-data. * Need a plan for migrating data from old system. FilesystemAttachmentStore fails over to old store if an attachment cannot be found. Is this correct? Should migration scripts be distributed instead? WDYT? * Consumer code all needs more tests. Any amount of time could be spent on this, I think a week is adequate. Since this contains a large amount of code and all storage code is critical, I think this should go through a release cycle in the platform but disabled by default so that it can be beta tested before it becomes official. WDYT? Caleb
Le 26 janv. 2011 à 18:59, Caleb James DeLisle a écrit :
Hello, Here's an update on the current state of new filesystem attachment storage:
This is real cool.
[...]
* Serialization - Finished refactoring of XMLWriter and generic Serializer and XMLSerializer.
Is this actually exporting the content in a separate file? And is the extension of this file somewhat configurable (at least .grv, .html, or .vm) ? The XML deserialization would then use XInclude. The interest is that it would allow back-end changes and/or svn commits and updates in a way that is compatible with IDEs.
[...] Since this contains a large amount of code and all storage code is critical, I think this should go through a release cycle in the platform but disabled by default so that it can be beta tested before it becomes official. WDYT?
To me this sounds like the best approach. paul
On 01/26/2011 02:10 PM, Paul Libbrecht wrote:
Le 26 janv. 2011 à 18:59, Caleb James DeLisle a écrit :
Hello, Here's an update on the current state of new filesystem attachment storage:
This is real cool.
[...]
* Serialization - Finished refactoring of XMLWriter and generic Serializer and XMLSerializer.
Is this actually exporting the content in a separate file? And is the extension of this file somewhat configurable (at least .grv, .html, or .vm) ? The XML deserialization would then use XInclude. The interest is that it would allow back-end changes and/or svn commits and updates in a way that is compatible with IDEs.
Yes, the attachment content goes into a file named whatever the attachment is named. It will be something like this: <storage dir>/xwiki/Sandbox/Test/~this/attachments/file.txt/file.txt The metadata such as date, uploader, and version will be made into an XML file by the serializer. That will look something like this: <?xml version="1.0" encoding="UTF-8"?> <attachment serializer="attachment-meta/1.0"> <filename>file1</filename> <filesize>10</filesize> <author>me</author> <version>1.1</version> <comment>something whitty</comment> <date>1293045632168</date> </attachment> serializer="attachment-meta/1.0" That is the component lookup name of the serializer which makes way for future xml serializer implementations. Versions of the file will be in the same directory: The naming rule is as follows: The filename is URL encoded and the version has "~v" prepended so that it cannot be mistaken for part of the filename. If the filename contains one or more '.' characters then the version is inserted before the last '.' character. Otherwise it is appended to the file name. This means a file such as: file.txt version 1.1 will become file~v1.1.txt and will still be recognized by a text editor A file with no extension such as myUnknownFile version 1.1 will become myUnknownFile~v1.1 Because of URL encoding, a file named file~v1.3.txt of version 1.1 will become file%7Ev1.3~1.1.txt and thus will not collide with file.txt version 1.1. Caleb
[...] Since this contains a large amount of code and all storage code is critical, I think this should go through a release cycle in the platform but disabled by default so that it can be beta tested before it becomes official. WDYT?
To me this sounds like the best approach.
paul _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Caleb, My question below was about the page content. (in the future it might apply to several other fields of objects, such as panel source). paul Le 26 janv. 2011 à 21:21, Caleb James DeLisle a écrit :
On 01/26/2011 02:10 PM, Paul Libbrecht wrote:
Le 26 janv. 2011 à 18:59, Caleb James DeLisle a écrit :
Hello, Here's an update on the current state of new filesystem attachment storage:
This is real cool.
[...]
* Serialization - Finished refactoring of XMLWriter and generic Serializer and XMLSerializer.
Is this actually exporting the content in a separate file? And is the extension of this file somewhat configurable (at least .grv, .html, or .vm) ? The XML deserialization would then use XInclude. The interest is that it would allow back-end changes and/or svn commits and updates in a way that is compatible with IDEs.
Yes, the attachment content goes into a file named whatever the attachment is named. It will be something like this: <storage dir>/xwiki/Sandbox/Test/~this/attachments/file.txt/file.txt The metadata such as date, uploader, and version will be made into an XML file by the serializer. That will look something like this: <?xml version="1.0" encoding="UTF-8"?> <attachment serializer="attachment-meta/1.0"> <filename>file1</filename> <filesize>10</filesize> <author>me</author> <version>1.1</version> <comment>something whitty</comment> <date>1293045632168</date> </attachment>
serializer="attachment-meta/1.0" That is the component lookup name of the serializer which makes way for future xml serializer implementations.
Versions of the file will be in the same directory: The naming rule is as follows: The filename is URL encoded and the version has "~v" prepended so that it cannot be mistaken for part of the filename. If the filename contains one or more '.' characters then the version is inserted before the last '.' character. Otherwise it is appended to the file name. This means a file such as: file.txt version 1.1 will become file~v1.1.txt and will still be recognized by a text editor A file with no extension such as myUnknownFile version 1.1 will become myUnknownFile~v1.1 Because of URL encoding, a file named file~v1.3.txt of version 1.1 will become file%7Ev1.3~1.1.txt and thus will not collide with file.txt version 1.1.
Caleb
[...] Since this contains a large amount of code and all storage code is critical, I think this should go through a release cycle in the platform but disabled by default so that it can be beta tested before it becomes official. WDYT?
To me this sounds like the best approach.
paul _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
I see. this is still far away from decision but in my own opinion: 1. Document content should be in a separate file where it is easily accessible, the file extension would probably have to be standardized to something like .xwiki2 2. document content, metadata and objects should all be integrated in the same directory structure. This will allow for easy storage, backup, and define an alternative and more robust import/export format. Caleb On 01/26/2011 03:37 PM, Paul Libbrecht wrote:
Caleb,
My question below was about the page content. (in the future it might apply to several other fields of objects, such as panel source).
paul
Le 26 janv. 2011 à 21:21, Caleb James DeLisle a écrit :
On 01/26/2011 02:10 PM, Paul Libbrecht wrote:
Le 26 janv. 2011 à 18:59, Caleb James DeLisle a écrit :
Hello, Here's an update on the current state of new filesystem attachment storage:
This is real cool.
[...]
* Serialization - Finished refactoring of XMLWriter and generic Serializer and XMLSerializer.
Is this actually exporting the content in a separate file? And is the extension of this file somewhat configurable (at least .grv, .html, or .vm) ? The XML deserialization would then use XInclude. The interest is that it would allow back-end changes and/or svn commits and updates in a way that is compatible with IDEs.
Yes, the attachment content goes into a file named whatever the attachment is named. It will be something like this: <storage dir>/xwiki/Sandbox/Test/~this/attachments/file.txt/file.txt The metadata such as date, uploader, and version will be made into an XML file by the serializer. That will look something like this: <?xml version="1.0" encoding="UTF-8"?> <attachment serializer="attachment-meta/1.0"> <filename>file1</filename> <filesize>10</filesize> <author>me</author> <version>1.1</version> <comment>something whitty</comment> <date>1293045632168</date> </attachment>
serializer="attachment-meta/1.0" That is the component lookup name of the serializer which makes way for future xml serializer implementations.
Versions of the file will be in the same directory: The naming rule is as follows: The filename is URL encoded and the version has "~v" prepended so that it cannot be mistaken for part of the filename. If the filename contains one or more '.' characters then the version is inserted before the last '.' character. Otherwise it is appended to the file name. This means a file such as: file.txt version 1.1 will become file~v1.1.txt and will still be recognized by a text editor A file with no extension such as myUnknownFile version 1.1 will become myUnknownFile~v1.1 Because of URL encoding, a file named file~v1.3.txt of version 1.1 will become file%7Ev1.3~1.1.txt and thus will not collide with file.txt version 1.1.
Caleb
[...] Since this contains a large amount of code and all storage code is critical, I think this should go through a release cycle in the platform but disabled by default so that it can be beta tested before it becomes official. WDYT?
To me this sounds like the best approach.
paul _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Le 26 janv. 2011 à 22:05, Caleb James DeLisle a écrit :
I see. this is still far away from decision but in my own opinion: 1. Document content should be in a separate file where it is easily accessible, the file extension would probably have to be standardized to something like .xwiki2
I would call it .xwiki2 or so unless there's a special object that would enable this to be different. I would make it become .vm and .grv many many times.
2. document content, metadata and objects should all be integrated in the same directory structure. This will allow for easy storage, backup, and define an alternative and more robust import/export format.
To me the immediate interest is to make such a set of directory, or part of that, be crawlable by an IDE (e.g. my dear IntelliJ IDEA telling me all the groovy and velocity calls to my java objects). paul
On Wed, Jan 26, 2011 at 11:23 PM, Paul Libbrecht <[email protected]> wrote:
Le 26 janv. 2011 à 22:05, Caleb James DeLisle a écrit :
I see. this is still far away from decision but in my own opinion: 1. Document content should be in a separate file where it is easily accessible, the file extension would probably have to be standardized to something like .xwiki2
I would call it .xwiki2 or so unless there's a special object that would enable this to be different. I would make it become .vm and .grv many many times.
2. document content, metadata and objects should all be integrated in the same directory structure. This will allow for easy storage, backup, and define an alternative and more robust import/export format.
To me the immediate interest is to make such a set of directory, or part of that, be crawlable by an IDE (e.g. my dear IntelliJ IDEA telling me all the groovy and velocity calls to my java objects).
This thread becomes relevant to my interests :) We definitely need to define a standard structure to represent a XWiki document on a filesystem. It could then also be used for example for a WebDAV implementation, so that you can benefit from your favorite IDE features but keeping the DB storage. IMO this would beat the XEclipse approach in the short term, as the barrier to start using your IDE for everything XWiki would be far lower (vs. in XEclipse you have to implement an object editor that knows that a XWiki.StyleSheetExtension is CSS etc.) Then the next step is to have syntax highlighting for .xwiki20 files for popular editors (Eclipse, textmate, vim, IDEA, etc.) Jerome.
paul _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Caleb, Thanks for the detailed information. So if my count is ok there's work left for a minimum of 21 men/days to finish it. We have 6 days till 3.0M2, 10 more for RC1 and 10 more for final, so a total of 26 days. However if we want to include it in 3.0 final (and we do want this since it's in our roadmap :)), I suggest that you'd need to have something usable (with known issues/limitations) in XE 3.0M2, planned for the 7th of February 2011. Is that doable on your side? The idea is to let users start using it and provide feedback and find potential issues which we could fix in 3.0 RC1 and RC2. I'm also +1 to move your work in platform ASAP (you need to start an official vote for this in a separate thread). Could you explain where it would go? Would you suggest to have it all go in platform/core/xwiki-store module? Does it have deps on the old xwiki-core? Thanks -Vincent On Jan 26, 2011, at 6:59 PM, Caleb James DeLisle wrote:
Hello, Here's an update on the current state of new filesystem attachment storage: There are a couple platforms on which attachment storage sits: * Transaction handling - This is finished to my satisfaction. We may decide later that it is inadequate or needs changing but it serves this purpose now. * Transaction safe file save/delete - Finished (based on transaction handling) * Serialization - Finished refactoring of XMLWriter and generic Serializer and XMLSerializer. * Providing of files - Mostly finished, needs review to see if there is a cleaner way. Barring any major changes, this should not take longer than 1 day to complete. * Locking - Needed, lock management exists but does not handle deadlock nor "greedy thread" situations. 3 days needed to do code review on Apache commons LockManager to determine if it handles these situations, I'm guessing 2 days are needed to integrate LockManager and in place of the existing lock mechanism and a further 3 days to implement one of the features on top of LockManager if need be.
"Consumer" code: * FilesystemAttachmentStore - Complete, enabling requires adding the .jar file to WEB-INF and changing a line in xwiki.cfg. Suffers from deadlock because the locking mechanism, is inadequate. * FilesystemAttachmentVersioningStore - Complete but needs tests, enabling is same as for FilesystemAttachmentStore. 1-2 days should be adequate for testing. * FilesystemAttachmentRecycleBinStore - in progress, 3 days to complete and test (?). Finished DeletedFilesystemAttachment, need TransactionRunnables for saving and deleting and serializer for deleted attachment meta-data. * Need a plan for migrating data from old system. FilesystemAttachmentStore fails over to old store if an attachment cannot be found. Is this correct? Should migration scripts be distributed instead? WDYT? * Consumer code all needs more tests. Any amount of time could be spent on this, I think a week is adequate.
Since this contains a large amount of code and all storage code is critical, I think this should go through a release cycle in the platform but disabled by default so that it can be beta tested before it becomes official. WDYT?
Caleb
On 01/27/2011 07:26 AM, Vincent Massol wrote:
Hi Caleb,
Thanks for the detailed information.
So if my count is ok there's work left for a minimum of 21 men/days to finish it. We have 6 days till 3.0M2, 10 more for RC1 and 10 more for final, so a total of 26 days.
However if we want to include it in 3.0 final (and we do want this since it's in our roadmap :)), I suggest that you'd need to have something usable (with known issues/limitations) in XE 3.0M2, planned for the 7th of February 2011. Is that doable on your side? The idea is to let users start using it and provide feedback and find potential issues which we could fix in 3.0 RC1 and RC2.
I could shift my attention back to FilesystemAttachmentStore and FilesystemAttachmentVersioningStore so that people can use the code and avoid memory exhaustion by disabling the attachment recycle bin (which is not currently available the user interface IIRC). Locking is the only part which critically needs attention for that path. WDYT?
I'm also +1 to move your work in platform ASAP (you need to start an official vote for this in a separate thread). Could you explain where it would go? Would you suggest to have it all go in platform/core/xwiki-store module? Does it have deps on the old xwiki-core?
Yes, platform/core/xwiki-store module is the path I have so far taken. xwiki-store/xwiki-store-filesystem-attachments depends on the old core. xwiki-store/xwiki-store-filesystem, xwiki-store/xwiki-store-serialization, xwiki-store/xwiki-store-transaction and xwiki-store-api do not.
Thanks -Vincent
On Jan 26, 2011, at 6:59 PM, Caleb James DeLisle wrote:
Hello, Here's an update on the current state of new filesystem attachment storage: There are a couple platforms on which attachment storage sits: * Transaction handling - This is finished to my satisfaction. We may decide later that it is inadequate or needs changing but it serves this purpose now. * Transaction safe file save/delete - Finished (based on transaction handling) * Serialization - Finished refactoring of XMLWriter and generic Serializer and XMLSerializer. * Providing of files - Mostly finished, needs review to see if there is a cleaner way. Barring any major changes, this should not take longer than 1 day to complete. * Locking - Needed, lock management exists but does not handle deadlock nor "greedy thread" situations. 3 days needed to do code review on Apache commons LockManager to determine if it handles these situations, I'm guessing 2 days are needed to integrate LockManager and in place of the existing lock mechanism and a further 3 days to implement one of the features on top of LockManager if need be.
"Consumer" code: * FilesystemAttachmentStore - Complete, enabling requires adding the .jar file to WEB-INF and changing a line in xwiki.cfg. Suffers from deadlock because the locking mechanism, is inadequate. * FilesystemAttachmentVersioningStore - Complete but needs tests, enabling is same as for FilesystemAttachmentStore. 1-2 days should be adequate for testing. * FilesystemAttachmentRecycleBinStore - in progress, 3 days to complete and test (?). Finished DeletedFilesystemAttachment, need TransactionRunnables for saving and deleting and serializer for deleted attachment meta-data. * Need a plan for migrating data from old system. FilesystemAttachmentStore fails over to old store if an attachment cannot be found. Is this correct? Should migration scripts be distributed instead? WDYT? * Consumer code all needs more tests. Any amount of time could be spent on this, I think a week is adequate.
Since this contains a large amount of code and all storage code is critical, I think this should go through a release cycle in the platform but disabled by default so that it can be beta tested before it becomes official. WDYT?
Caleb
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Jan 27, 2011, at 3:17 PM, Caleb James DeLisle wrote:
On 01/27/2011 07:26 AM, Vincent Massol wrote:
Hi Caleb,
Thanks for the detailed information.
So if my count is ok there's work left for a minimum of 21 men/days to finish it. We have 6 days till 3.0M2, 10 more for RC1 and 10 more for final, so a total of 26 days.
However if we want to include it in 3.0 final (and we do want this since it's in our roadmap :)), I suggest that you'd need to have something usable (with known issues/limitations) in XE 3.0M2, planned for the 7th of February 2011. Is that doable on your side? The idea is to let users start using it and provide feedback and find potential issues which we could fix in 3.0 RC1 and RC2.
I could shift my attention back to FilesystemAttachmentStore and FilesystemAttachmentVersioningStore so that people can use the code and avoid memory exhaustion by disabling the attachment recycle bin (which is not currently available the user interface IIRC). Locking is the only part which critically needs attention for that path. WDYT?
Sounds good to me, better have something operation that can be tested than nothing!
I'm also +1 to move your work in platform ASAP (you need to start an official vote for this in a separate thread). Could you explain where it would go? Would you suggest to have it all go in platform/core/xwiki-store module? Does it have deps on the old xwiki-core?
Yes, platform/core/xwiki-store module is the path I have so far taken.
xwiki-store/xwiki-store-filesystem-attachments depends on the old core. xwiki-store/xwiki-store-filesystem, xwiki-store/xwiki-store-serialization, xwiki-store/xwiki-store-transaction and xwiki-store-api do not.
Could you send a vote mail on this? Thanks -Vincent
Thanks -Vincent
On Jan 26, 2011, at 6:59 PM, Caleb James DeLisle wrote:
Hello, Here's an update on the current state of new filesystem attachment storage: There are a couple platforms on which attachment storage sits: * Transaction handling - This is finished to my satisfaction. We may decide later that it is inadequate or needs changing but it serves this purpose now. * Transaction safe file save/delete - Finished (based on transaction handling) * Serialization - Finished refactoring of XMLWriter and generic Serializer and XMLSerializer. * Providing of files - Mostly finished, needs review to see if there is a cleaner way. Barring any major changes, this should not take longer than 1 day to complete. * Locking - Needed, lock management exists but does not handle deadlock nor "greedy thread" situations. 3 days needed to do code review on Apache commons LockManager to determine if it handles these situations, I'm guessing 2 days are needed to integrate LockManager and in place of the existing lock mechanism and a further 3 days to implement one of the features on top of LockManager if need be.
"Consumer" code: * FilesystemAttachmentStore - Complete, enabling requires adding the .jar file to WEB-INF and changing a line in xwiki.cfg. Suffers from deadlock because the locking mechanism, is inadequate. * FilesystemAttachmentVersioningStore - Complete but needs tests, enabling is same as for FilesystemAttachmentStore. 1-2 days should be adequate for testing. * FilesystemAttachmentRecycleBinStore - in progress, 3 days to complete and test (?). Finished DeletedFilesystemAttachment, need TransactionRunnables for saving and deleting and serializer for deleted attachment meta-data. * Need a plan for migrating data from old system. FilesystemAttachmentStore fails over to old store if an attachment cannot be found. Is this correct? Should migration scripts be distributed instead? WDYT? * Consumer code all needs more tests. Any amount of time could be spent on this, I think a week is adequate.
Since this contains a large amount of code and all storage code is critical, I think this should go through a release cycle in the platform but disabled by default so that it can be beta tested before it becomes official. WDYT?
Caleb
participants (4)
-
Caleb James DeLisle -
Jerome Velociter -
Paul Libbrecht -
Vincent Massol