Re: [xwiki-devs] [discussion] Filesystem vs. database attachment storage. Re: [proposal] New database table binarychunk, interface c.x.x.store.BinaryStore, and interface c.x.x.doc.BinaryObject

On 10/20/2010 05:33 AM, Ludovic Dubost wrote: They are available, they require use of the blob type so we would have to add a column. I was warned about incompatibility issues. I understand that mysql and mssql stream the content back onto the heap before saving which ruins any memory savings. Postgres seems to support blobs but I was warned about strange issues, specifically this: http://in.relation.to/Bloggers/PostgreSQLAndBLOBs I was told that oracle has the best streaming support but I also read that oracle blob support requires the use of proprietary api. This is why I had opted for chinking the data instead of using blobs.

I had planned on committing the transaction only after all chunks are saved, if the database has memory issues with large commits, another possibility would be to verify after saving and throw an exception if that fails.

This is why I favor a generic BinaryStore rather than a change to XWikiHibernateAttachmentStore. Another issue which will have to be addressed is the memory consumption of JRCS for AttachmentArchive.

``little access to the database configuration'' This may also mean the xwiki user does not have permission to change the setting at runtime.

The fix to http://jira.xwiki.org/jira/browse/XWIKI-5405 has changed attachments so that the content and meta data are all saved in a single transaction and http://jira.xwiki.org/jira/browse/XWIKI-5474 prevents documents from being cached on save so we should have no more attachments which dissapear when the cache is purged.

Sounds like a test ;)

Caleb > Ludovic > > Le 18/10/10 19:55, Caleb James DeLisle a écrit : >> I talked with the Hibernate people about using streams and was told that it is not supported by all >> databases. >> >> As an alternative to the proposal below I would like to propose a filesystem based storage >> mechanism. >> The main advantage of using the database to store everything is that administrators need only use >> mysql_dump and they have their entire wiki backed up. >> >> If we are to abandon that requirement, we can have much faster attachment storage by using the >> filesystem. For this, I propose BinaryStore interface remains the same but >> com.xpn.xwiki.doc.BinaryObject would contain: >> >> void addContent(InputStream content) >> >> OutputStream addContent() >> >> void clear() >> >> InputStream getContent() >> >> void getContent(OutputStream writeTo) >> >> clear() would clear the underlying file whereas addContent would always append to it. >> >> >> The added column would look like this: >> >> <class name="com.xpn.xwiki.store.doc.FilesystemBinaryObject" table="filesystembinaryobject"> >> <id name="id" column="id"> >> <generator class="native" /> >> </id> >> >> <property name="fileURI" type="string"> >> <column name="fileuri" length="255" not-null="true"/> >> </property> >> </class> >> >> >> This would as with the original proposal be useful for not only storing attachments but attachment >> history, deleted attachments and even document history or deleted documents. >> >> >> WDYT? >> >> Caleb >> >> >> On 10/15/2010 04:21 PM, Caleb James DeLisle wrote: >>> Because the storage of large attachments is limited by database constraints and the fact that the >>> JDBC does not allow us to stream content out of the database, I propose we add a new database >>> table >>> binarychunk. >>> >>> The mapping will read as follows: >>> >>> <class name="com.xpn.xwiki.store.hibernate.HibernateBinaryStore$BinaryChunk" table="binarychunk"> >>> <composite-id unsaved-value="undefined"> >>> <key-property name="id" column="id" type="integer" /> >>> <key-property name="chunkNumber" column="chunknumber" type="integer" /> >>> </composite-id> >>> >>> <property name="content" type="binary"> >>> <column name="content" length="983040" not-null="true"/> >>> </property> >>> </class> >>> >>> Notice the maximum length (983040 bytes) is a number which is divisible by many common buffer >>> sizes >>> and is slightly less than the default max_packet_size in mysql which means that using the >>> xwikibinary table, we could store attachments of arbitrary size without hitting mysql default >>> limits. >>> >>> >>> com.xpn.xwiki.store.BinaryStore will contain: >>> >>> @param toLoad a binary object with an id number set, will be loaded. >>> void loadObject(BinaryObject toLoad) >>> >>> @param toStore a binary object, if no id is present then it will be given one upon successful >>> store, if id is present then that id number will be used. >>> void storeObject(BinaryObject toStore) >>> >>> This will be implemented by: com.xpn.xwiki.store.hibernate.HibernateBinaryStore >>> >>> >>> com.xpn.xwiki.doc.BinaryObject will contain: >>> >>> void setContent(InputStream content) >>> >>> OutputStream setContent() >>> >>> InputStream getContent() >>> >>> void getContent(OutputStream writeTo) >>> >>> Note: The get function and set functions will be duplicated with input or output streams to >>> maximize >>> ease of use. >>> >>> This will be implemented by com.xpn.xwiki.doc.TempFileBinaryObject which will store the binary >>> content in a temporary FileItem (see Apache commons fileupload). >>> >>> >>> >>> + This will be able to provide a back end for not only attachment content, but for attachment >>> archive and document archive if it is so desired. >>> + I have no intent of exposing it as public API at the moment. >>> >>> >>> WDYT? >>> >>> Caleb >>> >>> _______________________________________________ >>> devs mailing list >>> devs(a)xwiki.org >>> http://lists.xwiki.org/mailman/listinfo/devs >>> >> _______________________________________________ >> devs mailing list >> devs(a)xwiki.org >> http://lists.xwiki.org/mailman/listinfo/devs >>