[xwiki-users] Yet another thread on file-based attachments
I have seen another threads, but I didn't noticed any good idea. I have only seen JCR project in the sandbox. http://dev.xwiki.org/xwiki/bin/view/Design/JcrStore I have no idea what is JCR. «JCR» sounds very different from «filesystem». Installing and maintaining JCR is likely to be yet another brainache I'd like to skip. Finally, «Translate XWQL to JCRSQL2» sounds very different from «simple and reliable». «Storing 10Mb octet streams in database» doesn't sound like being reliable, and indeed it caused strange problems recently. So my humble wish is to see xwikiattachment_content table replaced with a mere filesystem storage. Other tables left intact. Maintaing «Space/Document/Attachment» structure is not a requirement (althoung it would be nice since I could share it in GreyLink DC++ this way). -- If you want to get to the top, you have to start at the bottom
Ivan Levashew wrote:
I have seen another threads, but I didn't noticed any good idea. I have only seen JCR project in the sandbox.
http://dev.xwiki.org/xwiki/bin/view/Design/JcrStore
I have no idea what is JCR. «JCR» sounds very different from «filesystem». Installing and maintaining JCR is likely to be yet another brainache I'd like to skip. Finally, «Translate XWQL to JCRSQL2» sounds very different from «simple and reliable». JCR seems to be proposed as a replacement for the database entirely. I would have to see it's performance before I would want to advocate for the idea.
«Storing 10Mb octet streams in database» doesn't sound like being reliable, and indeed it caused strange problems recently. On the one hand it definitely over-stresses the database but on the other hand it is very nice to be able to do a database dump, wipe the disk, load the dump and be right back where you were before. Maybe a hybrid where content is stored in the db and 'cached' on the hard disk but caches make debugging a huge pain. Still something has to be done about the max_packet issue.
So my humble wish is to see xwikiattachment_content table replaced with a mere filesystem storage. Other tables left intact. Maintaing «Space/Document/Attachment» structure is not a requirement (althoung it would be nice since I could share it in GreyLink DC++ this way).
Caleb James DeLisle writes:
«Storing 10Mb octet streams in database» doesn't sound like being reliable, and indeed it caused strange problems recently. On the one hand it definitely over-stresses the database but on the other hand it is very nice to be able to do a database dump, wipe the disk, load the dump and be right back where you were before. Maybe a hybrid where content is stored in the db and 'cached' on the hard disk but caches make debugging a huge pain. Still something has to be done about the max_packet issue.
With regards to dumping, I'd like it to be done via FTP or WebDAV means. I don't know whether current WebDAV implemetation is suitable for dumping. Anyway, dumps maid that way are going to be more portable compared to e. g. MySQL dumps. Also, MySQL dump isn't browsable with tools like FAR Manager. XARs are portable and browsable. -- If you want to get to the top, you have to start at the bottom
I'm testing WebDAV now and getting lots of exceptions when copying photos from PhotoAlbum. There are entries like this one: Caused by: java.lang.OutOfMemoryError: Java heap space Web server isn't stable in these moments. javaw.exe consumes about 600Mb of memory. Although I can adjust it to use 3Gb of memory, this seems to be a wrong way. Storing big attachments in a database is a wrong way. Handling them in-memory is a wrong way. They should be handled in another manner. Some kind of event-based servlet streams attachments block by block without putting any whole octet stream in memory (it seems that stroring attachments in database implies this as a consequence). No matter what I (and you) do, wrong way will constantly show its wrongness and limitations. X-Wiki is far from being perfect yet indeed it's outstanding. I don't regret I have spent some time to get it working. I hope, current way of handling attachments will be obsoleted soon. -- If you want to get to the top, you have to start at the bottom
On Sat, Jun 12, 2010 at 11:02, Ivan Levashew <[email protected]> wrote:
I'm testing WebDAV now and getting lots of exceptions when copying photos from PhotoAlbum.
There are entries like this one:
Caused by: java.lang.OutOfMemoryError: Java heap space
Web server isn't stable in these moments.
javaw.exe consumes about 600Mb of memory. Although I can adjust it to use 3Gb of memory, this seems to be a wrong way. Storing big attachments in a database is a wrong way. Handling them in-memory is a wrong way. They should be handled in another manner. Some kind of event-based servlet streams attachments block by block without putting any whole octet stream in memory (it seems that stroring attachments in database implies this as a consequence).
No matter what I (and you) do, wrong way will constantly show its wrongness and limitations.
X-Wiki is far from being perfect yet indeed it's outstanding. I don't regret I have spent some time to get it working. I hope, current way of handling attachments will be obsoleted soon.
+1, storing attachments in the DB is a pretty bad idea. Not to mention that storing large objects in MySQL restrains from using InnoDB.
-- If you want to get to the top, you have to start at the bottom
_______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users
-- Je prête mes livres sur Bibale, http://www.bibale.com/bibale/-/user/hbs
On Jun 12, 2010, at 6:55 AM, Caleb James DeLisle wrote:
Ivan Levashew wrote:
I have seen another threads, but I didn't noticed any good idea. I have only seen JCR project in the sandbox.
http://dev.xwiki.org/xwiki/bin/view/Design/JcrStore
I have no idea what is JCR. «JCR» sounds very different from «filesystem». Installing and maintaining JCR is likely to be yet another brainache I'd like to skip. Finally, «Translate XWQL to JCRSQL2» sounds very different from «simple and reliable». JCR seems to be proposed as a replacement for the database entirely.
This is not correct. JCR is simply a Java API and an SPI for implementers to plug any storage. There are SPI implementations for filesystem, RDBMS, and more. If you prefer it serves a similar purpose as Hibernate but in a standard way and more generic since it supports any type of storage. See http://en.wikipedia.org/wiki/Content_repository_API_for_Java for example.
I would have to see it's performance before I would want to advocate for the idea.
The only potential problem re performance is the mapping between the java objects and the tables if you choose the RDBMS SPI. We'll need to compare the performance with our current Hibernate solution indeed.
«Storing 10Mb octet streams in database» doesn't sound like being reliable, and indeed it caused strange problems recently. On the one hand it definitely over-stresses the database but on the other hand it is very nice to be able to do a database dump, wipe the disk, load the dump and be right back where you were before. Maybe a hybrid where content is stored in the db and 'cached' on the hard disk but caches make debugging a huge pain. Still something has to be done about the max_packet issue.
So my humble wish is to see xwikiattachment_content table replaced with a mere filesystem storage. Other tables left intact. Maintaing «Space/Document/Attachment» structure is not a requirement (althoung it would be nice since I could share it in GreyLink DC++ this way).
Actually we have a separate Attachment storage (there's an interface called XWikiAttachmentStoreInterface) and it should be pretty easy to implement a file-based implementation of it. I wonder why nobody has done it before.... Thanks -Vincent
participants (4)
-
Caleb James DeLisle -
Ivan Levashew -
Mathias Herberts -
Vincent Massol