Re: [xwiki-users] Hibernate exception when trying to delete attachment

2009/3/19 Stefan Woehrer &lt;stefan_woehrer(a)yahoo.de&gt; I believe this is another instance of the :"can't delete oversize attachments" issue that I saw in 1.8. What happens is that past a certain size limit ( smaller than the default setting per /xwiki/bin/edit/XWiki/XWikiPreferences?editor=object --> *XWiki.XWikiPreferences[0]* -> "Maximum Upload Size: 33554432" ) the full attachment gets stored (if there's RAM and/or swap to hold it all in it's hugely-exploded-for-storage-in-java-format). However, despite actually storing successfully, these oversize files cause something in the sql->hibernate->java->xwiki->html layers to die; the UI showing upload completion never appears. After that, the oversize file can't be deleted either, because it invokes the same problem somewhere in the sql->hibernate->java->xwiki->html layers. There's several problems here.. first is that there needs to be a way of incrementally uploading files without doing it all as a "batch" operation. This could avoid the exploding-file-data problem from affecting java/hibernate layers, however, it would require a change in the attachments table to allow better incremental update of the uploaded data as it comes in over the network. Second, having an attachment recycle bin may not be a very good idea in the first place -- it forces copying of data betweeen tables, causes extra data-copying, invoking the large-file bug (disable the attachment recyclebin in xwiki.cfg and avoid the deletion issue entirely -- IMHO it should be so as default to avoid these bugs.). Alternately, the entire issue could be avoided with a pluggable storage module that allows regular files and uses the underlying filesystem directly for storing attachments, large files, media, video, etc. Because the attachments tables can get rather unwieldy in a database when they're storing large files -- to the point where backup etc becomes an issue. For more info: Date: Mon, 9 Mar 2009 10:11:52 -0800 Subject: Re: [xwiki-users] Bug : unable to delete attached files in virtualwikis From: Niels Mayer <nielsmayer at gmail.com> To: XWiki Users &lt;users(a)xwiki.org&gt; Christophe -- Thanks for testing this. Your confirmation suggests I will follow your lead. Unfortunately, I only noticed the xwiki.cfg options that control this after i deleted them directly from the database in SQL: Here are the xwiki.cfg settings I'm using to achieve this: #-# Whether the document recycle bin feature is activated or not # xwiki.recyclebin=1 #-# Whether the attachment recycle bin feature is activated or not storage.attachment.recyclebin=0 #-# Whether the document versioning feature is activated or not # xwiki.store.versioning=1 #-# Whether the attachment versioning feature is activated or not xwiki.store.attachment.versioning=0 #-# Whether the attachments should also be rolled back when a document is reverted. xwiki.store.rollbackattachmentwithdocuments=0 It appears that xwiki.cfg setting xwiki.store.attachment.versioning.hint=void is redundant with xwiki.store.attachment.versioning=0 Question for all: Is the latter the preferable way of turning off attachment versionining? Niels http://nielsmayer.com PS: doesn't it make sense to avoid such problems in the future and bypass the recycle bin for deleted attachments. Alternately, some kind of heuristic could be employed, as the default, e.g. the recylcle-bin is used to hold deleted attachments, except when the attachments exceed a certain size, say 1Mb. In that case the user is warned of permanent deletion and can "ok" it away. -------------------------------------------------------------------------------- Date: Sat, 7 Mar 2009 01:47:08 -0800 Subject: Re: [xwiki-users] Bug : unable to delete attached files in virtual wikis From: Niels Mayer <nielsmayer at gmail.com> To: XWiki Users &lt;users(a)xwiki.org&gt; I had a similar problem, with attachments not being deletable, which also prevented the document itself from being deleted. Unfortunately, this also suggested an additional issue arising with attachments. If a user makes large attachments to their user-document, e.g. XWiki.MyName then even in a "closed" wiki that only allows users to edit their own user document, they can cause a denial of service attack through large attachments. In other words, even if the wiki is setup so that regular registered XWikiAllGroup users don't have write access to any directory, just comment-access to documents; they'll still have write access to their own XWiki.MyName document created through registration... and that means they'll be able to add attachments even when you think they couldn't. The "Rights" adminstration checkbox "Prevent unregistered users from editing pages, regardless of the page or space rights" doesn't prevent attachments from being added to user's own pages. Anyways, I ended up solving the blowups and OOM errors by deleting the attachments directly in the database: mysql> select XWA_ID,XWA_DOC_ID,XWA_FILENAME,XWA_SIZE,XWA_AUTHOR from xwikiattachment where XWA_AUTHOR='XWiki.JG'; | XWA_ID | XWA_DOC_ID | XWA_FILENAME | XWA_SIZE | XWA_AUTHOR| | 1185703559 | 168880978 | foo.pdf | 9817587 | XWiki.JG | | -352107721 | 168880978 | bar.pdf | 13049680 | XWiki.JG | | 1527849923 | 168880978 | baz.pdf | 293 | XWiki.JG | | -2073884056 | 168880978 | frop.pdf | 5904061 | XWiki.JG | | 1039500510 | 168880978 | schlop.pdf | 4440028 | XWiki.JG | | 942569068 | 168880978 | plop.pdf | 14033466 | XWiki.JG | | 1363529635 | 168880978 | schnops.pdf | 0 | XWiki.JG | | -1054875919 | 168880978 | hops.jpg | 3081 | XWiki.JG | 8 rows in set (0.00 sec) mysql> delete from xwikiattachment where XWA_AUTHOR='XWiki.JG'; Query OK, 8 rows affected (0.00 sec) Someone please tell me if this is not the right way to fix such issues... As afterthought, I realized that setting xwiki.store.attachment.versioning.hint=3Dvoid in xwiki.cfg might prevent xwiki from reading/diffing and attempting to version a large attachment. Doing so could certainly make it run out of memory, and changing this option might prevent the bug?: #-# The attachment versioning storage. Use 'void' to disable attachment versioning. # xwiki.store.attachment.versioning.hint=default Also, perhaps by not sending deleted attachments to the recycle bin, it won't need to be copied anywhere, and therefore there'll not be any memory to run out copying a giant attachment through java (assuming that's what happens on delete). #-# The attachment recycle bin storage. # xwiki.store.attachment.recyclebin.hint=Ddefault Niels http://nielsmayer.com PS: perhaps the defaults for xwiki.store.attachment.versioning.hint and xwiki.store.attachment.recyclebin.hint need to be changed? On Fri, Mar 6, 2009 at 2:37 AM, PERINAUD Christophe < Christophe.PERINAUD(a)kbl-bank.com&gt; wrote: ed e re.java:1052) seStore.java:1098) cycleBin(HibernateAttachmentRecycleBinStore.java:83) va:72) Subject: Re: [xwiki-users] Bug : unable to delete attached files in virtualwikis From: Niels Mayer &lt;nielsmayer(a)gmail.com&gt; To: XWiki Users &lt;users(a)xwiki.org&gt; Content-Type: multipart/alternative; boundary=000e0cd21476a7301d0464b391b3 --000e0cd21476a7301d0464b391b3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Christophe -- Thanks for testing this. Your confirmation suggests I will follow your lead. Unfortunately, I only noticed the xwiki.cfg options that control this after i deleted them directly from the database in SQL: Here are the xwiki.cfg settings I'm using to achieve this: #-# Whether the document recycle bin feature is activated or not # xwiki.recyclebin=1 #-# Whether the attachment recycle bin feature is activated or not storage.attachment.recyclebin=0 #-# Whether the document versioning feature is activated or not # xwiki.store.versioning=1 #-# Whether the attachment versioning feature is activated or not xwiki.store.attachment.versioning=0 #-# Whether the attachments should also be rolled back when a document is reverted. xwiki.store.rollbackattachmentwithdocuments=0 It appears that xwiki.cfg setting xwiki.store.attachment.versioning.hint=void is redundant with xwiki.store.attachment.versioning=0 Question for all: Is the latter the preferable way of turning off attachment versionining? Niels http://nielsmayer.com PS: doesn't it make sense to avoid such problems in the future and bypass the recycle bin for deleted attachments. Alternately, some kind of heuristic could be employed, as the default, e.g. the recylcle-bin is used to hold deleted attachments, except when the attachments exceed a certain size, say 1Mb. In that case the user is warned of permanent deletion and can "ok" it away. -------------------------------------------------------------------------------- Niels http://nielsmayer.com