New subject: [xwiki-users] Fwd: Re: XWiki big file attachments

23 Oct 2015

I runned tomcat8 with hsqldb latest 7.3-1 xwiki release and uploaded a 
700mb file.
Java had 8gb heap and I still got a out of memory exception.

As I read about using databases for filestorage, I now found out
that this is a enormous task, becouse big databases are difficult
to maintain. So I stick with filesystem storage for now.

One solution is to break up big files into chunks, and store them
as parts within a database.

An interresting bug I encountered was:
If I changed the database from hibernate to file, I still do see the old 
file attachment references,
but I am not able to delete them.
Same for virtual wikis who are created before you change the file 
storage from hibernate to file.

All of those wikis have broken images.

It would be a great idea (at least from my point of view)
to set a option for "delete attachment references anyway (even if file 
not found in db)"
and "Reinstall virtual wiki".

One working hack however is, to export everything, change the database type,
reninstall the wiki (delete catalina directories etc.) and then import the
backup with a proper file database configuration.

The import process took arround 3 minutes for 900 pages.

Ps.: Conflu* needs arround 10 minutes to move 1000 pages (Crashes 
included), so great work so far ;)

Greetings, Louis

Am 23.10.2015 um 10:11 schrieb vincent(a)massol.net:
...
  Hi Louis

 On 22 Oct 2015 at 23:14:05, Louis Höfler 
 (louis.hoefler@mathematek.de(mailto:louis.hoefler@mathematek.de)) wrote:

  That perfectly solves my problem.

 As a point of interest,
 would it be possible to store big files within mysql too,
 or is this just impossible? 
 It’s possible with very large memory settings. There are several 
 reasons why it takes a lot of memory but one reason is that AFAIR 
 mysql doesn’t have a streaming api for blobs and thus the whole 
 attachments needs to be fully in memory. In addition our code does 
 manipulation of the attachment in memory, requiring even more memory. 
 At some point we computed the memory required and it was like 27 times 
 or so the size of the attachment but we improved it and I don’t know 
 what’s the requirement now (would be interesting to find out).

 If you’re interested by details
 * http://jira.xwiki.org/browse/XWIKI-8910
 * http://jira.xwiki.org/browse/XWIKI-11209
 * http://jira.xwiki.org/browse/XWIKI-10611
 * http://jira.xwiki.org/browse/XWIKI-8189

 Thanks
 -Vincent

 > Thank you, Louis
 >
 > Am 22.10.2015 um 22:47 schrieb vincent(a)massol.net:
 > > Hi Louis,
 > >
 > > Please check
 > > http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Attachments and
 > > let us know if it solves your problems (hint: configure xwiki to use
 > > filesystem if you wish to store large attachments).
 > >
 > > Thanks
 > > -Vincent
 > >
 > > On 22 Oct 2015 at 22:42:33, Louis Höfler (louis.hoefler(a)mathematek.de
 > > ) wrote:
 > >
 > >> Hello Everyone,
 > >> i currently try to upload big files as attachment.
 > >>
 > >> I run mysql 5 with xwiki within tomcat 8 on debian.
 > >>
 > >> However, if i upload the File (700mb) it stops after 100%,
 > >> showing an error "There was an error uploading your file".
 > >>
 > >> Can someone help me fixing that error?
 > >>
 > >> Thank you, Louis
 > >>
 >
 > _______________________________________________
 > users mailing list
 > users(a)xwiki.org
 > http://lists.xwiki.org/mailman/listinfo/users 