[xwiki-users] XWiki Does Not Handle Attachments Efficiently
Is there anything I can do to make XWiki 1.4 handle large attachments? That is, attachments from a few megabytes up to a few dozen megabytes. XWiki does not appear to efficiently handle attachments in general; large attachments appear to require extraordinary measures and resources. In testing prior to a deployment I attempted to attach a 56 MB file to an instance running on a server with 2GB of RAM and using MySQL as the database. To attach the file: 1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a larger value, since by default it's only 1MB and XWiki stores attachments directly in the database, rather than saving them in the file system and referencing the attachment's path. 2. Next I needed change the maximum attachment size in XWiki, which can be done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a NumberProperty field of type long, named upload_maxsize" which I finally figured out I could accomplish by visiting a page similar to "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object" ...although this wasn't immediately obvious. 3. At this point attempting to attach the file resulted in a JDBC error, which I resolved by manually changing some columns in the database from type mediumblob to longblob. 4. Attempting to attach the file still didn't work, this time due to Java heap space limitations. I increased the size of the heap, modifying the JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes to 512MB, up from 256). This had no effect on the error, so I increased the size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the file. Next I attempted to delete the attachment along with the page to which it is attached. My naïve assumption was that deleting an attachment should be easier than adding one and I didn't anticipate problems. Unfortunately, this again resulted in out of memory errors. To work around the problem I tried deleting the attachment by clicking the delete button in the show --> attachments screen. This still resulted in an out of memory error, so I upped the heap size to 1.6 GB, or 80% of the RAM in the server. This resulted in very heavy swapping, yet still failed due to an out of memory exception. (On the positive side, at least the swapping helped keep the processor utilization under 100%.) I've ordered another 2GB of RAM and will try increasing the heap size again after it's installed. Until then, is there anything I can do to keep attachments from eating the server alive? If I recall correctly, there has been talk about saving attachments externally, rather than within the database. Has anyone built a mechanism to allow this, or is there any news to report on that front? Also, does anyone have any advice on how I can excise this page and its 56MB attachment from the system prior to more memory arriving? (I'd like to do so as, aside from the attachment problems and dysfunctional WYSIWYG editor, the system looks deployable.) As it stands, unless I'm doing something incorrectly, I have to conclude that it's basically impractical to use attachments larger than a few megabytes with XWiki, and even small attachments are frightfully ungainly. - AW
Hi AW, I think there is a general agreement about the fact that an option to store attachments, particularly big ones, outside the database. I think it was of the main reasons to accept/promote WebDAV integration as one of the projects to be developed during the current Google Summer of Code (GSoC) mentored by Ludovic Dubost. http://code.google.com/soc/2008/xwiki/about.html Perhaps these threads will help you to catch up with this issues and make some contribution!!! http://www.nabble.com/About-WebDav-Integration-tp16358565p16358565.html http://www.nabble.com/Need-help-with-WebDAV-Support.-tp16453419p16453419.htm... http://www.nabble.com/-GSOC--WebDAV--Status-...-tp17588475p17588475.html http://www.nabble.com/-GSOC---WebDAV--Ground-work-done.-tp17674844p17674844.... Hope this helps, Ricardo AW wrote:
Is there anything I can do to make XWiki 1.4 handle large attachments? That is, attachments from a few megabytes up to a few dozen megabytes.
XWiki does not appear to efficiently handle attachments in general; large attachments appear to require extraordinary measures and resources.
In testing prior to a deployment I attempted to attach a 56 MB file to an instance running on a server with 2GB of RAM and using MySQL as the database.
To attach the file:
1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a larger value, since by default it's only 1MB and XWiki stores attachments directly in the database, rather than saving them in the file system and referencing the attachment's path.
2. Next I needed change the maximum attachment size in XWiki, which can be done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a NumberProperty field of type long, named upload_maxsize" which I finally figured out I could accomplish by visiting a page similar to "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object" ...although this wasn't immediately obvious.
3. At this point attempting to attach the file resulted in a JDBC error, which I resolved by manually changing some columns in the database from type mediumblob to longblob.
4. Attempting to attach the file still didn't work, this time due to Java heap space limitations. I increased the size of the heap, modifying the JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes to 512MB, up from 256). This had no effect on the error, so I increased the size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the file.
Next I attempted to delete the attachment along with the page to which it is attached. My naïve assumption was that deleting an attachment should be easier than adding one and I didn't anticipate problems. Unfortunately, this again resulted in out of memory errors. To work around the problem I tried deleting the attachment by clicking the delete button in the show --> attachments screen. This still resulted in an out of memory error, so I upped the heap size to 1.6 GB, or 80% of the RAM in the server. This resulted in very heavy swapping, yet still failed due to an out of memory exception. (On the positive side, at least the swapping helped keep the processor utilization under 100%.)
I've ordered another 2GB of RAM and will try increasing the heap size again after it's installed. Until then, is there anything I can do to keep attachments from eating the server alive? If I recall correctly, there has been talk about saving attachments externally, rather than within the database. Has anyone built a mechanism to allow this, or is there any news to report on that front?
Also, does anyone have any advice on how I can excise this page and its 56MB attachment from the system prior to more memory arriving? (I'd like to do so as, aside from the attachment problems and dysfunctional WYSIWYG editor, the system looks deployable.)
As it stands, unless I'm doing something incorrectly, I have to conclude that it's basically impractical to use attachments larger than a few megabytes with XWiki, and even small attachments are frightfully ungainly.
- AW
_______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users
-- Ricardo Rodríguez Your EPEC Network ICT Team
Hi AW, you may want to look at this thread that addresses most of the issues you encountered AFAIR : http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query:atta... Guillaume On Mon, Jun 9, 2008 at 11:24 PM, [Ricardo Rodriguez] Your EPEC Network ICT Team <[email protected]> wrote:
Hi AW,
I think there is a general agreement about the fact that an option to store attachments, particularly big ones, outside the database. I think it was of the main reasons to accept/promote WebDAV integration as one of the projects to be developed during the current Google Summer of Code (GSoC) mentored by Ludovic Dubost.
http://code.google.com/soc/2008/xwiki/about.html
Perhaps these threads will help you to catch up with this issues and make some contribution!!!
http://www.nabble.com/About-WebDav-Integration-tp16358565p16358565.html
http://www.nabble.com/Need-help-with-WebDAV-Support.-tp16453419p16453419.htm... http://www.nabble.com/-GSOC--WebDAV--Status-...-tp17588475p17588475.html
http://www.nabble.com/-GSOC---WebDAV--Ground-work-done.-tp17674844p17674844....
Hope this helps,
Ricardo
AW wrote:
Is there anything I can do to make XWiki 1.4 handle large attachments? That is, attachments from a few megabytes up to a few dozen megabytes.
XWiki does not appear to efficiently handle attachments in general; large attachments appear to require extraordinary measures and resources.
In testing prior to a deployment I attempted to attach a 56 MB file to an instance running on a server with 2GB of RAM and using MySQL as the database.
To attach the file:
1. I first had to set the MySQL max_allowed_packet value in /etc/my.cnf to a larger value, since by default it's only 1MB and XWiki stores attachments directly in the database, rather than saving them in the file system and referencing the attachment's path.
2. Next I needed change the maximum attachment size in XWiki, which can be done by "edit[ing] the XWiki.XWikiPreferences class and add[ing] a NumberProperty field of type long, named upload_maxsize" which I finally figured out I could accomplish by visiting a page similar to "http://my.server/xwiki/bin/edit/XWiki/XWikiPreferences?editor=object" ...although this wasn't immediately obvious.
3. At this point attempting to attach the file resulted in a JDBC error, which I resolved by manually changing some columns in the database from type mediumblob to longblob.
4. Attempting to attach the file still didn't work, this time due to Java heap space limitations. I increased the size of the heap, modifying the JAVA_OPTS with -Xmx512m -Xms512m (setting the initial and maximum heap sizes to 512MB, up from 256). This had no effect on the error, so I increased the size to 1 GB, but the error persisted. At 1.2 GB I was able to attach the file.
Next I attempted to delete the attachment along with the page to which it is attached. My naïve assumption was that deleting an attachment should be easier than adding one and I didn't anticipate problems. Unfortunately, this again resulted in out of memory errors. To work around the problem I tried deleting the attachment by clicking the delete button in the show --> attachments screen. This still resulted in an out of memory error, so I upped the heap size to 1.6 GB, or 80% of the RAM in the server. This resulted in very heavy swapping, yet still failed due to an out of memory exception. (On the positive side, at least the swapping helped keep the processor utilization under 100%.)
I've ordered another 2GB of RAM and will try increasing the heap size again after it's installed. Until then, is there anything I can do to keep attachments from eating the server alive? If I recall correctly, there has been talk about saving attachments externally, rather than within the database. Has anyone built a mechanism to allow this, or is there any news to report on that front?
Also, does anyone have any advice on how I can excise this page and its 56MB attachment from the system prior to more memory arriving? (I'd like to do so as, aside from the attachment problems and dysfunctional WYSIWYG editor, the system looks deployable.)
As it stands, unless I'm doing something incorrectly, I have to conclude that it's basically impractical to use attachments larger than a few megabytes with XWiki, and even small attachments are frightfully ungainly.
- AW
_______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users
-- Ricardo Rodríguez Your EPEC Network ICT Team
_______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users
Hi. You can set xwiki.store.attachment.versioning=0 in xwiki.cfg This will turn off attachment versioning and *very* much reduce memory consumption. see http://jira.xwiki.org/jira/browse/XWIKI-2275 and http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Configuration#HOptionalS... AW wrote:
Is there anything I can do to make XWiki 1.4 handle large attachments? That is, attachments from a few megabytes up to a few dozen megabytes.
...
-- Artem Melentyev
Hi Artem, On Jun 12, 2008, at 2:21 PM, Artem Melentyev wrote:
Hi.
You can set xwiki.store.attachment.versioning=0 in xwiki.cfg This will turn off attachment versioning and *very* much reduce memory consumption. see http://jira.xwiki.org/jira/browse/XWIKI-2275 and http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Configuration#HOptionalS...
Why is it using some much memory? Isn't that a bug? (I don't understand why we would need to load more than 1 attachment version at a time in memory). Thanks -Vincent
AW wrote:
Is there anything I can do to make XWiki 1.4 handle large attachments? That is, attachments from a few megabytes up to a few dozen megabytes.
Hi, Vincent. Vincent Massol wrote:
Why is it using some much memory? Isn't that a bug? (I don't understand why we would need to load more than 1 attachment version at a time in memory).
Look at http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query:atta...
Worse, if this is not the first version of the attachment, then the complete attachment history is loaded in memory, so add another 24*versionsize*versions of memory needed during upload.
This is a feature of JRCS :) -- Artem Melentyev
On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote:
Hi, Vincent.
Vincent Massol wrote:
Why is it using some much memory? Isn't that a bug? (I don't understand why we would need to load more than 1 attachment version at a time in memory).
Look at http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results
Worse, if this is not the first version of the attachment, then the complete attachment history is loaded in memory, so add another 24*versionsize*versions of memory needed during upload.
This is a feature of JRCS :)
I see. So I guess the only solution would be drop JRCS and do our own stuff... (or find another library that does it fast). Thanks -Vincent
Hi There was another discussion on this topic, where an effort has been made to store the files directly on the filesystem. Not sure how this would reduce memory consumption. Please read Sergiu's comments carefully though, before you try this out. Link http://www.nabble.com/AttachmentStore---my-filesystem-based-implementation-t... here Of course, in the meantime, we await the WebDAV implementation. Shiva vmassol wrote:
On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote:
Hi, Vincent.
Vincent Massol wrote:
Why is it using some much memory? Isn't that a bug? (I don't understand why we would need to load more than 1 attachment version at a time in memory).
Look at http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results
Worse, if this is not the first version of the attachment, then the complete attachment history is loaded in memory, so add another 24*versionsize*versions of memory needed during upload.
This is a feature of JRCS :)
I see. So I guess the only solution would be drop JRCS and do our own stuff... (or find another library that does it fast).
Thanks -Vincent
_______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users
-- View this message in context: http://www.nabble.com/XWiki-Does-Not-Handle-Attachments-Efficiently-tp177415... Sent from the XWiki- Users mailing list archive at Nabble.com.
shivshan wrote:
Hi There was another discussion on this topic, where an effort has been made to store the files directly on the filesystem. Not sure how this would reduce memory consumption. Please read Sergiu's comments carefully though, before you try this out. Link http://www.nabble.com/AttachmentStore---my-filesystem-based-implementation-t... here Of course, in the meantime, we await the WebDAV implementation. Shiva
I'm not sure the webdav implementation will solve the memory problem. If I understand correctly how it will work, then it will just be an interface over the same core, meaning that all the performance issues will still be there.
vmassol wrote:
On Jun 12, 2008, at 2:45 PM, Artem Melentyev wrote:
Hi, Vincent.
Vincent Massol wrote:
Why is it using some much memory? Isn't that a bug? (I don't understand why we would need to load more than 1 attachment version at a time in memory). Look at http://xwiki.markmail.org/search/?q=attachments+storage+profiling#query :attachments%20storage%20profiling+page:1+mid:5avjhiu2nkg4xtzt+state:results Worse, if this is not the first version of the attachment, then the complete attachment history is loaded in memory, so add another 24*versionsize*versions of memory needed during upload. This is a feature of JRCS :) I see. So I guess the only solution would be drop JRCS and do our own stuff... (or find another library that does it fast).
Thanks -Vincent
-- Sergiu Dumitriu http://purl.org/net/sergiu/
participants (7)
-
[Ricardo Rodriguez] Your EPEC Network ICT Team -
Artem Melentyev -
AW -
Guillaume Lerouge -
Sergiu Dumitriu -
shivshan -
Vincent Massol