[xwiki-devs] Reduce heap exhaustion during export operation (and partly import for now)
Hi, As you knows, Import/Export operation on large XAR file cause problem. We felt on an even worse situation, where a single XWiki document is not properly exported due to heap exhaustion during the built of its XML DOM or due to large attachments. Having a look at the source, I have notice that many optimizations on the way the export is produce could be quite easily introduced. Currently, XWiki stored the exported document several times in memory during the operation, which impact performance negativly and is uselessly heavy on memory usage, even for reasonable documents. Therefore, I have started a large patch to avoid these caveats, here is the strategies I have followed: 1) The current implementation mostly build a DOM in memory for immediately serialized it into a stream. So I have remove the intermediate DOM and provide direct streaming of Element content by: 1.1) extending org.dom4j.XMLWriter to allow direct streaming of Element content into the output stream, as is, or Base64 encoded. Accessorily, my extension also ensure proper pairing of open/close tag. 1.2) writing a minimal DOMXMLWriter which extends my XMLWriter and could be used with the same toXML() code to build a DOMDocument to provide the toXMLDocument() methods to support the older implementation unchanged if ever needed. 1.3) using the above, minimal change to the current XML code was required 1.3.1) replacing element.add(Element) by either writer.writeElement) or writer.writeOpen(Element) 1.3.2) for large content, use my extensions, either writer.write(Element, InputStream) or writer.writeBase64(Element, InputStream) which use the InputStream for the element content 2) The current implementation for binary data such as attachments and export zip file is mostly based on in memory byte[] passing from function to function while these data initially came from a request.getInputStream() or are written to a response.getOutputStream (). So I have change these to passover the stream instead of the data: 2.1) using IOUtils.copy when required 2.2) using org.apache.commons.codec.binary.Base64OutputStream for base64 encoding when required 2.3) using an extension of ZipInputStream to cope with unexpected close() 2.4) avoid buffer duplication in favor of stream filters 3) Since most oftently used large data came from the database through an attachment content, it would be nice to have these attachment streamed from the database when they are too large. However, I feel that it is still too early to convert our binary into a blob, mainly because HSQLDB and MySQL still does not really support blob, just an emulation. These are also used to be cached in the document cache, and this will require improvement to support blob. However I propose to take the occasion to go in the direction of the blob by: 3.1) deprecating setContent(byte[]) and Byte[] getContent() in favor of newly created setContent(InputStream, int), InputStream getContentInputStream() and getSize() 3.2) Begin to use these new function as much as possible as 2) implied 3.3) this also open the ability to store attachment in another repository that support better the streaming aspect (ie: a filesystem) I am currently testing the above changes onto our client project, and I expect to provide a patch really soon. It will require an upgrade of org.apache.commons.codec to version 1.4, to have access to Base64OutputStream. I feel it will be a first step into the right direction, further improvement should be to: - import XML using a SAXParser without building a DOM in memory - manage JRCS Archive better, the way they are built and store raise the same issue then attachment - manage the recycle bin better for the same reason - improve caching to avoid caching very large stuffs WDYT ? Denis Gervalle -- SOFTEC sa http://www.softec.st
On Thu, Sep 17, 2009 at 14:40, Denis Gervalle <[email protected]> wrote:
Hi,
As you knows, Import/Export operation on large XAR file cause problem. We felt on an even worse situation, where a single XWiki document is not properly exported due to heap exhaustion during the built of its XML DOM or due to large attachments.
Having a look at the source, I have notice that many optimizations on the way the export is produce could be quite easily introduced. Currently, XWiki stored the exported document several times in memory during the operation, which impact performance negativly and is uselessly heavy on memory usage, even for reasonable documents.
Therefore, I have started a large patch to avoid these caveats, here is the strategies I have followed:
1) The current implementation mostly build a DOM in memory for immediately serialized it into a stream. So I have remove the intermediate DOM and provide direct streaming of Element content by: 1.1) extending org.dom4j.XMLWriter to allow direct streaming of Element content into the output stream, as is, or Base64 encoded. Accessorily, my extension also ensure proper pairing of open/close tag. 1.2) writing a minimal DOMXMLWriter which extends my XMLWriter and could be used with the same toXML() code to build a DOMDocument to provide the toXMLDocument() methods to support the older implementation unchanged if ever needed. 1.3) using the above, minimal change to the current XML code was required 1.3.1) replacing element.add(Element) by either writer.writeElement) or writer.writeOpen(Element) 1.3.2) for large content, use my extensions, either writer.write(Element, InputStream) or writer.writeBase64(Element, InputStream) which use the InputStream for the element content
2) The current implementation for binary data such as attachments and export zip file is mostly based on in memory byte[] passing from function to function while these data initially came from a request.getInputStream() or are written to a response.getOutputStream (). So I have change these to passover the stream instead of the data: 2.1) using IOUtils.copy when required 2.2) using org.apache.commons.codec.binary.Base64OutputStream for base64 encoding when required 2.3) using an extension of ZipInputStream to cope with unexpected close() 2.4) avoid buffer duplication in favor of stream filters
3) Since most oftently used large data came from the database through an attachment content, it would be nice to have these attachment streamed from the database when they are too large. However, I feel that it is still too early to convert our binary into a blob, mainly because HSQLDB and MySQL still does not really support blob, just an emulation. These are also used to be cached in the document cache, and this will require improvement to support blob. However I propose to take the occasion to go in the direction of the blob by: 3.1) deprecating setContent(byte[]) and Byte[] getContent() in favor of newly created setContent(InputStream, int), InputStream getContentInputStream() and getSize() 3.2) Begin to use these new function as much as possible as 2) implied 3.3) this also open the ability to store attachment in another repository that support better the streaming aspect (ie: a filesystem)
I am currently testing the above changes onto our client project, and I expect to provide a patch really soon. It will require an upgrade of org.apache.commons.codec to version 1.4, to have access to Base64OutputStream.
Sounds great I'm eager to see your first patch ;)
I feel it will be a first step into the right direction, further improvement should be to: - import XML using a SAXParser without building a DOM in memory - manage JRCS Archive better, the way they are built and store raise the same issue then attachment - manage the recycle bin better for the same reason - improve caching to avoid caching very large stuffs
WDYT ?
Denis Gervalle -- SOFTEC sa http://www.softec.st
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
Hi, 1) The current implementation mostly build a DOM in memory for
immediately serialized it into a stream. So I have remove the intermediate DOM and provide direct streaming of Element content by: 1.1) extending org.dom4j.XMLWriter to allow direct streaming of Element content into the output stream, as is, or Base64 encoded. Accessorily, my extension also ensure proper pairing of open/close tag. 1.2) writing a minimal DOMXMLWriter which extends my XMLWriter and could be used with the same toXML() code to build a DOMDocument to provide the toXMLDocument() methods to support the older implementation unchanged if ever needed. 1.3) using the above, minimal change to the current XML code was required 1.3.1) replacing element.add(Element) by either writer.writeElement) or writer.writeOpen(Element) 1.3.2) for large content, use my extensions, either writer.write(Element, InputStream) or writer.writeBase64(Element, InputStream) which use the InputStream for the element content
I'm not aware of the context so I might be wrong, but can't we use StAX to achieve this xml streaming? ( http://www.xml.com/pub/a/2003/09/17/stax.html?page=2)
2) The current implementation for binary data such as attachments and export zip file is mostly based on in memory byte[] passing from function to function while these data initially came from a request.getInputStream() or are written to a response.getOutputStream (). So I have change these to passover the stream instead of the data: 2.1) using IOUtils.copy when required 2.2) using org.apache.commons.codec.binary.Base64OutputStream for base64 encoding when required 2.3) using an extension of ZipInputStream to cope with unexpected close() 2.4) avoid buffer duplication in favor of stream filters
Sounds good. May be FileUploadPlugin should also pass streams instead of data so we do not run out of memory.( http://maven.xwiki.org/site/xwiki-core-parent/xwiki-core/apidocs/com/xpn/xwi... )
3) Since most oftently used large data came from the database through an attachment content, it would be nice to have these attachment streamed from the database when they are too large. However, I feel that it is still too early to convert our binary into a blob, mainly because HSQLDB and MySQL still does not really support blob, just an emulation. These are also used to be cached in the document cache, and this will require improvement to support blob. However I propose to take the occasion to go in the direction of the blob by: 3.1) deprecating setContent(byte[]) and Byte[] getContent() in favor of newly created setContent(InputStream, int), InputStream getContentInputStream() and getSize() 3.2) Begin to use these new function as much as possible as 2) implied 3.3) this also open the ability to store attachment in another repository that support better the streaming aspect (ie: a filesystem)
+1 Still, I wonder if streaming blobs from databases is a mature technology. (Mysql has an extension : http://blobstreaming.org/) Keep up the good work :) Thanks. - Asiri
On Sep 17, 2009, at 19:10, Asiri Rathnayake wrote:
Hi,
1) The current implementation mostly build a DOM in memory for
immediately serialized it into a stream. So I have remove the intermediate DOM and provide direct streaming of Element content by: 1.1) extending org.dom4j.XMLWriter to allow direct streaming of Element content into the output stream, as is, or Base64 encoded. Accessorily, my extension also ensure proper pairing of open/close tag. 1.2) writing a minimal DOMXMLWriter which extends my XMLWriter and could be used with the same toXML() code to build a DOMDocument to provide the toXMLDocument() methods to support the older implementation unchanged if ever needed. 1.3) using the above, minimal change to the current XML code was required 1.3.1) replacing element.add(Element) by either writer.writeElement) or writer.writeOpen(Element) 1.3.2) for large content, use my extensions, either writer.write(Element, InputStream) or writer.writeBase64(Element, InputStream) which use the InputStream for the element content
I'm not aware of the context so I might be wrong, but can't we use StAX to achieve this xml streaming? ( http://www.xml.com/pub/a/2003/09/17/stax.html?page=2)
In fact, I have not changed the library currently in use (DOM4J), especially because I want 1.3). Minimizing code change in the XML processing part avoid the introduction of new bugs there. Since the code has to be almost completely rewritten soon or later, I had prefer this approach for now.
2) The current implementation for binary data such as attachments and export zip file is mostly based on in memory byte[] passing from function to function while these data initially came from a request.getInputStream() or are written to a response.getOutputStream (). So I have change these to passover the stream instead of the data: 2.1) using IOUtils.copy when required 2.2) using org.apache.commons.codec.binary.Base64OutputStream for base64 encoding when required 2.3) using an extension of ZipInputStream to cope with unexpected close() 2.4) avoid buffer duplication in favor of stream filters
Sounds good. May be FileUploadPlugin should also pass streams instead of data so we do not run out of memory.( http://maven.xwiki.org/site/xwiki-core-parent/xwiki-core/apidocs/com/xpn/xwi... )
This is in my patch already ;) But, due to 3), this is not enough to prevent out of memory completely, this put the limit higher for export but not much for upload. Of course, this is no more an issue of the plugin now. Storage and caching are were improvement should go later.
3) Since most oftently used large data came from the database through an attachment content, it would be nice to have these attachment streamed from the database when they are too large. However, I feel that it is still too early to convert our binary into a blob, mainly because HSQLDB and MySQL still does not really support blob, just an emulation. These are also used to be cached in the document cache, and this will require improvement to support blob. However I propose to take the occasion to go in the direction of the blob by: 3.1) deprecating setContent(byte[]) and Byte[] getContent() in favor of newly created setContent(InputStream, int), InputStream getContentInputStream() and getSize() 3.2) Begin to use these new function as much as possible as 2) implied 3.3) this also open the ability to store attachment in another repository that support better the streaming aspect (ie: a filesystem)
+1
Still, I wonder if streaming blobs from databases is a mature technology. (Mysql has an extension : http://blobstreaming.org/)
Well, I was not thinking about direct streaming, but using classical blob with a Locator, which is not much supported by open source databases. As far as I know, Oracle began to do it quite well beginning with 9.x version, but there is still some caveat. So I prefer to first provide an interface that allow the switch and put the switch in another patch when appropriate. This one is already large enough and it move an issue that is everywhere in the core, to a single place. (Almost, since Archive is not yet fixed).
Keep up the good work :)
Thanks ;) Denis
Hi Denis, Your improvements sound really great! I saw mentioned a patch several times but didn't manage to find it... Any hints where I can find it? Thanks! Anamaria On Thu, Sep 17, 2009 at 7:49 PM, Denis Gervalle <[email protected]> wrote:
On Sep 17, 2009, at 19:10, Asiri Rathnayake wrote:
Hi,
1) The current implementation mostly build a DOM in memory for
immediately serialized it into a stream. So I have remove the intermediate DOM and provide direct streaming of Element content by: 1.1) extending org.dom4j.XMLWriter to allow direct streaming of Element content into the output stream, as is, or Base64 encoded. Accessorily, my extension also ensure proper pairing of open/close tag. 1.2) writing a minimal DOMXMLWriter which extends my XMLWriter and could be used with the same toXML() code to build a DOMDocument to provide the toXMLDocument() methods to support the older implementation unchanged if ever needed. 1.3) using the above, minimal change to the current XML code was required 1.3.1) replacing element.add(Element) by either writer.writeElement) or writer.writeOpen(Element) 1.3.2) for large content, use my extensions, either writer.write(Element, InputStream) or writer.writeBase64(Element, InputStream) which use the InputStream for the element content
I'm not aware of the context so I might be wrong, but can't we use StAX to achieve this xml streaming? ( http://www.xml.com/pub/a/2003/09/17/stax.html?page=2)
In fact, I have not changed the library currently in use (DOM4J), especially because I want 1.3). Minimizing code change in the XML processing part avoid the introduction of new bugs there. Since the code has to be almost completely rewritten soon or later, I had prefer this approach for now.
2) The current implementation for binary data such as attachments and export zip file is mostly based on in memory byte[] passing from function to function while these data initially came from a request.getInputStream() or are written to a response.getOutputStream (). So I have change these to passover the stream instead of the data: 2.1) using IOUtils.copy when required 2.2) using org.apache.commons.codec.binary.Base64OutputStream for base64 encoding when required 2.3) using an extension of ZipInputStream to cope with unexpected close() 2.4) avoid buffer duplication in favor of stream filters
Sounds good. May be FileUploadPlugin should also pass streams instead of data so we do not run out of memory.(
http://maven.xwiki.org/site/xwiki-core-parent/xwiki-core/apidocs/com/xpn/xwi...
)
This is in my patch already ;) But, due to 3), this is not enough to prevent out of memory completely, this put the limit higher for export but not much for upload. Of course, this is no more an issue of the plugin now. Storage and caching are were improvement should go later.
3) Since most oftently used large data came from the database through an attachment content, it would be nice to have these attachment streamed from the database when they are too large. However, I feel that it is still too early to convert our binary into a blob, mainly because HSQLDB and MySQL still does not really support blob, just an emulation. These are also used to be cached in the document cache, and this will require improvement to support blob. However I propose to take the occasion to go in the direction of the blob by: 3.1) deprecating setContent(byte[]) and Byte[] getContent() in favor of newly created setContent(InputStream, int), InputStream getContentInputStream() and getSize() 3.2) Begin to use these new function as much as possible as 2) implied 3.3) this also open the ability to store attachment in another repository that support better the streaming aspect (ie: a filesystem)
+1
Still, I wonder if streaming blobs from databases is a mature technology. (Mysql has an extension : http://blobstreaming.org/)
Well, I was not thinking about direct streaming, but using classical blob with a Locator, which is not much supported by open source databases. As far as I know, Oracle began to do it quite well beginning with 9.x version, but there is still some caveat. So I prefer to first provide an interface that allow the switch and put the switch in another patch when appropriate. This one is already large enough and it move an issue that is everywhere in the core, to a single place. (Almost, since Archive is not yet fixed).
Keep up the good work :)
Thanks ;)
Denis
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Anamaria, Sorry, I was so busy these days that I have overlooked both jabber and the dev ML. I was finishing the migration of our last old site from 1.4 to 1.9, very boring :) This was the prerequisite for me to migrate all our client to 2.0, which I will start to deploy and test next week. This is why my current patch is against 1.9 currently, and I hope to have a patch against 2.0 probably next week. I will create a JIRA issue for the 1.9 patch so you may access it: XWIKI-4627 Please let me know if you see anything I could have missed. I suggest to use the LargeExportXar snippet for the best export possible, hacking it a little bit to not export history (I have a improve version I will consider to share as well, just a matter of time), so you should be able to export anything. With kind regards, Denis On Nov 25, 2009, at 22:49, Anamaria Stoica wrote:
Hi Denis,
Your improvements sound really great!
I saw mentioned a patch several times but didn't manage to find it... Any hints where I can find it?
Thanks! Anamaria
On Thu, Sep 17, 2009 at 7:49 PM, Denis Gervalle <[email protected]> wrote:
On Sep 17, 2009, at 19:10, Asiri Rathnayake wrote:
Hi,
1) The current implementation mostly build a DOM in memory for
immediately serialized it into a stream. So I have remove the intermediate DOM and provide direct streaming of Element content by: 1.1) extending org.dom4j.XMLWriter to allow direct streaming of Element content into the output stream, as is, or Base64 encoded. Accessorily, my extension also ensure proper pairing of open/close tag. 1.2) writing a minimal DOMXMLWriter which extends my XMLWriter and could be used with the same toXML() code to build a DOMDocument to provide the toXMLDocument() methods to support the older implementation unchanged if ever needed. 1.3) using the above, minimal change to the current XML code was required 1.3.1) replacing element.add(Element) by either writer.writeElement) or writer.writeOpen(Element) 1.3.2) for large content, use my extensions, either writer.write(Element, InputStream) or writer.writeBase64(Element, InputStream) which use the InputStream for the element content
I'm not aware of the context so I might be wrong, but can't we use StAX to achieve this xml streaming? ( http://www.xml.com/pub/a/2003/09/17/stax.html?page=2)
In fact, I have not changed the library currently in use (DOM4J), especially because I want 1.3). Minimizing code change in the XML processing part avoid the introduction of new bugs there. Since the code has to be almost completely rewritten soon or later, I had prefer this approach for now.
2) The current implementation for binary data such as attachments and export zip file is mostly based on in memory byte[] passing from function to function while these data initially came from a request.getInputStream() or are written to a response.getOutputStream (). So I have change these to passover the stream instead of the data: 2.1) using IOUtils.copy when required 2.2) using org.apache.commons.codec.binary.Base64OutputStream for base64 encoding when required 2.3) using an extension of ZipInputStream to cope with unexpected close() 2.4) avoid buffer duplication in favor of stream filters
Sounds good. May be FileUploadPlugin should also pass streams instead of data so we do not run out of memory.(
http://maven.xwiki.org/site/xwiki-core-parent/xwiki-core/apidocs/com/xpn/xwi...
)
This is in my patch already ;) But, due to 3), this is not enough to prevent out of memory completely, this put the limit higher for export but not much for upload. Of course, this is no more an issue of the plugin now. Storage and caching are were improvement should go later.
3) Since most oftently used large data came from the database through an attachment content, it would be nice to have these attachment streamed from the database when they are too large. However, I feel that it is still too early to convert our binary into a blob, mainly because HSQLDB and MySQL still does not really support blob, just an emulation. These are also used to be cached in the document cache, and this will require improvement to support blob. However I propose to take the occasion to go in the direction of the blob by: 3.1) deprecating setContent(byte[]) and Byte[] getContent() in favor of newly created setContent(InputStream, int), InputStream getContentInputStream() and getSize() 3.2) Begin to use these new function as much as possible as 2) implied 3.3) this also open the ability to store attachment in another repository that support better the streaming aspect (ie: a filesystem)
+1
Still, I wonder if streaming blobs from databases is a mature technology. (Mysql has an extension : http://blobstreaming.org/)
Well, I was not thinking about direct streaming, but using classical blob with a Locator, which is not much supported by open source databases. As far as I know, Oracle began to do it quite well beginning with 9.x version, but there is still some caveat. So I prefer to first provide an interface that allow the switch and put the switch in another patch when appropriate. This one is already large enough and it move an issue that is everywhere in the core, to a single place. (Almost, since Archive is not yet fixed).
Keep up the good work :)
Thanks ;)
Denis
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
participants (4)
-
Anamaria Stoica -
Asiri Rathnayake -
Denis Gervalle -
Thomas Mortagne