Hi developers,
A while ago I was looking for some ways to track how much memory is used
by our internal cache and was not able to find anything.
I've tried it again and this time I found the following code:
http://www.javamex.com/classmexer/
This requires a simple instrumentation to work, but I was able to get
some results out of it to measure the size of our documents in cache.
You can see the result on a personal server:
Measuring one page:
http://www.zapnews.tv/xwiki/bin/view/Admin/MemoryUsage
Measuring all pages in cache:
http://www.zapnews.tv/xwiki/bin/view/Admin/MemoryUsage?page=all
The first results I can see, is that with no surprise the items taking
most memory are:
- attachment content
- attachment archive
- archive
What I was able to see is that as expected these fields won't consume
memory until we are asking for the data.
And after a while, the memory is indeed discarded for these fields, so
the usage of SoftReferences for them seem to work.
Now what I can see is that the attachment archive can be very very very
costly in memory.
Also it does not seem clear how the memory from these fields is garbage
collected (a GC did not recover it).
With some experience of massive loading of attachments that lead to
OutOfMemory errors in the server, I do suspect that the SoftReferences
are not necessarly discarded fast enough to avoid the OutOfMemory. I
also believe that a search engine that is walking all our pages
including our archive pages can genearate important memory usage that
could lead to problems. But this is only an intuition that needs to be
proved.
I believe we need to run some testing under stress to see if the cache
and memory usage do behave properly and if the cache(s) are never able
to go over the memory usage.
We also should try the classmexer on servers that are heavily used an be
able to look at the memory usage and see if we are "controling" it. I'm
not 100% sure how intrusive the "instrumentation" module is but I
believe it's quite light.
We could try it on xwiki.org or on myxwiki.org.
WDYT ?
Ludovic
--
Ludovic Dubost
Blog:http://blog.ludovic.org/XWiki:http://www.xwiki.com
Skype: ldubost GTalk: ldubost
On May 17, 2010, at 2:38 PM, tmortagne (SVN) wrote:
> Author: tmortagne
> Date: 2010-05-17 14:38:00 +0200 (Mon, 17 May 2010)
> New Revision: 28930
>
> Modified:
> platform/xwiki-tools/trunk/xwiki-configuration-resources/src/main/resources/xwiki.properties.vm
> Log:
> XTCONFRES-48: Add configuration for document rendering cache
>
> Modified: platform/xwiki-tools/trunk/xwiki-configuration-resources/src/main/resources/xwiki.properties.vm
> ===================================================================
> --- platform/xwiki-tools/trunk/xwiki-configuration-resources/src/main/resources/xwiki.properties.vm 2010-05-17 12:33:16 UTC (rev 28929)
> +++ platform/xwiki-tools/trunk/xwiki-configuration-resources/src/main/resources/xwiki.properties.vm 2010-05-17 12:38:00 UTC (rev 28930)
> @@ -12,6 +12,27 @@
> #-# Default value is xwiki/2.0.
> # core.defaultDocumentSyntax = xwiki/1.0
>
> +#-# [Since 2.4M1]
> +#-# Indicate if the rendering cache is enabled.
> +#-# Default value is false.
> +# core.renderingcache.enabled=true
> +
> +#-# [Since 2.4M1]
> +#-# A list of JAVA regex based patterns matching full documents reference.
Is this param about listing documents that are cached and for which the rendered result is cached? If so I think we should mention this in the documentation.
Also JAVA --> Java ? (uppercase would mean that it's an acronym, which it isn't).
"JAVA regex based" --> "Java regex" ?
> +# core.renderingcache.documents=wiki:Space\.Page
> +# core.renderingcache.documents=wiki:Space\..*
> +
> +#-# [Since 2.4M1]
> +#-# Indicate after which duration (in seconds) a data should be removed from the cache when not used.
"The time after which data should be removed from the cache when not used."
> +#-# Default value is 300 (5 min).
> +# core.renderingcache.duration=300
> +
> +#-# [Since 2.4M1]
> +#-# The size of the rendering cache. Not that its not the number of document cached but the number of results cached.
"its" --> "it's"
"document" --> "documents"
"document cached" --> "cached documents"
"results cached" --> "cached results"
> +#-# For a same document a variety of result is produced because each action, language and request query string produce unique result.
I would put this in parenthesis to mention that it's an explanation of "results" above.
Also:
"For a same document a variety of result is produced because each action, language and request query string produce unique result."
-->
"For a single document several cache entries are created, because each action, language and request query string produces a unique rendering result."
> +#-# Default value is 100.
> +# core.renderingcache.size=100
> +
> #-------------------------------------------------------------------------------------
> # Rendering
> #-------------------------------------------------------------------------------------
>
Thanks
-Vincent
On May 17, 2010, at 2:34 AM, sdumitriu (SVN) wrote:
> Author: sdumitriu
> Date: 2010-05-17 02:34:18 +0200 (Mon, 17 May 2010)
> New Revision: 28901
>
> Modified:
> enterprise/trunk/distribution-test/ui-tests/src/test/it/org/xwiki/it/ui/EscapeTest.java
> Log:
> [misc] Added a test.
>
> Modified: enterprise/trunk/distribution-test/ui-tests/src/test/it/org/xwiki/it/ui/EscapeTest.java
[snip]
> + /**
> + * Go to a working page after each test run to prevent failures in {@link #setUp()}
> + */
> + @After
> + public void tearDown()
> + {
> + TestUtils.gotoPage("Main", "WebHome", getDriver());
> + }
It's a little bit better to do this in setUp rather than in tearDown, for performance (Each test is responsible to set up a working state).
Thanks
-Vincent
I think I remember Ludovic was using that status action some time ago.
Ludovic, can you confirm it's ok to remove it?
Thanks
-Vincent
On May 17, 2010, at 3:29 AM, sdumitriu (SVN) wrote:
> Author: sdumitriu
> Date: 2010-05-17 03:29:40 +0200 (Mon, 17 May 2010)
> New Revision: 28906
>
> Removed:
> platform/core/trunk/xwiki-core/src/main/java/com/xpn/xwiki/web/StatusAction.java
> Log:
> XWIKI-5200: Remove non-working /status/ action
> Removed Java code.
>
> Deleted: platform/core/trunk/xwiki-core/src/main/java/com/xpn/xwiki/web/StatusAction.java
> ===================================================================
> --- platform/core/trunk/xwiki-core/src/main/java/com/xpn/xwiki/web/StatusAction.java 2010-05-17 01:29:34 UTC (rev 28905)
> +++ platform/core/trunk/xwiki-core/src/main/java/com/xpn/xwiki/web/StatusAction.java 2010-05-17 01:29:40 UTC (rev 28906)
> @@ -1,45 +0,0 @@
> -/*
> - * See the NOTICE file distributed with this work for additional
> - * information regarding copyright ownership.
> - *
> - * This is free software; you can redistribute it and/or modify it
> - * under the terms of the GNU Lesser General Public License as
> - * published by the Free Software Foundation; either version 2.1 of
> - * the License, or (at your option) any later version.
> - *
> - * This software is distributed in the hope that it will be useful,
> - * but WITHOUT ANY WARRANTY; without even the implied warranty of
> - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> - * Lesser General Public License for more details.
> - *
> - * You should have received a copy of the GNU Lesser General Public
> - * License along with this software; if not, write to the Free
> - * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
> - * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
> - *
> - */
> -package com.xpn.xwiki.web;
> -
> -import com.xpn.xwiki.XWiki;
> -import com.xpn.xwiki.XWikiContext;
> -import org.apache.struts.action.ActionForward;
> -
> -public class StatusAction extends XWikiAction
> -{
> - /**
> - * We override {@link XWikiAction#execute(XWikiContext) since we don't want any database calls
> - * at all to happen during the execution of this action since we're displaying statuses about
> - * the database.
> - */
> - public ActionForward execute(XWikiContext context) throws Exception
> - {
> - // We only get the main wiki since the database statuses are available from it and getting
> - // the other wiki will call the database.
> - XWiki.getMainXWiki(context);
> -
> - String page = Utils.getPage(context.getRequest(), "status");
> - Utils.parseTemplate(page, !page.equals("direct"), context);
> -
> - return null;
> - }
> -}
Hi devs,
There are several templates and actions that haven't been used in quite
a long time, most of which don't even work anymore. I'd like to remove
them, since they are a burden to maintain and they give the false
impression that some functionality exists and is supposed to work.
Actions to remove:
- /status/, which was supposed to give status information about the
server (memory + database connections), but it doesn't work at all now,
and it would need some major work to make it functional + safe. The same
(and more) functionality is provided by the Monitoring Application.
- /gettables/ /createchart/ /previewchart/ and /chartwizard/, which were
supposed to be used by a chart wizard developed in 2005, but
unmaintained since then, and which don't work anymore. Viewing manually
entered charts still works without these actions.
- /lifeblog/ (and the entire com.xpn.xwiki.atom package) which doesn't
seem to work, since it was written a very long time ago, and uses a lot
of old, not working stuff, like a XWiki.BlogPreferences class and
expects unencrypted passwords.
Templates to remove:
- status.vm, used by the /status/ action
- chwmain.vm, gettables.vm, getpagedatasources.vm and the chw/
directory, used by the non-working chart wizard
- wml.vm, since it offers very little functionality, and nobody really
uses WML anymore
- s5*.vm, used by the old Presentation Application, which was removed
from the default XE and has been unmaintained for a long time
All these probably haven't been used since before XWiki 1.0, and in case
we'll ever need them, they can be found in the SVN history.
--
Sergiu Dumitriu
http://purl.org/net/sergiu/
Hi Devs,
I'd like to add a bridge event to allow components to get notified when
old core has been initialized (at the end of XWiki#initXWiki for example).
This would be useful for components that needs to access the store, for
example to register XWiki classes.
I recognize this solution is not perfect since it's more bridged code, but
it would be better that what we have been doing so far (hard-coded
component initialization in XWiki.java, for example for the wiki macro
bridge to register its classes)
I've attached a patch proposal to
http://jira.xwiki.org/jira/browse/XWIKI-5194.
Note that I would also fix the wiki macro bridge initialization.
+1 from me to go ahead with that.
WDYT ?
Thanks,
Jerome
Hi devs,
I would like to commit my configuration based rendering cache system before
2.4M1 for it to be well tested and improved during 2.4 dev.
You can find details about it on
http://markmail.org/message/tnggdulplm73cxm5
Here is my +1
Thanks,
--
Thomas Mortagne