[xwiki-devs] XWiki Security Cache global read/write lock scalability

We ran into an issue recently where users were not able to access the Wiki due to the XWiki security cache holding up access to read requests as we were experiencing heap exhaustion. A thread dump of a host showed that there were 211 security cache read locks and 2 security cache write locks during the event. Read locks are supposedly parallelizable, so we suspect that it was the write locks that blocked the read requests. Looking into the write locks, we found *DefaultSecurityCacheRulesInvalidatorListener* shares a read-write lock (*SecurityCacheRulesInvalidator*) with the security cache loader (*DefaultSecurityCacheLoader*). It appears that these write requests to the security cache were potentially creating a deadlock, blocking any further read requests. Our investigation of this event resulted in the following questions that we are hoping to get answered by reaching out here: 1. What exactly happens in the event of a write failure to the Security Cache? Are we correct in understanding that the write request will continue to hold a lock on the cache, preventing all read requests from proceeding? 2. If so, are there any mitigations around the global read-write lock on the security cache? Or, perhaps, is the best action is to simply move toward a more scalable caching mechanism (i.e. a centralized cache such as Elastic Cache)? 3. It looks as if all threads doing a read/write request to the SecurityCache were all parked waiting for the same thread (3081f946); however, the thread is not showing up in the thread dump so we are not sure what it is stuck on. Does the XWiki team have any suspicions as to what could be hold up the requests to the SecurityCache? Thank you! --- For more information on the event: 1. Remote events triggered a security cache update and requested a couple of write locks 2. The writes could not execute (perhaps due to the heap exhaustion, although root cause of write failure is unknown) 3. ReentrantReadWriteLock lock followed a fair policy, which releases locks in time order, so read requests were forced to wait for any previous write requests to finish. Because the write requests couldn't finish, the remaining 211 threads were blocked. * Possible remote events that trigger security cache update: changes to an XWiki group such as leaving the admin group, user first time visiting XWiki, etc.. ---- *LDAP Connection Lock example:* Thread t@225108 java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) - waiting on <7249fc39> (a com.novell.ldap.Connection) at com.novell.ldap.Connection.waitForReader(Unknown Source) at com.novell.ldap.Connection.startReader(Unknown Source) at com.novell.ldap.Connection.connect(Unknown Source) at com.novell.ldap.Connection.connect(Unknown Source) at com.novell.ldap.LDAPConnection.connect(Unknown Source) at com.ownership.groups.ldap.DefaultLdapConnection.connect(DefaultLdapConnection.java:25) at com.ownership.groups.ldap.LdapClient.queryGroups(LdapClient.java:99) at com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:64) at com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:59) at com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:113) - locked <7cc1cfca> (a java.lang.String) at com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:75) at org.xwiki.security.internal.DefaultUserBridge.getGroupsReferencesFor(DefaultUserBridge.java:122) at org.xwiki.security.internal.DefaultUserBridge.getAllGroupsFor(DefaultUserBridge.java:78) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:350) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:342) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserEntry(DefaultSecurityCacheLoader.java:318) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadAccessEntries(DefaultSecurityCacheLoader.java:221) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadRequiredEntries(DefaultSecurityCacheLoader.java:193) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:148) at org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218) at org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159) at org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152) at org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109) at org.xwiki.display.internal.AbstractDocumentTitleDisplayer.displayTitle(AbstractDocumentTitleDisplayer.java:163) at org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:149) at org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:58) at org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:96) at org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:39) at org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:123) at org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:52) at org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:68) at org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:42) at com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1538) at com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1558) at com.xpn.xwiki.api.Document.getDisplayTitle(Document.java:345) at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.runtime.parser.node.PropertyExecutor.execute(PropertyExecutor.java:142) at org.apache.velocity.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:539) at org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:198) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567) at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.apache.velocity.runtime.parser.node.ASTStringLiteral.value(ASTStringLiteral.java:330) at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71) at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180) at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463) at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449) at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215) at com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176) at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) - locked <1cc68e56> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - locked <79120011> (a java.util.concurrent.ThreadPoolExecutor$Worker) ---- *Security Cache Read Lock example:* Thread t@225106 java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Native Method) - parking to wait for <3081f946> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidator.suspend(DefaultSecurityCacheRulesInvalidator.java:56) at org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:144) at org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218) at org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159) at org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152) at org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109) at org.xwiki.security.authorization.internal.XWikiCachingRightService.hasAccessLevel(XWikiCachingRightService.java:309) at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:270) at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:254) at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567) at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71) at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106) at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) at org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) at com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) at com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) at com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) at com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) at com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180) at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463) at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449) at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215) at com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176) at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) - locked <4da86584> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - locked <1479d987> (a java.util.concurrent.ThreadPoolExecutor$Worker) ---- *Security Cache Write Lock example:* Thread t@178 java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Native Method) - parking to wait for <3081f946> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943) at org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidatorListener.onEvent(DefaultSecurityCacheRulesInvalidatorListener.java:186) at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:304) at org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:269) at org.xwiki.observation.remote.internal.DefaultRemoteObservationManager.notify(DefaultRemoteObservationManager.java:168) at com.wiki.observation.remote.jgroups.JGroupsReceiver.receive(JGroupsReceiver.java:87) at org.jgroups.JChannel.up(JChannel.java:768) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1034) at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) at org.jgroups.protocols.FlowControl.up(FlowControl.java:438) at org.jgroups.stack.Protocol.up(Protocol.java:426) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:487) at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:989) at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:919) at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:851) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) at org.jgroups.protocols.FD.up(FD.java:260) at org.jgroups.protocols.MERGE3.up(MERGE3.java:292) at org.jgroups.protocols.Discovery.up(Discovery.java:296) at org.jgroups.protocols.TP.passMessageUp(TP.java:1657) at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1873) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - locked <2ea19041> (a java.util.concurrent.ThreadPoolExecutor$Worker) -- Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html