25 Oct
2017
25 Oct
'17
6:31 p.m.
We ran into an issue recently where users were not able to access the Wiki
due to the XWiki security cache holding up access to read requests as we
were experiencing heap exhaustion. A thread dump of a host showed that there
were 211 security cache read locks and 2 security cache write locks during
the event. Read locks are supposedly parallelizable, so we suspect that it
was the write locks that blocked the read requests.
Looking into the write locks, we found
*DefaultSecurityCacheRulesInvalidatorListener* shares a read-write lock
(*SecurityCacheRulesInvalidator*) with the security cache loader
(*DefaultSecurityCacheLoader*). It appears that these write requests to the
security cache were potentially creating a deadlock, blocking any further
read requests.
Our investigation of this event resulted in the following questions that we
are hoping to get answered by reaching out here:
1. What exactly happens in the event of a write failure to the Security
Cache? Are we correct in understanding that the write request will continue
to hold a lock on the cache, preventing all read requests from proceeding?
2. If so, are there any mitigations around the global read-write lock on the
security cache? Or, perhaps, is the best action is to simply move toward a
more scalable caching mechanism (i.e. a centralized cache such as Elastic
Cache)?
3. It looks as if all threads doing a read/write request to the
SecurityCache were all parked waiting for the same thread (3081f946);
however, the thread is not showing up in the thread dump so we are not sure
what it is stuck on. Does the XWiki team have any suspicions as to what
could be hold up the requests to the SecurityCache?
Thank you!
---
For more information on the event:
From our investigation, we concluded that the outage
happened because:
1. Remote events triggered a security cache update and requested a couple of
write locks
2. The writes could not execute (perhaps due to the heap exhaustion,
although root cause of write failure is unknown)
3. ReentrantReadWriteLock lock followed a fair policy, which releases locks
in time order, so read requests were forced to wait for any previous write
requests to finish. Because the write requests couldn't finish, the
remaining 211 threads were blocked.
* Possible remote events that trigger security cache update: changes to an
XWiki group such as leaving the admin group, user first time visiting XWiki,
etc..
----
*LDAP Connection Lock example:*
Thread t@225108
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <7249fc39> (a com.novell.ldap.Connection)
at com.novell.ldap.Connection.waitForReader(Unknown Source)
at com.novell.ldap.Connection.startReader(Unknown Source)
at com.novell.ldap.Connection.connect(Unknown Source)
at com.novell.ldap.Connection.connect(Unknown Source)
at com.novell.ldap.LDAPConnection.connect(Unknown Source)
at
com.ownership.groups.ldap.DefaultLdapConnection.connect(DefaultLdapConnection.java:25)
at
com.ownership.groups.ldap.LdapClient.queryGroups(LdapClient.java:99)
at
com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:64)
at
com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:59)
at
com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:113)
- locked <7cc1cfca> (a java.lang.String)
at
com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:75)
at
org.xwiki.security.internal.DefaultUserBridge.getGroupsReferencesFor(DefaultUserBridge.java:122)
at
org.xwiki.security.internal.DefaultUserBridge.getAllGroupsFor(DefaultUserBridge.java:78)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:350)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:342)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserEntry(DefaultSecurityCacheLoader.java:318)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadAccessEntries(DefaultSecurityCacheLoader.java:221)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadRequiredEntries(DefaultSecurityCacheLoader.java:193)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:148)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109)
at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.displayTitle(AbstractDocumentTitleDisplayer.java:163)
at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:149)
at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:58)
at
org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:96)
at
org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:39)
at
org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:123)
at
org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:52)
at
org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:68)
at
org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:42)
at
com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1538)
at
com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1558)
at com.xpn.xwiki.api.Document.getDisplayTitle(Document.java:345)
at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.runtime.parser.node.PropertyExecutor.execute(PropertyExecutor.java:142)
at
org.apache.velocity.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:539)
at
org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:198)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.apache.velocity.runtime.parser.node.ASTStringLiteral.value(ASTStringLiteral.java:330)
at
org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
at
org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463)
at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210)
at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
at
org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215)
at
com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176)
at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
- locked <1cc68e56> (a org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- locked <79120011> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
----
*Security Cache Read Lock example:*
Thread t@225106
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <3081f946> (a
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at
org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidator.suspend(DefaultSecurityCacheRulesInvalidator.java:56)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:144)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109)
at
org.xwiki.security.authorization.internal.XWikiCachingRightService.hasAccessLevel(XWikiCachingRightService.java:309)
at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:270)
at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:254)
at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at
org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
at
org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463)
at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210)
at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
at
org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215)
at
com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176)
at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
- locked <4da86584> (a org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- locked <1479d987> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
----
*Security Cache Write Lock example:*
Thread t@178
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <3081f946> (a
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at
org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidatorListener.onEvent(DefaultSecurityCacheRulesInvalidatorListener.java:186)
at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:304)
at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:269)
at
org.xwiki.observation.remote.internal.DefaultRemoteObservationManager.notify(DefaultRemoteObservationManager.java:168)
at
com.wiki.observation.remote.jgroups.JGroupsReceiver.receive(JGroupsReceiver.java:87)
at org.jgroups.JChannel.up(JChannel.java:768)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1034)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:438)
at org.jgroups.stack.Protocol.up(Protocol.java:426)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:487)
at
org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:989)
at
org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:919)
at
org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:851)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
at org.jgroups.protocols.FD.up(FD.java:260)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:292)
at org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1657)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1873)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- locked <2ea19041> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html
26 Oct
26 Oct
9:28 a.m.
Hi Ktc,
Thanks for this detailed email with lots of details and hard questions!
This is not an easy mail to answer and I certainly don’t have the answers.
You’re the first one to report such issue AFAIK and that’s most certainly due to the scale
at which you use XWiki :)
Let’s see if someone has the answer here. If you don’t get any answer, know that there’s
an option for you which is to contract a company offering dedicated professional support
for XWiki and have them spend quality time debugging this with you (see
http://www.xwiki.org/xwiki/bin/view/Main/Support#HProfessionalSupport).
If you find the solution, some Pull Requests would be much appreciated to fix the
problems.
Thanks
-Vincent
On 25 Oct 2017, at 18:31, ktc
<ktc-wiki(a)amazon.com> wrote:
We ran into an issue recently where users were not able to access the Wiki
due to the XWiki security cache holding up access to read requests as we
were experiencing heap exhaustion. A thread dump of a host showed that there
were 211 security cache read locks and 2 security cache write locks during
the event. Read locks are supposedly parallelizable, so we suspect that it
was the write locks that blocked the read requests.
Looking into the write locks, we found
*DefaultSecurityCacheRulesInvalidatorListener* shares a read-write lock
(*SecurityCacheRulesInvalidator*) with the security cache loader
(*DefaultSecurityCacheLoader*). It appears that these write requests to the
security cache were potentially creating a deadlock, blocking any further
read requests.
Our investigation of this event resulted in the following questions that we
are hoping to get answered by reaching out here:
1. What exactly happens in the event of a write failure to the Security
Cache? Are we correct in understanding that the write request will continue
to hold a lock on the cache, preventing all read requests from proceeding?
2. If so, are there any mitigations around the global read-write lock on the
security cache? Or, perhaps, is the best action is to simply move toward a
more scalable caching mechanism (i.e. a centralized cache such as Elastic
Cache)?
3. It looks as if all threads doing a read/write request to the
SecurityCache were all parked waiting for the same thread (3081f946);
however, the thread is not showing up in the thread dump so we are not sure
what it is stuck on. Does the XWiki team have any suspicions as to what
could be hold up the requests to the SecurityCache?
Thank you!
---
For more information on the event:
From our investigation, we concluded that the outage happened because:
1. Remote events triggered a security cache update and requested a couple of
write locks
2. The writes could not execute (perhaps due to the heap exhaustion,
although root cause of write failure is unknown)
3. ReentrantReadWriteLock lock followed a fair policy, which releases locks
in time order, so read requests were forced to wait for any previous write
requests to finish. Because the write requests couldn't finish, the
remaining 211 threads were blocked.
* Possible remote events that trigger security cache update: changes to an
XWiki group such as leaving the admin group, user first time visiting XWiki,
etc..
----
*LDAP Connection Lock example:*
Thread t@225108
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <7249fc39> (a com.novell.ldap.Connection)
at com.novell.ldap.Connection.waitForReader(Unknown Source)
at com.novell.ldap.Connection.startReader(Unknown Source)
at com.novell.ldap.Connection.connect(Unknown Source)
at com.novell.ldap.Connection.connect(Unknown Source)
at com.novell.ldap.LDAPConnection.connect(Unknown Source)
at
com.ownership.groups.ldap.DefaultLdapConnection.connect(DefaultLdapConnection.java:25)
at
com.ownership.groups.ldap.LdapClient.queryGroups(LdapClient.java:99)
at
com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:64)
at
com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:59)
at
com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:113)
- locked <7cc1cfca> (a java.lang.String)
at
com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:75)
at
org.xwiki.security.internal.DefaultUserBridge.getGroupsReferencesFor(DefaultUserBridge.java:122)
at
org.xwiki.security.internal.DefaultUserBridge.getAllGroupsFor(DefaultUserBridge.java:78)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:350)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:342)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserEntry(DefaultSecurityCacheLoader.java:318)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadAccessEntries(DefaultSecurityCacheLoader.java:221)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadRequiredEntries(DefaultSecurityCacheLoader.java:193)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:148)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109)
at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.displayTitle(AbstractDocumentTitleDisplayer.java:163)
at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:149)
at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:58)
at
org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:96)
at
org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:39)
at
org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:123)
at
org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:52)
at
org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:68)
at
org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:42)
at
com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1538)
at
com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1558)
at com.xpn.xwiki.api.Document.getDisplayTitle(Document.java:345)
at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.runtime.parser.node.PropertyExecutor.execute(PropertyExecutor.java:142)
at
org.apache.velocity.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:539)
at
org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:198)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.apache.velocity.runtime.parser.node.ASTStringLiteral.value(ASTStringLiteral.java:330)
at
org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
at
org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463)
at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210)
at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
at
org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215)
at
com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176)
at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
- locked <1cc68e56> (a org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- locked <79120011> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
----
*Security Cache Read Lock example:*
Thread t@225106
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <3081f946> (a
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at
org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidator.suspend(DefaultSecurityCacheRulesInvalidator.java:56)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:144)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109)
at
org.xwiki.security.authorization.internal.XWikiCachingRightService.hasAccessLevel(XWikiCachingRightService.java:309)
at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:270)
at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:254)
at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at
org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
at
org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463)
at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210)
at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
at
org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215)
at
com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176)
at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
- locked <4da86584> (a org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- locked <1479d987> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
----
*Security Cache Write Lock example:*
Thread t@178
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <3081f946> (a
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at
org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidatorListener.onEvent(DefaultSecurityCacheRulesInvalidatorListener.java:186)
at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:304)
at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:269)
at
org.xwiki.observation.remote.internal.DefaultRemoteObservationManager.notify(DefaultRemoteObservationManager.java:168)
at
com.wiki.observation.remote.jgroups.JGroupsReceiver.receive(JGroupsReceiver.java:87)
at org.jgroups.JChannel.up(JChannel.java:768)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1034)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:438)
at org.jgroups.stack.Protocol.up(Protocol.java:426)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:487)
at
org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:989)
at
org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:919)
at
org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:851)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
at org.jgroups.protocols.FD.up(FD.java:260)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:292)
at org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1657)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1873)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- locked <2ea19041> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html
2579
days inactive
2580
days old
1 comments
2 participants
participants (2)
-
ktc -
Vincent Massol