Cache contention conundrum (was RE: RSS macro make the entire site to hang)

Ludovic: We've battled this problem for quite a while now, without much success - even to the point of putting a ridiculously long (e.g. > 24hrs) cache lifetime, and independently refreshing the cache. In some cases, we finally just removed the problematic feed. When I built a plugin to cache the results of a long-lived and expensive fetch of an XML document, I found another difficulty that may shed light on this situation: Occasionally, my plugin hangs (forever) even though the fetch isn't having problems. Once this starts, any other thread that attempts to access the cached XML object also hangs. My working hypothesis is that either an error in the fetching - I'm using org.dom4j.io.SAXReader.read(InputStream) - or a user abandoning the page is causing the refresh in one thread to leave the lock in place, causing any other thread to wait for the update, which now won't happen. I haven't been able to verify this, but it's the nearest thing to an explanation I can come up with. If this is the case, I'm thinking that I should set the cache parameters to disable waiting on a refresh, if I can do it at the granularity of an individual cache instance. The correct way is probably to find what's causing the orphaned lock, if that's what's happening. I have not been able to evoke any log output to give me clues on this. brain[sic]