Re: [xwiki-devs] [Vote] Add a cache directory concept and make the XWiki launch faster (was: Move the caches files to permdir and make the launch of XWiki faster)

On 17 Feb 2015 at 14:14:36, Thomas Mortagne (thomas.mortagne@xwiki.com(mailto:thomas.mortagne@xwiki.com)) wrote: > On Tue, Feb 17, 2015 at 2:04 PM, vincent(a)massol.net wrote: > > Just to be clear: I’m -1 till I understand why we need a new concept and why deciding where we put our temporary directory is not good enough. > > > > To summarize my position: > > * Cache files are temporary files. They can be recreated. And it’s not because files are in a temporary folder that they have to be deleted every few minutes :) > > > * In order to prevent some process to mess with our temp files when XWiki is running (for ex) we could want to control where we put our temporary directory. This is already configurable and doable > > Temporary directory is not configurable at XWiki level so it depends > on the application server. Yes I know and the idea would be to introduce a configuration property in xwiki.properties to set the temporary directory in case the user wants to change it. FTR I’ve checked Jetty/Tomcat: * By default recent versions of jetty use the java.io.tmdir by default if not set. We could easily set it in our standalone distribution (we used to have a jetty/temp dir AFAIR). Also not that by default Jetty cleans the temp dir at each restart unless you tell it not to or unless you put your tmp dir inside a work directory. See see http://eclipse.org/jetty/documentation/current/ref-temporary-directories.ht…. Note that the work dir in jetty is just there for backward compat and jetty only uses a temp dir now apparently. * Tomcat has the 2 notions of work and temp directories. I don’t think it cleans the temp directory automatically at each restart though. So we have 2 options: Option 1: we don’t introduce a new concept and we just make our temporary directory configurable Option 2: we do the same as tomcat and introduce a concept of work/cache directory. I think I prefer “work” over “cache” if I have to choose since it’s less restrictive. In this case the difference is small: - temporary directory: anything in there can (and this is what happens with a default, non-configured jetty) be removed between restarts of the container without impacting anything at runtime (no performance degradation for example). This is really used for short-lived temporary files that should be removed when the container stops. - work/cache directory: files that can be removed without endangering the runtime execution of XWiki but it must not be removed at each container restart. Should be removed when performing an XWiki upgrade though. Where would we put where:

- work dir: solr/lucene indexes, LESS cache, image resizes, chart-generated images, aether cache, formula-generates images, office viewer, graphviz image cache, svg image cache, TempResourceAction

- temporary dir: mail sender AttachmentMimeBodyPartFactory, office conversion result, oldcore attachment content cache, html export, pdf export So, after more research and thinking, I think there’s a case for differentiating between temp and work/cache directories, as defined above. So I’m +1 for option 2, based on that. Thanks -Vincent > > Now when in a Servlet environment we already put temporary data in the Servlet’s temporary directory (which is **NOT** the OS temporary folder). That container temporary folder is not deleted or removed by any process. It is already a safe place where to put our data. That’s why it exists! ;) > > > > So I wouldn’t change anything in the end ;) > > > > What have I missed? > > > > Thanks > > -Vincent > > > > > > On 17 Feb 2015 at 13:44:24, vincent(a)massol.net (vincent@massol.net(mailto:vincent@massol.net)) wrote: > > > >> Hi, > >> > >> On 16 Feb 2015 at 17:13:45, Guillaume Louis-Marie Delhumeau (gdelhumeau@xwiki.com(mailto:gdelhumeau@xwiki.com)) wrote: > >> > >> > Thanks for your answers. > >> > > >> > I change the name of this thread since my proposal has changed: > >> > > >> > 1 - Add getCacheDirectory() in the class Environment: > >> > https://github.com/xwiki/xwiki-commons/blob/master/xwiki-commons-core/xwiki… > >> > >> So if I understand correctly: > >> * Permanent directory: mandatory data that is needed for XWiki to work and that you need to keep when you upgrade your XWiki instance > >> * Cache directory: data used to make XWiki faster but the data in there is not mandatory (it can be deleted) and is recreated if not found > >> > >> Now what about the temp dir: > >> * Temporary directory: not used? > >> > >> Basically it seems we want a controlled temporary directory, correct? What would we put in the temporary directory? Data that can be removed between XWiki startups? The difference seems pretty slim and your proposal doesn’t explain that. > >> > >> At this point I’m not sure I understand enough the differences between the Cache dir and the Temp dir to vote. > > Yes something you can recreate but that make sense to keep between > restarts (which is far from being the case of all temporary stuff). > IMO we would also move embedded solr index in this new directory for > example. > > >> > >> > It means breaking an interface. > >> > > >> > 2 - Add a cache directory setting in xwiki.properties, to change this > >> > directory if needed (for example: set a directory on a faster disk) > >> > > >> > 3 - For our distributions, set this cache directory in a folder inside our > >> > distributions (for example a "cache" directory aside the "data" directory). > >> > This setting can be changed by the user. > >> > >> This will work only for the Standalone zip distribution and the Debian one. What would be the default location for the Debian distribution? > > As I suggested in the other mail thread the default path for Debian > package would most probably be /var/cache/xwiki/. What is important is > that it's configurable at build time like permanent dir is. > > >> > >> What would be the default value for the WAR distribution? > >> > >> I think I’d put the temporary directory as default and issue a warning, as we do for the Permanent directory, WDYT? Technically this means using the temporary directory in the Environment’s implementationq (StandardEnvironment, ServletEnvironment) by default if not set. > >> > >> > 4 - Do not purge the LESS cache on startup. > >> > (see my previous mail for this) > >> > >> It’s not only LESS. This whole cache directory should not be purged on startup. > > Guillaume is talking about LESS infinispan filesystem cache configuration here. > > >> > >> > 5 - add a new module that pre-generate the LESS cache file to make the > >> > first XWiki launch faster > >> > (see my previous mail for this) > >> > > >> > +1 for all of this. > >> > >> +0 on the principle but I haven’t checked the complexity that it brings. I guess I need to read that other thread first (will reply there if I have any comment). > >> > >> [snip] > >> > >> Thanks > >> -Vincent > >> > >> > >> > >> > > _______________________________________________ > > devs mailing list > > devs(a)xwiki.org > > http://lists.xwiki.org/mailman/listinfo/devs > > > > -- > Thomas Mortagne > _______________________________________________ > devs mailing list > devs(a)xwiki.org > http://lists.xwiki.org/mailman/listinfo/devs _______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs