Hi Hamish, On Mar 30, 2007, at 4:06 PM, Hamish Cunningham wrote:
Hi,
I'm investigating what it would take to store (or duplicate) some of the data in XWiki's database in Subversion.
Rationale:
- A plain text version of wiki pages stored in SVN buys you: - off-line edit, merge, etc - access to the pages using other tools, editors and so on - replication options - versionning, branching etc, and version display via e.g. ViewCVS - insulation from database crashes, with easier incremental backup - a kind of poor-man's content management system - SVN is very robust, reliable, and has a mature Java interface (SVNKit) - if Tomcat crashes a static version of the site can still be served - I have a large static site that is managed in SVN (8 Gb in 40,000 files) which I want to migrate to a more dynamic system.
Yep I agree with most of these but there are limitations (see below).
This looks like a significant size job, but we have a fair amount of resources in my team, and possibly we might fund some of the work in the XWiki community if anyone is interested.
That would be real cool.
I wonder if this type of idea has been discussed before, and if anyone has comments on the best way to go about the problem?
Yes we've discussed this in the past. Someone even said he had started working on it but I tried to get a status of the work and never got an answer so we can consider this dead. That said, we've also opened a Google Summer of Code project for this (http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/Subversion +Integration). If you're interested you could work with your team and that person on it. We could create a branch in SVN for this work.
It seems to me that trying to replace the database entirely with a file-based store is too much work and would lead to performance problems, but perhaps it would be possible to have a process which pushed changes from the database into an SVN working copy and vice versa?
Yes, this is needed for several reasons: 1) Performances but that's not really the main issue as we could use caching for searching (lucene, etc). 2) XWiki pages can contain objects and we don't currently have a textual representation of a full page with it's objects, etc apart from an XML representation but that's not nice for editing at all. Thus 2 thoughts: 1) It's probably better to decide what pages are stored in SVN and which pages are stored in the wiki. Indeed some pages such as Classes probably don't need to be stored in SVN 2) We need to allow storing the page content separately from the rest of the metadata (class definition, objects, etc) Based on this, we could possibly implement in 2 ways (haven't thought more about it): 1) by implementing a XWikiStoreInterface. That means it would get stored both in the DB and in SVN for some docs. We would chain the storage systems (that probably requires some minor changes to the core) 2) by using the XWiki notification service. I know there's an event called when storing. I don't know if there's an event for getting back a doc from storage. This possibly requires some small changes to the core. Obviously one issue is that if we do a svn update before editing a document (to check if there's a newer version) that'll affect performances. Thus we might want to schedule updates regularly or offer a button for the user to do that. In any case it means we need to cater for SVN merge conflicts and offer a solution for that. Anyone has any other idea? I suggest we gather ideas here on the list and when we have some agreement we can start putting those on a wiki page on xwiki.org. Hamish, I'd be happy if you wanted to drive this effort.
Where would be good places to start looking at the code from this point of view?
See above for some ideas.
I also need to include a JavaCC-based wiki-to-html parser that we have as an option, but I'm betting that's an easier task?
Actually Christian and Nam are developing one right now. It would be cool if you could have a look and possibly help them. It's in SVN in xwiki/branches/XWIKI_WYSIWYG_NEWARCHI/core/src/main/java/com/xpn/ xwiki/wysiwyg Thanks -Vincent