30 Mar
2007
30 Mar
'07
2:06 p.m.
Hi,
I'm investigating what it would take to store (or duplicate) some of
the data in XWiki's database in Subversion.
Rationale:
- A plain text version of wiki pages stored in SVN buys you:
- off-line edit, merge, etc
- access to the pages using other tools, editors and so on
- replication options
- versionning, branching etc, and version display via e.g. ViewCVS
- insulation from database crashes, with easier incremental backup
- a kind of poor-man's content management system
- SVN is very robust, reliable, and has a mature Java interface
(SVNKit)
- if Tomcat crashes a static version of the site can still be served
- I have a large static site that is managed in SVN (8 Gb in 40,000
files) which I want to migrate to a more dynamic system.
This looks like a significant size job, but we have a fair amount of
resources in my team, and possibly we might fund some of the work in
the XWiki community if anyone is interested. I wonder if this type of
idea has been discussed before, and if anyone has comments on the best
way to go about the problem?
It seems to me that trying to replace the database entirely with a
file-based store is too much work and would lead to performance
problems, but perhaps it would be possible to have a process which
pushed changes from the database into an SVN working copy and vice
versa?
Where would be good places to start looking at the code from this
point of view?
I also need to include a JavaCC-based wiki-to-html parser that we
have as an option, but I'm betting that's an easier task?
Thanks, best,
Hamish
--
Prof. Hamish Cunningham
Department of Computer Science
University of Sheffield
Regent Court
211 Portobello St.
Sheffield S1 4DP
United Kingdom
http://www.dcs.shef.ac.uk/~hamish/
30 Mar
30 Mar
2:28 p.m.
New subject: [xwiki-dev] XWiki and SVN as a backend
Hi Hamish,
On Mar 30, 2007, at 4:06 PM, Hamish Cunningham wrote:
Hi,
I'm investigating what it would take to store (or duplicate) some of
the data in XWiki's database in Subversion.
Rationale:
- A plain text version of wiki pages stored in SVN buys you:
- off-line edit, merge, etc
- access to the pages using other tools, editors and so on
- replication options
- versionning, branching etc, and version display via e.g. ViewCVS
- insulation from database crashes, with easier incremental backup
- a kind of poor-man's content management system
- SVN is very robust, reliable, and has a mature Java interface
(SVNKit)
- if Tomcat crashes a static version of the site can still be served
- I have a large static site that is managed in SVN (8 Gb in 40,000
files) which I want to migrate to a more dynamic system.
Yep I agree with most of these but there are limitations (see below).
This looks like a significant size job, but we have a fair amount of
resources in my team, and possibly we might fund some of the work in
the XWiki community if anyone is interested.
That would be real cool.
I wonder if this type of
idea has been discussed before, and if anyone has comments on the best
way to go about the problem?
Yes we've discussed this in the past. Someone even said he had
started working on it but I tried to get a status of the work and
never got an answer so we can consider this dead.
That said, we've also opened a Google Summer of Code project for this
(http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/Subversion
+Integration). If you're interested you could work with your team and
that person on it. We could create a branch in SVN for this work.
It seems to me that trying to replace the database
entirely with a
file-based store is too much work and would lead to performance
problems, but perhaps it would be possible to have a process which
pushed changes from the database into an SVN working copy and vice
versa?
Yes, this is needed for several reasons:
1) Performances but that's not really the main issue as we could use
caching for searching (lucene, etc).
2) XWiki pages can contain objects and we don't currently have a
textual representation of a full page with it's objects, etc apart
from an XML representation but that's not nice for editing at all.
Thus 2 thoughts:
1) It's probably better to decide what pages are stored in SVN and
which pages are stored in the wiki. Indeed some pages such as Classes
probably don't need to be stored in SVN
2) We need to allow storing the page content separately from the rest
of the metadata (class definition, objects, etc)
Based on this, we could possibly implement in 2 ways (haven't thought
more about it):
1) by implementing a XWikiStoreInterface. That means it would get
stored both in the DB and in SVN for some docs. We would chain the
storage systems (that probably requires some minor changes to the core)
2) by using the XWiki notification service. I know there's an event
called when storing. I don't know if there's an event for getting
back a doc from storage. This possibly requires some small changes to
the core.
Obviously one issue is that if we do a svn update before editing a
document (to check if there's a newer version) that'll affect
performances. Thus we might want to schedule updates regularly or
offer a button for the user to do that. In any case it means we need
to cater for SVN merge conflicts and offer a solution for that.
Anyone has any other idea?
I suggest we gather ideas here on the list and when we have some
agreement we can start putting those on a wiki page on xwiki.org.
Hamish, I'd be happy if you wanted to drive this effort.
Where would be good places to start looking at the
code from this
point of view?
See above for some ideas.
I also need to include a JavaCC-based wiki-to-html
parser that we
have as an option, but I'm betting that's an easier task?
Actually Christian and Nam are developing one right now. It would be
cool if you could have a look and possibly help them. It's in SVN in
xwiki/branches/XWIKI_WYSIWYG_NEWARCHI/core/src/main/java/com/xpn/
xwiki/wysiwyg
Thanks
-Vincent
3:19 p.m.
New subject: [xwiki-dev] XWiki and SVN as a backend
vincent
That said, we've also opened a Google Summer of
Code project for this
(http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/Subversion+Integrati…).
If you're interested you could work with your team and that person on
sounds good!
1) It's probably better to decide what pages are
stored in SVN and which
pages are stored in the wiki. Indeed some pages such as Classes probably
don't need to be stored in SVN
2) We need to allow storing the page content separately from the rest of
the metadata (class definition, objects, etc)
I agree that not all of the data needs to be in SVN, but I think it would be
nice if some of the textual content of pages was available for external
editing and so on. An advantage of pushing all the data out into some textual
form (even if it isn't very readable) is that you could then use SVN
replication (or just checking out another copy) to duplicate a site, and
branching for experimenting with a new approach, etc.
Obviously one issue is that if we do a svn update
before editing a
document (to check if there's a newer version) that'll affect
performances. Thus we might want to schedule updates regularly or offer
a button for the user to do that. In any case it means we need to cater
for SVN merge conflicts and offer a solution for that.
Anyone has any other idea?
One option is to get the SVN server to hit a URL whenever anything changes and
trigger import of the changes at that point. Some other mechanism (like the
scheduled update you propose) is also necessary to cope with network outages etc.
I suggest we gather ideas here on the list and when we
have some
agreement we can start putting those on a wiki page on xwiki.org.
Hamish, I'd be happy if you wanted to drive this effort.
I'd like to participate; at present I don't understand well how xwiki works,
so I'm not sure I'm the right person to drive it.
Actually Christian and Nam are developing one right
now. It would be
cool if you could have a look and possibly help them. It's in SVN in
xwiki/branches/XWIKI_WYSIWYG_NEWARCHI/core/src/main/java/com/xpn/xwiki/wysiwyg
We're using a different syntax (see attached) and allowing embedding of
controlled language statements for ontology editing
(http://gate.ac.uk/sale/lrec2006/clie/clie.pdf). Actually if I was going to do
this again (write a wiki parser) I'd be tempted not to use JavaCC, which is
great for fully deterministic parsing but difficult when you want to return a
reasonable translation of messy data even when the author has made mistakes.
Our JAPE regular expressions over annotation graphs tool might have been
easier (http://gate.ac.uk/, imperfectly documented in
http://gate.ac.uk/sale/tao/index.html), but I've invested too much time in the
JavaCC version now!
So this wouldn't replace your existing parser efforts, but I'd like to try and
make it an optional syntax.
Best,
--
Hamish
http://www.dcs.shef.ac.uk/~hamish/
31 Mar
31 Mar
5:17 p.m.
New subject: [xwiki-dev] XWiki and SVN as a backend
Hi,
A quick&simple solution would be to make a cron task that dumps the content
of the wiki periodically (it is done now in XML format) and performs a SVN
commit. Of course, this only resolves the backup part. See below for some
more comments.
On 3/30/07, Hamish Cunningham <hamish(a)dcs.shef.ac.uk> wrote:
vincent
Since the wiki documents can be retrieved as XML, separating the document
content and document metadata is an easy task, which can be performed with
many possible methods: SAX, DOM, XSLT. Moreover, since your documents will
mainly contain text, without the need for objects and classes, it might be a
good idea to store just the content and simple metadata (author and
modification time) in SVN. Even more, XWiki has the notion of spaces, or a
group of related documents. You can put all your documents in spaces which
don't have any XWiki-related content (special class documents, or
templates), and only sync these spaces with the SVN repository.
About branching XWiki can be of use in here, since it supports setting up
virtual wikis on the same server.
That said, we've also opened a Google Summer
of Code project for this
(
http://www.xwiki.org/xwiki/bin/view/GoogleSummerOfCode/Subversion+Integrati…
).
If you're interested you could work with your
team and that person on
sounds good!
1) It's probably better to decide what pages
are stored in SVN and which
pages are stored in the wiki. Indeed some pages such as Classes probably
don't need to be stored in SVN
2) We need to allow storing the page content separately from the rest of
the metadata (class definition, objects, etc)
I agree that not all of the data needs to be in SVN, but I think it would
be
nice if some of the textual content of pages was available for external
editing and so on. An advantage of pushing all the data out into some
textual
form (even if it isn't very readable) is that you could then use SVN
replication (or just checking out another copy) to duplicate a site, and
branching for experimenting with a new approach, etc.
Obviously one issue is that if we do a svn update
before editing a
This is a good idea. I know that SVN allows registering a kind of listeners
(Hook Scripts is the term they use) which can be triggered before or after
committing files. One hook could be used to re-get the content from the wiki
when starting the commit, and one could be used to update the wiki after the
commit. See
http://svnbook.red-bean.com/en/1.0/ch05s02.html#svn-ch-5-sect-2.1 for more
details.
document (to check if there's a newer
version) that'll affect
performances. Thus we might want to schedule updates regularly or offer
a button for the user to do that. In any case it means we need to cater
for SVN merge conflicts and offer a solution for that.
Anyone has any other idea?
One option is to get the SVN server to hit a URL whenever anything changes
and
trigger import of the changes at that point. Some other mechanism (like
the
scheduled update you propose) is also necessary to cope with network
outages etc.
I suggest we gather ideas here on the list and when we
have some
This is easy to accomplish. For most of these items there are existing
filters, only the syntax must be changed, and the rest of them are easy to
add.
In order to change the syntax, you just need to change a file,
core/src/main/resources/radeox_markup_xwiki.properties. In order to add new
filters, I think there is a page describing this on www.XWiki.org. Anyway,
it's not hard at all.
Best,
agreement we can start putting those on a wiki
page on xwiki.org.
Hamish, I'd be happy if you wanted to drive this effort.
I'd like to participate; at present I don't understand well how xwiki
works,
so I'm not sure I'm the right person to drive it.
Actually Christian and Nam are developing one
right now. It would be
cool if you could have a look and possibly help them. It's in SVN in
xwiki/branches/XWIKI_WYSIWYG_NEWARCHI/core/src/main/java/com/xpn/xwiki/wysiwyg
We're using a different syntax (see attached) and allowing embedding of
controlled language statements for ontology editing
(http://gate.ac.uk/sale/lrec2006/clie/clie.pdf). Actually if I was going
to do
this again (write a wiki parser) I'd be tempted not to use JavaCC, which
is
great for fully deterministic parsing but difficult when you want to
return a
reasonable translation of messy data even when the author has made
mistakes.
Our JAPE regular expressions over annotation graphs tool might have been
easier (http://gate.ac.uk/, imperfectly documented in
http://gate.ac.uk/sale/tao/index.html), but I've invested too much time in
the
JavaCC version now!
So this wouldn't replace your existing parser efforts, but I'd like to try
and
make it an optional syntax.
--
Hamish
http://www.dcs.shef.ac.uk/~hamish/ <http://www.dcs.shef.ac.uk/%7Ehamish/>
Regards,
Sergiu Dumitriu
--
http://purl.org/net/sergiu
6717
days inactive
6718
days old
3 comments
3 participants
participants (3)
-
Hamish Cunningham -
Sergiu Dumitriu -
Vincent Massol