[xwiki-devs] Contributing MediaWiki to XWiki migration toolkit
Hi, I'm near to finish my MediaWiki2XWiki migration "toolkit". I've developed such, to migrate our homegrown MediaWiki installation to XWiki. The footprint of our Wiki is about 230 pages with around 570 attachments. This will end up in round about 150MB of data. Not that big, but to much to migrate it manually ;-) I'm dumping the MediaWiki content (exported XML) into wikimodel and wrote a XWiki/2.0 serializer. After resolving all templates and downloading the attachments, they get renamed to fit the XWiki rules. Afterwards they are uploaded automatically (including all files and images) to your XWiki instance. The process is splitted into two steps: "extraction and preparation" and "uploading". Everyone with some basic skills in Java and Eclipse can use this "toolkit". But it doesn't claim to be an end user ready to use product ;-) I want to contribute it to your XWiki Code section (apache license). Any doubts from your side or I'm free to do so? Regards Martin
My understanding is contributions to code.xwiki.org can be licensed as anything you want as long as the license is clearly displayed. Thanks for the contribution! Caleb Kirst Martin Wolfgang wrote:
Hi,
I'm near to finish my MediaWiki2XWiki migration "toolkit". I've developed such, to migrate our homegrown MediaWiki installation to XWiki. The footprint of our Wiki is about 230 pages with around 570 attachments. This will end up in round about 150MB of data. Not that big, but to much to migrate it manually ;-)
I'm dumping the MediaWiki content (exported XML) into wikimodel and wrote a XWiki/2.0 serializer. After resolving all templates and downloading the attachments, they get renamed to fit the XWiki rules. Afterwards they are uploaded automatically (including all files and images) to your XWiki instance.
The process is splitted into two steps: "extraction and preparation" and "uploading". Everyone with some basic skills in Java and Eclipse can use this "toolkit". But it doesn't claim to be an end user ready to use product ;-)
I want to contribute it to your XWiki Code section (apache license).
Any doubts from your side or I'm free to do so?
Regards Martin
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Fine with me, thanks for the contribution. Anca On 08/11/2010 03:52 PM, Kirst Martin Wolfgang wrote:
Hi,
I'm near to finish my MediaWiki2XWiki migration "toolkit". I've developed such, to migrate our homegrown MediaWiki installation to XWiki. The footprint of our Wiki is about 230 pages with around 570 attachments. This will end up in round about 150MB of data. Not that big, but to much to migrate it manually ;-)
I'm dumping the MediaWiki content (exported XML) into wikimodel and wrote a XWiki/2.0 serializer. After resolving all templates and downloading the attachments, they get renamed to fit the XWiki rules. Afterwards they are uploaded automatically (including all files and images) to your XWiki instance.
The process is splitted into two steps: "extraction and preparation" and "uploading". Everyone with some basic skills in Java and Eclipse can use this "toolkit". But it doesn't claim to be an end user ready to use product ;-)
I want to contribute it to your XWiki Code section (apache license).
Any doubts from your side or I'm free to do so?
Regards Martin
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Wed, Aug 11, 2010 at 14:52, Kirst Martin Wolfgang <[email protected]> wrote:
Hi,
I'm near to finish my MediaWiki2XWiki migration "toolkit". I've developed such, to migrate our homegrown MediaWiki installation to XWiki. The footprint of our Wiki is about 230 pages with around 570 attachments. This will end up in round about 150MB of data. Not that big, but to much to migrate it manually ;-)
I'm dumping the MediaWiki content (exported XML) into wikimodel and wrote a XWiki/2.0 serializer. After resolving all templates and downloading the attachments, they get renamed to fit the XWiki rules. Afterwards they are uploaded automatically (including all files and images) to your XWiki instance.
The process is splitted into two steps: "extraction and preparation" and "uploading". Everyone with some basic skills in Java and Eclipse can use this "toolkit". But it doesn't claim to be an end user ready to use product ;-)
I want to contribute it to your XWiki Code section (apache license).
Any doubts from your side or I'm free to do so?
Note: there is a wiki importer framework in process (with a MediaWiki module working POC already) on http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-wikiimporter/. It's supposed to be run inside XWiki but most it's code does not really need to be run inside XWiki (basically only the part that save the converted document needs that). Also there is already an official XWiki serializer you can find in the XWiki Rendering framework (see http://code.xwiki.org/xwiki/bin/view/Modules/RenderingModule) that can be used outside of XWiki. The wiki importer framework is based on XWiki Rendering module. That was the note, now let's say competition is good so lets go for it and at some point we should try to share as much as possible the work done ;)
Regards Martin
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
On Wed, Aug 11, 2010 at 14:52, Kirst Martin Wolfgang <[email protected]> wrote:
Hi,
I'm near to finish my MediaWiki2XWiki migration "toolkit". I've developed such, to migrate our homegrown MediaWiki installation to XWiki. The footprint of our Wiki is about 230 pages with around 570 attachments. This will end up in round about 150MB of data. Not that big, but to much to migrate it manually ;-)
I'm dumping the MediaWiki content (exported XML) into wikimodel and wrote a XWiki/2.0 serializer. After resolving all templates and downloading the attachments, they get renamed to fit the XWiki rules. Afterwards they are uploaded automatically (including all files and images) to your XWiki instance.
The process is splitted into two steps: "extraction and preparation" and "uploading". Everyone with some basic skills in Java and Eclipse can use this "toolkit". But it doesn't claim to be an end user ready to use product ;-)
I want to contribute it to your XWiki Code section (apache license).
Any doubts from your side or I'm free to do so?
Note: there is a wiki importer framework in process (with a MediaWiki module working POC already) on
http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-wikiimporter/.
It's supposed to be run inside XWiki but most it's code does not really need to be run inside XWiki (basically only the part that save the converted document needs that).
Also there is already an official XWiki serializer you can find in the XWiki Rendering framework (see http://code.xwiki.org/xwiki/bin/view/Modules/RenderingModule) that can be used outside of XWiki. The wiki importer framework is based on XWiki Rendering module.
That was the note, now let's say competition is good so lets go for it and at some point we should try to share as much as possible the work done ;)
Interesting point. I've watch at it, but it seems to be a different approach to me. Cause, I've had a more complete batch migration in mind, not so much focussed on having a fancy wizard for. Additionally setting up my Eclipse workspace with just XMLRPC-API and wikimodel was way easier ;-) So it was simpler to patch/enhance wikimodel for my tasks. But as I read on page 'RenderingModel' your importer also uses wikimodel. Thus my patches also benefit your project :-) And of course I'm open for sharing work and thoughts. Regards Martin
On Fri, Aug 13, 2010 at 16:32, Kirst Martin Wolfgang <[email protected]> wrote:
On Wed, Aug 11, 2010 at 14:52, Kirst Martin Wolfgang <[email protected]> wrote:
Hi,
I'm near to finish my MediaWiki2XWiki migration "toolkit". I've developed such, to migrate our homegrown MediaWiki installation to XWiki. The footprint of our Wiki is about 230 pages with around 570 attachments. This will end up in round about 150MB of data. Not that big, but to much to migrate it manually ;-)
I'm dumping the MediaWiki content (exported XML) into wikimodel and wrote a XWiki/2.0 serializer. After resolving all templates and downloading the attachments, they get renamed to fit the XWiki rules. Afterwards they are uploaded automatically (including all files and images) to your XWiki instance.
The process is splitted into two steps: "extraction and preparation" and "uploading". Everyone with some basic skills in Java and Eclipse can use this "toolkit". But it doesn't claim to be an end user ready to use product ;-)
I want to contribute it to your XWiki Code section (apache license).
Any doubts from your side or I'm free to do so?
Note: there is a wiki importer framework in process (with a MediaWiki module working POC already) on
http://svn.xwiki.org/svnroot/xwiki/contrib/sandbox/xwiki-wikiimporter/.
It's supposed to be run inside XWiki but most it's code does not really need to be run inside XWiki (basically only the part that save the converted document needs that).
Also there is already an official XWiki serializer you can find in the XWiki Rendering framework (see http://code.xwiki.org/xwiki/bin/view/Modules/RenderingModule) that can be used outside of XWiki. The wiki importer framework is based on XWiki Rendering module.
That was the note, now let's say competition is good so lets go for it and at some point we should try to share as much as possible the work done ;)
Interesting point. I've watch at it, but it seems to be a different approach to me. Cause, I've had a more complete batch migration in mind, not so much focussed on having a fancy wizard for.
I don't understand what you mean here, wiki importer goal is to import complete wikis. The UI part is just on the framework side and when you wriite a wiki importer module you don't really care about that and just declare what you want to get as parameter (the file containing an exported wiki, the URL of a wiki to import using REST, etc...), then you do whatever you want with this. IMO both project could use the exact same code for the MediaWiki -> XWiki convertion which is what is in the MediaWiki module of wiki importer, then the UI and the way to store the produced documents depends of the tool.
Additionally setting up my Eclipse workspace with just XMLRPC-API and wikimodel was way easier ;-) So it was simpler to patch/enhance wikimodel for my tasks.
With M2Eclipse it's very easy to use XWiki Rendering in a new project.
But as I read on page 'RenderingModel' your importer also uses wikimodel. Thus my patches also benefit your project :-)
Actually wiki importer use XWiki Rendering which use WikiModel specifically for the MediaWiki parser but use it's own XWiki 2.0 serializer, use Doxia for other languages etc...
And of course I'm open for sharing work and thoughts.
Regards Martin _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
[...] Interesting point. I've watch at it, but it seems to be a different approach to me. Cause, I've had a more complete batch migration in mind, not so much focussed on having a fancy wizard for.
I don't understand what you mean here, wiki importer goal is to import complete wikis. The UI part is just on the framework side and when you wriite a wiki importer module you don't really care about that and just declare what you want to get as parameter (the file containing an exported wiki, the URL of a wiki to import using REST, etc...), then you do whatever you want with this. Oh, than I misunderstood this importer project. Sorry about that.
IMO both project could use the exact same code for the MediaWiki -> XWiki convertion which is what is in the MediaWiki module of wiki importer, then the UI and the way to store the produced documents depends of the tool.
Additionally setting up my Eclipse workspace with just XMLRPC-API
and
wikimodel was way easier ;-) So it was simpler to patch/enhance wikimodel for my tasks.
With M2Eclipse it's very easy to use XWiki Rendering in a new project. Hmm, not to my experience. I've tried maven a couple of times for different open source projects I've worked on. I'm willing to learn about, but most often I was disappointed. Even after reading the XWiki building instructions it tooks me hours to brought up an error free workspace. Even 'svn co xwiki core && mvn install' is not possible without manually importing some unresolved JARs - easy to solve, but annoying. After all, I've ended up with problems regarding missing dependencies of XWiki's own DI framework. Oh ... I'm drifting away ;-)
But as I read on page 'RenderingModel' your importer also uses wikimodel. Thus my patches also benefit your project :-)
Actually wiki importer use XWiki Rendering which use WikiModel specifically for the MediaWiki parser but use it's own XWiki 2.0 serializer, use Doxia for other languages etc... My patches also targeting the parser code ... Writing my own serializer was more a need because wikimodel doesn't have one and I didn't want to have xwiki code base in my dependencies (see above).
I would really give the importer framework a try, but currently I don't know where to start actually - and POC or not, it shouldn't be take any skilled Java developer more than 2 hours to bring it up and running. It would be easier for me and others, when there would be a documentation "how to setup (sandbox) xwiki importer framework". Then, most likely my starting point would have been a different one. Regards Martin
On Fri, Aug 13, 2010 at 18:39, Kirst Martin Wolfgang <[email protected]> wrote:
[...] Interesting point. I've watch at it, but it seems to be a different approach to me. Cause, I've had a more complete batch migration in mind, not so much focussed on having a fancy wizard for.
I don't understand what you mean here, wiki importer goal is to import complete wikis. The UI part is just on the framework side and when you wriite a wiki importer module you don't really care about that and just declare what you want to get as parameter (the file containing an exported wiki, the URL of a wiki to import using REST, etc...), then you do whatever you want with this. Oh, than I misunderstood this importer project. Sorry about that.
IMO both project could use the exact same code for the MediaWiki -> XWiki convertion which is what is in the MediaWiki module of wiki importer, then the UI and the way to store the produced documents depends of the tool.
Additionally setting up my Eclipse workspace with just XMLRPC-API
and
wikimodel was way easier ;-) So it was simpler to patch/enhance wikimodel for my tasks.
With M2Eclipse it's very easy to use XWiki Rendering in a new project. Hmm, not to my experience. I've tried maven a couple of times for different open source projects I've worked on. I'm willing to learn about, but most often I was disappointed. Even after reading the XWiki building instructions it tooks me hours to brought up an error free workspace.
I do that all the time when we create a new branch of XWiki and it is always very easy.
Even 'svn co xwiki core && mvn install' is not possible without manually importing some unresolved JARs - easy to solve, but annoying.
This should never happen and when it does weusually see and fix it very quickly.
After all, I've ended up with problems regarding missing dependencies of XWiki's own DI framework. Oh ... I'm drifting away ;-)
But as I read on page 'RenderingModel' your importer also uses wikimodel. Thus my patches also benefit your project :-)
Actually wiki importer use XWiki Rendering which use WikiModel specifically for the MediaWiki parser but use it's own XWiki 2.0 serializer, use Doxia for other languages etc... My patches also targeting the parser code ... Writing my own serializer was more a need because wikimodel doesn't have one and I didn't want to have xwiki code base in my dependencies (see above).
Again using XWiki Rendering is like using any other independent tool like wikimodel. It has been designed to be used outside of XWiki core, the only other XWiki module it really depends on is the XWiki Component Manager module which is also independent and pretty small. Basically you make your project depends on http://maven.xwiki.org/snapshots/org/xwiki/platform/xwiki-core-rendering-syn... and see what dependencies you get. It will be more dependencies than wikimodel of course but not so much and it would be a lot better to use the official XWiki serializer and is a big work and is a well tested.
I would really give the importer framework a try, but currently I don't know where to start actually - and POC or not, it shouldn't be take any skilled Java developer more than 2 hours to bring it up and running.
It would be easier for me and others, when there would be a documentation "how to setup (sandbox) xwiki importer framework". Then, most likely my starting point would have been a different one.
Sure, thing is there is still crappy things in the wiki importer framework I really don't like and that I would like to change when i have some time before putting it in standard with the MediaWiki module, then i will write some documentation for people to add more importers.
Regards Martin _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
participants (4)
-
Anca Luca -
Caleb James DeLisle -
Kirst Martin Wolfgang -
Thomas Mortagne