Re: [xwiki-devs] [Discussion] Designing the new Rendering/Parsing component/API

Hello! Just some words about what the wiki model is and what it is not. The main goal of the WikiModel is the creation of an API giving access and control to the internal structure of individual wiki documents. Some features of the WikiModel: - WikiModel itself does not depend on any particular wiki syntax - The number of possible structural elements and their possible assembling order is strictly fixed (which greatly simplifies the validation and manipulation) but the final result is almost as expressive as XHTML (and even more expressive, taking into account notions of properties and embedded documents which can recursively contain their own embedded documents :-)). - WikiModel manipulates with a super-set of structural elements available in existing wikis. And it has some features not available in other wikis. For example using embedded documents in WikiModel it is possible to put a table in a list and this table can contains its own headers, paragraphs, and lists... Or using embedded documents with the notion of properties it is possible to define very complex structured objects directly on a wiki page. - There is at least one wiki syntax ("Common" syntax) giving access to all features of the Wiki Model. This syntax guaranties that all structural elements of the WikiModel can be serialized/de- serialized without loose of information and structure. Using any other syntaxes can lead to the information lost (example: you can not put table in a table in XWiki or in JSPWiki which is possible using the Common Syntax). - One of the goals of the WikiModel is to give a mean to *import* information from various wiki engines without information lost. The structure of documents can be serialized in various wiki syntaxes as well, but there is no guaranties that some information will not be lost. The information can be lost in the case when a document contains some elements which have no representation in a particular wiki syntax. Example: properties; tables in lists; parameters of lists, paragraphs, and tables and so on... - All elements managed by the WikiModel can be serialized/ deserialized using XHTML with additional annotations (microformat- like annotations) Some features of the CommonSyntax: - It is a native syntax for the WikiModel. It provides full access to all features of the WikiModel. All structures in the WikiModel can be serizlized/deszerialized in this syntax without any information lost - It uses markup characters available in most (in ideal situation - in all) keyboard layouts (including Russian :-)). So you don't have to switch keyboard layouts to write text, tables, lists and headers. For example tables can be defined using pipe symbols ("|" - which is not available in many keyboard layouts) or the "::" sequence. - If there is a choice then the most commonly used markups are used The current version of the WikiModel provides just an event-based interface to work with the structure of documents (like SAX for XML). In previous versions of WIkiModel I had Document Object Model in which each structural element had its own object representation. In the current version an Object Model is not implemented (yet). I thought to create just a set of utility classes manipulating with the standard XML DOM. Example: the method WikiTable#setCellContent (int row, int column, String content) should create an XHTML table object, create the required number of cells and columns and put the given string content in this node. The same for all other structural elements (headers, lists, internal documents, properties, styles, macros...) On 9/14/07, Vincent Massol <vincent(a)massol.net > wrote: +1 to all that. So let me summarizes and rephrase to see if I have understood :) 1) We have 4 types of objects: * TextProcessors: take text and generate text * Parsers: take text and generate an internal DOM format (pivot format) * DomProcessors: take DOM and generate DOM * Renderers: take DOM and generate anything (text, PDF, RTF, HTML, XML, etc) Yes. 2) Document contents are stored in the database in textual format in the main xwiki syntax (whatever we decide it is - we could standardize on creole for example) It can be the "Common Syntax" for the reasons mentioned above :-). Creole syntax is one of the most restrictive syntaxes. And I tried to uses in the CommonSyntax as much markups of the Creole as possible. An another possibility is to store directly in XML or in XHTML +microformat enhancements (for additional structural elements). pro: - it can be exported/imported directly and used by external applications which knows nothing about wikis; just a standard XML or XHTML - this content can be transformed with XSLT processors directly without usage of the WikiModel - it can be faster to parse XML than the CommonWiki syntax (I have no comparisons) con: - it is more difficult to work with diffs (but for diffs it is *better* to use WkiModel and to generate a specific wiki syntax; for example "Common syntax"); - it is not a "human readable" format; it is difficult to understand what you loads from the DB 3) Use case 1: Viewing a document a) Get the doc from the DB --> text1 (xwiki text format) b) Apply TextProcessors --> text2 c) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an internal DOM) d) Apply DomProcessors --> DOM2 e) Call the required Renderer --> PDF, XML, HTML, RTF, text, etc Yes. 4) Use case 2: Editing a document, assuming the user wants to use the MediaWiki syntax for editing a) Get the doc from the DB --> text1 (xwiki text format) b) Call XWikiParser --> DOM1 (transforms XWiki text syntax into an internal DOM) c) Call MediaWikiRenderer --> text2 (text in MediaWiki format) d) the user edits and hits save e) MediaWikiParser --> DOM2 (transforms MediaWiki text syntax into the internal DOM) f) Call XWikiRenderer --> text" (transforms DOM into xwiki textual format) g) Save text3 in the database Yes. (text1 and text3 can be XML, as I said above) 5) In practice this means the following classes: * TextProcessorManager: to chain several text processors Yes. But it can be just a composite processor implementing the same ProcessorManager interfaces. * TextProcessor - VelocityTextProcessor - GroovyTextProcessor Yes. * WikiParser: Takes wiki syntax and generates a DOM in a XWiki- specific format (independent of the different wiki syntaxes). - LegacyXWikiWikiParser - XWikiWikiParser (or simply use CreoleWikiParser if we want our internal format to be Creole) - ConfluenceWikiParser - MediaWikiWikiParser - JSPWikiWikiParser - CreoleWikiParser - HTMLParser: I think all parsers above need to support HTML since the wiki syntaxes can be mixed with HTML. So this HTMLParser is probably a parent of the other parsers in some regard. Anyway we need this one for the WYSIWYG editor which may need to transform HTML to wiki syntax (so we may need a XWikiDomProcessor too to transform into XWiki syntax). The alternative (much better) is to have the WYSIWYG editor only use the internal XWiki-specific DOM format for all its manipulations. If you want, you can put HTML as a non-interpreted block ("verbatim blocks") and interpret it in the client code. But internally the WikiModel does not support "embedded" (X)HTML. The main reason: in this case I loose control of the document structure. And this control is the main goal of the WikiModel. * DomProcessorManager: to chain several DOM processors * DomProcessor - Don't know yet what we're going to use this for. TOCDomProcessor as you say above maybe. DOMProcessor can be used to transform the original DOM object representing the document in the DB into a new (user and query- specific) DOM object which can contain new elements, generated dynamically. Now all dynamic page elements are interpreted as simple Velocity or Groovy scripts and they generate text documents which should be parsed using Radeox and transformed to the final HTML document. Using the DOM representation it is possible to interpret some nodes of this graph as Groovy scripts. In WikiModel they will correspond to Verbatim blocks which are opaque for WikiModel but they can be interpreted as scripts by the DomProcessor (s). And these "Groovy"-nodes can be executed and they will add new DOM elements to the DOM2. For example this approach can be used to generate search results. The advantages of this approach: - You can put your parsed document DOM1 in the cache, which will avoid you to to parse the document for each query. It is a slowest step in the page processing. Even if the current version of WikiModel is faster than before and it should be faster than Radeox processor. - Your Groovy scripts will manipulate with normal java classes (DOM nodes) and it will produce DOM nodes and not a plain text. It seems especially interesting taking into account Groovy's Builders ( http://groovy.codehaus.org/Builders). It is enough to write a very simple builder (see http://groovy.codehaus.org/BuilderSupport) generating DOM nodes and ... voila! Your Groovy node from a wiki page generates search results as DOM nodes! These manipulations with DOM objects should be MUCH faster that process plain text for every request. And all following steps are fast as well - to generate an HTML page it is enough to visit all node with an "XHTMLVisitor". BTW: do you need Velocity at all? Using only Groovy is much cleaner. It can be used as THE language of XWiki. It can be used as a template *and* programming language at the same time. And if you *really* want it is possible to integrate Jasper (from Tomcat) engine to use it for pure templating. The code from Jetty (th e org.mortbay.jetty.jspc.plugin package) can be used as an example of integration with Jasper (see http://jetty.mortbay.org/xref/ index.html). In this case in templates it will be possible to use: - JSP tag libraries (including standard ones) - Multiple scripting languages (like javabeans, javascript, jpython, jruby, groovy,...) * Renderer - XMLRenderer - HTMLRenderer - PDFRenderer - RTFRenderer - XWikiRenderer (or simply use CreoleRenderer if we want our internal format to be Creole) - ConfluenceRenderer - MediaWikiRenderer - JSPWikiRenderer - CreoleRenderer Yes. All these renderers should be written if you want to support all these syntaxes. I think that it should not be very difficult. WDYT? Do I have it right? :) Best regards, Mikhail Thanks -Vincent On Sep 13, 2007, at 6:37 PM, Stéphane Laurière wrote: Note GWT the (example: PDF that property: a MediaWiki, the syntaxes to with document serialized document item, not, "CommonSyntax", XML, content >> and generate wiki syntax from it. >> >> >> This is my very earlier thinking but I wanted to make it visible to >> give everyone the change to 1) know what's happening and 2) suggest >> ideas. >> >> I'll refine this in the coming days and post again on this thread. >>