[xwiki-devs] Parser Issuer
Hi vincent Now I'm working on the costume implementation of the IDocument Partitioner which uses the XDOM parser. But in order to to register partitions with the IDocument,requires the offset and the length of each token returned by the parser. But the current implementation of the parser don't have that information.Do I need to look at that issuer as well or what? I have very little understanding of the implementation of the parser and I will be difficult for me to implement that functionality. Vincent can you help me on this issuer. -- Malaka Ekanayake CSE UOM
Hi Malaka, On Jul 1, 2008, at 6:20 AM, malaka ekanayake wrote:
Hi vincent
Now I'm working on the costume implementation of the IDocument Partitioner which uses the XDOM parser. But in order to to register partitions with the IDocument,requires the offset and the length of each token returned by the parser.
Did you get Marius' input on this? I remember I said: " * Re the offset and lengthn I think this is something interesting that could be added to the new rendering module. Since the underlying parsers (wikimodel, doxia) don't pass this information to us we would need to compute it. It would allow us to answer the question "what block is located at location NN in the source document". * We need to get Marius' input since he's working on the new WYSIWYG editor for XWiki and maybe he has some similar needs/issues. " The reason I said this is because if the only requirement is for your use case then you could easily compute the offset/lenght for your use case only (by writing a very simple Listener and running XDOM.traverse(yourllistener). However if the need is generic then it makes sense to have it in the rendering component itself. I'm still hesitating about the generic need about this. If we agree to add it in the rendering component then I can implement it for you. Another question for you: once you have this information what's the algorithm for finding the block that corresponds to a given location in the source document? Dichotomy on the tree? Thanks -Vincent
But the current implementation of the parser don't have that information.Do I need to look at that issuer as well or what? I have very little understanding of the implementation of the parser and I will be difficult for me to implement that functionality.
Vincent can you help me on this issuer.
-- Malaka Ekanayake CSE UOM
Just thought about a potential use case for storing the offset: error reporting. We'll also need it if we want to do auto completion in the wiki editor which I think would be nice to have. Ok I'll try to add offset/length to the rendering module. It's going to cost us some performance actually since when there are block transformations you need to recompute the blocks offsets... I still need to know how it's best to store them for your needs Malaka. Storing them in the Blocks themselves might cost too much to retrieve them. I could also store them in some other structure with references to the blocks. Let me know what you need. Thanks -Vincent On Jul 1, 2008, at 10:19 AM, Vincent Massol wrote:
Hi Malaka,
On Jul 1, 2008, at 6:20 AM, malaka ekanayake wrote:
Hi vincent
Now I'm working on the costume implementation of the IDocument Partitioner which uses the XDOM parser. But in order to to register partitions with the IDocument,requires the offset and the length of each token returned by the parser.
Did you get Marius' input on this?
I remember I said:
" * Re the offset and lengthn I think this is something interesting that could be added to the new rendering module. Since the underlying parsers (wikimodel, doxia) don't pass this information to us we would need to compute it. It would allow us to answer the question "what block is located at location NN in the source document". * We need to get Marius' input since he's working on the new WYSIWYG editor for XWiki and maybe he has some similar needs/issues. "
The reason I said this is because if the only requirement is for your use case then you could easily compute the offset/lenght for your use case only (by writing a very simple Listener and running XDOM.traverse(yourllistener). However if the need is generic then it makes sense to have it in the rendering component itself.
I'm still hesitating about the generic need about this.
If we agree to add it in the rendering component then I can implement it for you.
Another question for you: once you have this information what's the algorithm for finding the block that corresponds to a given location in the source document? Dichotomy on the tree?
Thanks -Vincent
But the current implementation of the parser don't have that information.Do I need to look at that issuer as well or what? I have very little understanding of the implementation of the parser and I will be difficult for me to implement that functionality.
Vincent can you help me on this issuer.
-- Malaka Ekanayake CSE UOM
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On 1 juil. 08, at 11:12, Vincent Massol wrote:
Just thought about a potential use case for storing the offset: error reporting.
We'll also need it if we want to do auto completion in the wiki editor which I think would be nice to have.
Ok I'll try to add offset/length to the rendering module. It's going to cost us some performance actually since when there are block transformations you need to recompute the blocks offsets... I still need to know how it's best to store them for your needs Malaka. Storing them in the Blocks themselves might cost too much to retrieve them. I could also store them in some other structure with references to the blocks. Let me know what you need.
To speed up things I've done a bit of investigations (Malaka, please integrate with your own findings). I've looked at how these things are done in other complex projects, and maybe I think it will not be possible to use the actual rendering engine's parser as the basis for the Eclipse infrastructure. The Eclipse JDT uses a FastJavaPartitionScanner (http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.jdt.ui/ui/org/eclipse/j... ) that basically implements an ad-hoc scanner for isolating blocks (there is plenty of while(true), etc., i.e. clearly it isn't a standard Java parser) The Groovy editor uses the standard plain RuleBasedPartitionScanner (http://svn.codehaus.org/groovy/trunk/groovy/ide/groovy-eclipse/org.codehaus.... ) suitably customized for its own needs, with a lot of ad-hoc rule- based scanners (e.g., http://svn.codehaus.org/groovy/trunk/groovy/ide/groovy-eclipse/org.codehaus....) . Apparently the parsing of the file with the real parser is done only in the generation of the content outline (and in the error reporting). This is noticeable in the Java IDE: the content outline is updated only when the file is saved (i.e., compiled and so parsed with the actual Java parser). The same is also true in the Groovy code and in the XMLEditor example where the SAXParser is called in the OutlineContentProvider when the document is saved. Why this architecture is necessary, imho, is because the editor's parser should be as loose as possible and resilient to (transient) errors. In fact, when the user is typing the document is almost always in an inconsistent state (which makes the real parser fail). Examining the document while editing with the actual parser also means to parse the document at every keystroke: that's what happens in practice: the partitioner is asked to return next tokens starting from arbitrary positions of the text: a request that is not very well fulfilled by the real parser. From these considerations, I think that probably using the rendering engine's parser is not feasible, and we will need to backtrack and build a "parallel" infrastructure for parsing XWiki documents. Infrastructure that has already been put in place by Venkatesh and that Malaka, at this point, will finish and improve for supporting all the necessary features and variants. Malaka could you comment on this? And in general, WDYT? -Fabio
Sounds like some good points in favor of not reusing the rendering module. A pity since that'll cost us a lot of development time and maintenance but I agree with your findings. OTOH is it possible to build some solution that will be reusable for the wiki editor for example? The wiki editor executes in the browser and thus for autocompletion and syntax highlighting you would need to do ajax request to the server if we wanted to reuse something. Does that work or do we need to reimplement the logic in javascript (i.e. reuse some existing javascript fwk for autocompletion/syntax highlighting)? Thanks -Vincent On Jul 1, 2008, at 9:00 PM, Fabio Mancinelli wrote:
On 1 juil. 08, at 11:12, Vincent Massol wrote:
Just thought about a potential use case for storing the offset: error reporting.
We'll also need it if we want to do auto completion in the wiki editor which I think would be nice to have.
Ok I'll try to add offset/length to the rendering module. It's going to cost us some performance actually since when there are block transformations you need to recompute the blocks offsets... I still need to know how it's best to store them for your needs Malaka. Storing them in the Blocks themselves might cost too much to retrieve them. I could also store them in some other structure with references to the blocks. Let me know what you need.
To speed up things I've done a bit of investigations (Malaka, please integrate with your own findings).
I've looked at how these things are done in other complex projects, and maybe I think it will not be possible to use the actual rendering engine's parser as the basis for the Eclipse infrastructure.
The Eclipse JDT uses a FastJavaPartitionScanner (http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.jdt.ui/ui/org/eclipse/j... ) that basically implements an ad-hoc scanner for isolating blocks (there is plenty of while(true), etc., i.e. clearly it isn't a standard Java parser)
The Groovy editor uses the standard plain RuleBasedPartitionScanner (http://svn.codehaus.org/groovy/trunk/groovy/ide/groovy-eclipse/org.codehaus.... ) suitably customized for its own needs, with a lot of ad-hoc rule- based scanners (e.g., http://svn.codehaus.org/groovy/trunk/groovy/ide/groovy-eclipse/org.codehaus....) .
Apparently the parsing of the file with the real parser is done only in the generation of the content outline (and in the error reporting). This is noticeable in the Java IDE: the content outline is updated only when the file is saved (i.e., compiled and so parsed with the actual Java parser). The same is also true in the Groovy code and in the XMLEditor example where the SAXParser is called in the OutlineContentProvider when the document is saved.
Why this architecture is necessary, imho, is because the editor's parser should be as loose as possible and resilient to (transient) errors. In fact, when the user is typing the document is almost always in an inconsistent state (which makes the real parser fail). Examining the document while editing with the actual parser also means to parse the document at every keystroke: that's what happens in practice: the partitioner is asked to return next tokens starting from arbitrary positions of the text: a request that is not very well fulfilled by the real parser.
From these considerations, I think that probably using the rendering engine's parser is not feasible, and we will need to backtrack and build a "parallel" infrastructure for parsing XWiki documents. Infrastructure that has already been put in place by Venkatesh and that Malaka, at this point, will finish and improve for supporting all the necessary features and variants.
Malaka could you comment on this? And in general, WDYT?
-Fabio _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Fabio,/Vincent/Venkatesh On Wed, Jul 2, 2008 at 12:30 AM, Fabio Mancinelli < [email protected]> wrote:
On 1 juil. 08, at 11:12, Vincent Massol wrote:
Just thought about a potential use case for storing the offset: error
reporting.
We'll also need it if we want to do auto completion in the wiki editor which I think would be nice to have.
Ok I'll try to add offset/length to the rendering module. It's going to cost us some performance actually since when there are block transformations you need to recompute the blocks offsets... I still need to know how it's best to store them for your needs Malaka. Storing them in the Blocks themselves might cost too much to retrieve them. I could also store them in some other structure with references to the blocks. Let me know what you need.
To speed up things I've done a bit of investigations (Malaka, please integrate with your own findings).
I've looked at how these things are done in other complex projects, and maybe I think it will not be possible to use the actual rendering engine's parser as the basis for the Eclipse infrastructure.
The Eclipse JDT uses a FastJavaPartitionScanner ( http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.jdt.ui/ui/org/eclipse/j...) that basically implements an ad-hoc scanner for isolating blocks (there is plenty of while(true), etc., i.e. clearly it isn't a standard Java parser)
The Groovy editor uses the standard plain RuleBasedPartitionScanner ( http://svn.codehaus.org/groovy/trunk/groovy/ide/groovy-eclipse/org.codehaus....) suitably customized for its own needs, with a lot of ad-hoc rule-based scanners (e.g., http://svn.codehaus.org/groovy/trunk/groovy/ide/groovy-eclipse/org.codehaus.... ).
Apparently the parsing of the file with the real parser is done only in the generation of the content outline (and in the error reporting). This is noticeable in the Java IDE: the content outline is updated only when the file is saved (i.e., compiled and so parsed with the actual Java parser). The same is also true in the Groovy code and in the XMLEditor example where the SAXParser is called in the OutlineContentProvider when the document is saved.
Thank you very much for the finding.I think what you are saying is correct.I wrote a wrapper class of FastPartitioner and see what are the methods which are called when typeing.It shows that for each keyperss the following methodes woll be called to repartition the document getContentType documentAboutToBeChanged getPartition documentChanged getPartition computePartitioning If I'm going to use the rendering passer then I need to recompute the XDOM for each keypress and it would be too costly to do.
Why this architecture is necessary, imho, is because the editor's parser should be as loose as possible and resilient to (transient) errors. In fact, when the user is typing the document is almost always in an inconsistent state (which makes the real parser fail). Examining the document while editing with the actual parser also means to parse the document at every keystroke: that's what happens in practice: the partitioner is asked to return next tokens starting from arbitrary positions of the text: a request that is not very well fulfilled by the real parser.
Now I think I needed to write a some kind of xwiki parser which suit the requirments.I dont think available RuleBasedPartitionScanner alone cannot be use to define the partitions because of the recursive nature of the xwiki syntax.
From these considerations, I think that probably using the rendering engine's parser is not feasible, and we will need to backtrack and build a "parallel" infrastructure for parsing XWiki documents. Infrastructure that has already been put in place by Venkatesh and that Malaka, at this point, will finish and improve for supporting all the necessary features and variants.
I will first look at the sample parseres you have given above and try to write a xwiki parser that suite the job.Is that ok.Then for the syntax coloring,we can use the rule base partitioning which is already available in jface because it is done in per partition basis.
Malaka could you comment on this? And in general, WDYT?
-Fabio
WDYT ? . --Malaka Ekanayake CSE UOM
On 2 juil. 08, at 20:12, malaka ekanayake wrote:
Thank you very much for the finding.I think what you are saying is correct.I wrote a wrapper class of FastPartitioner and see what are the methods which are called when typeing.It shows that for each keyperss the following methodes woll be called to repartition the document
getContentType documentAboutToBeChanged getPartition documentChanged getPartition computePartitioning
If I'm going to use the rendering passer then I need to recompute the XDOM for each keypress and it would be too costly to do.
Right.
I will first look at the sample parseres you have given above and try to write a xwiki parser that suite the job.Is that ok.Then for the syntax coloring,we can use the rule base partitioning which is already available in jface because it is done in per partition basis.
WDYT ? .
What do you mean by "write an xwiki parser"? Anyway, please have a look at what Venkatesh has already done and that he has committed in http://svn.xwiki.org/svnroot/sandbox/xeclipse-gsoc/org.xwiki.eclipse.ui/ . You can propose changes and improvements starting from that code without restarting from scratch. -Fabio
Hi Fabio, I think the Xdomexplorer can be a replacement (a better one that too) to the content outline that I had done.. just minor changes required.. I'll keep you posted. Venkatesh On Wed, Jul 2, 2008 at 21:19, Fabio Mancinelli <[email protected]> wrote:
On 2 juil. 08, at 20:12, malaka ekanayake wrote:
Thank you very much for the finding.I think what you are saying is correct.I wrote a wrapper class of FastPartitioner and see what are the methods which are called when typeing.It shows that for each keyperss the following methodes woll be called to repartition the document
getContentType documentAboutToBeChanged getPartition documentChanged getPartition computePartitioning
If I'm going to use the rendering passer then I need to recompute the XDOM for each keypress and it would be too costly to do.
Right.
I will first look at the sample parseres you have given above and try to write a xwiki parser that suite the job.Is that ok.Then for the syntax coloring,we can use the rule base partitioning which is already available in jface because it is done in per partition basis.
WDYT ? .
What do you mean by "write an xwiki parser"?
Anyway, please have a look at what Venkatesh has already done and that he has committed in http://svn.xwiki.org/svnroot/sandbox/xeclipse-gsoc/org.xwiki.eclipse.ui/ . You can propose changes and improvements starting from that code without restarting from scratch.
-Fabio
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Fabio/Venkatesh, I have looked at the code Venkatesh committed and there is a problem in syntax coloring in the integrated code.(in org.xwiki.eclipse.ui) .But the stand alone plugine written by Venkatesh dont have tthis probleam and it works fine.(It color the valide syntax) .Shall I look in to this probleam and try to fix the problem in the org.xwiki.eclipse.ui so the syntax will be colored in that code also. -- Malaka Ekanayake CSE UOM
On 7 juil. 08, at 06:07, malaka ekanayake wrote:
Hi Fabio/Venkatesh,
I have looked at the code Venkatesh committed and there is a problem in syntax coloring in the integrated code.(in org.xwiki.eclipse.ui) .But the stand alone plugine written by Venkatesh dont have tthis probleam and it works fine.(It color the valide syntax) .Shall I look in to this probleam and try to fix the problem in the org.xwiki.eclipse.ui so the syntax will be colored in that code also.
Yes, go ahead and fix it. -Fabio
Hi Fabio/Venkatesh, regarding the above issuer see my comment below On Mon, Jul 7, 2008 at 12:54 PM, Fabio Mancinelli < [email protected]> wrote:
On 7 juil. 08, at 06:07, malaka ekanayake wrote:
Hi Fabio/Venkatesh,
I have looked at the code Venkatesh committed and there is a problem in syntax coloring in the integrated code.(in org.xwiki.eclipse.ui) .But the stand alone plugine written by Venkatesh dont have tthis probleam and it works fine.(It color the valide syntax) .Shall I look in to this probleam and try to fix the problem in the org.xwiki.eclipse.ui so the syntax will be colored in that code also.
Yes,
go ahead and fix it.
-Fabio
I have look at both the standalone org.xwiki.eclipse.plugin and the org.xwiki.eclipse.ui.It seams that in the standalone pluging XwikiDocumentSetupParticipant referred as a extention in plugin.xml.But there is no such extention in the plugin.xml in org.xwiki.eclipse.ui.Since XwikiDocumentSetupParticipant is responsible for conneting the partitions created by the FastPartitioner this may be the course of the problem of syntax coloring. Could you please check on this and tell me weather I'm wright or am I missing something?. To Fabio : I will send my report in detail tonight cheers --Malaka Ekanayake CSE UOM
Hi, It doesn't work for me.. But, now I've realised that I didn't look into the extension part of the plugin while integrating the code, maybe the answer lies there somewhere. Venkatesh malaka ekanayake wrote:
Hi Fabio/Venkatesh,
regarding the above issuer see my comment below
On Mon, Jul 7, 2008 at 12:54 PM, Fabio Mancinelli < [email protected]> wrote:
On 7 juil. 08, at 06:07, malaka ekanayake wrote:
Hi Fabio/Venkatesh,
I have looked at the code Venkatesh committed and there is a problem in syntax coloring in the integrated code.(in org.xwiki.eclipse.ui) .But the stand alone plugine written by Venkatesh dont have tthis probleam and it works fine.(It color the valide syntax) .Shall I look in to this probleam and try to fix the problem in the org.xwiki.eclipse.ui so the syntax will be colored in that code also.
Yes, go ahead and fix it.
-Fabio
I have look at both the standalone org.xwiki.eclipse.plugin and the org.xwiki.eclipse.ui.It seams that in the standalone pluging XwikiDocumentSetupParticipant referred as a extention in plugin.xml.But there is no such extention in the plugin.xml in org.xwiki.eclipse.ui.Since XwikiDocumentSetupParticipant is responsible for conneting the partitions created by the FastPartitioner this may be the course of the problem of syntax coloring.
Could you please check on this and tell me weather I'm wright or am I missing something?.
To Fabio : I will send my report in detail tonight
cheers --Malaka Ekanayake CSE UOM
Hi Malaka,
Hi Malaka,
On Jul 1, 2008, at 6:20 AM, malaka ekanayake wrote:
Hi vincent
Now I'm working on the costume implementation of the IDocument Partitioner which uses the XDOM parser. But in order to to register partitions with the IDocument,requires the offset and the length of each token returned by the parser.
Did you get Marius' input on this?
I don't think I'll need the offset and length for any of the new WYSIWYG editor's features but if we're going to implement syntax highlighting in the wiki editor at some point, then they will be required for the same reason: determine what block is located at a specific location in the source document. So I guess Vincent's rendering component could provide you the offset and length, and you could compute the rest by yourself.
I remember I said:
" * Re the offset and lengthn I think this is something interesting that could be added to the new rendering module. Since the underlying parsers (wikimodel, doxia) don't pass this information to us we would need to compute it. It would allow us to answer the question "what block is located at location NN in the source document". * We need to get Marius' input since he's working on the new WYSIWYG editor for XWiki and maybe he has some similar needs/issues. "
The reason I said this is because if the only requirement is for your use case then you could easily compute the offset/lenght for your use case only (by writing a very simple Listener and running XDOM.traverse(yourllistener). However if the need is generic then it makes sense to have it in the rendering component itself.
I'm still hesitating about the generic need about this.
If we agree to add it in the rendering component then I can implement it for you.
Another question for you: once you have this information what's the algorithm for finding the block that corresponds to a given location in the source document? Dichotomy on the tree?
Thanks -Vincent
But the current implementation of the parser don't have that information.Do I need to look at that issuer as well or what? I have very little understanding of the implementation of the parser and I will be difficult for me to implement that functionality.
Vincent can you help me on this issuer.
-- Malaka Ekanayake CSE UOM
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
participants (5)
-
Fabio Mancinelli -
malaka ekanayake -
Marius Dumitru Florea -
Venkatesh Nandakumar -
Vincent Massol