[xwiki-devs] [discussion] Connector feature for XWiki
Hi Devs, I'd like to start a discussion about a Connector module for XWiki. XWiki is a great tool to create, edit and organize information. But while we can use macros to display external data, many XWiki's organization (such as the livetable), editing and viewing features can only be used for data stored in the XWiki System. However from time to time we've seen that it would be really interesting to: - allow editing/viewing of data from external systems using the XWiki form system - using livetable and search on external data I've been thinking for a while that we can achieve that in XWiki by transforming data read using an external API into XWiki Documents and then use the XWiki APIs to display that data. We can also implement livetable backends that would transform the livetable filters into query supported by the external tool. And if all this could be implemented as a "Connector Interface" most of the presentation code would be generic and we could develop and install connectors and magically you could make use of the XWiki development system to display, present and create forms to edit data. I've made a prototype for Salesforce and JIRA that shows the concept and what the API could be (in this case in Groovy) and wrote a design document that explains the system and the open questions: http://dev.xwiki.org/xwiki/bin/view/Design/ConnectorModule In my view this system is super powerfull if we can build it in the XWiki Core so that the view/edit/save/search can be hooked to the connector so that from the user experience the system is transparent to the user. WDYT ? Could the devs provide some insight of what the right architecture should be for such a feature. Ludovic -- Ludovic Dubost Founder and CEO Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost
I had given this idea some thought when I was listening to how palantir connects to outside sources and allows data to be mapped in, even if it cannot be imported. What came to my mind was some kind of user interface to create a mapping between information in a database with an arbitrary schema and XDocuments and XObjects and then "map in a space" so to speak. Trying to imagine how the user would explain their schema in terms of documents and objects is where I ran out of ideas, it would certainly take a masterpiece of user interface. The search matter is another mess since everything handles search differently. If we're going to map in someone's legacy Oracle database and their JIRA, we can probably make queries work okay. If we're going to think big and map in Excel spreadsheets, directories full of Word documents and sqlite email spools, we have to look at ways to soup up Lucine to handle our XWQL searches in-house. Of course if these things can be imported, we can use what we have but importing brings up issues with synchronization and possibly authorization (where the wiki operator might have permission to access a database but not have permission to mirror it). Thanks, Caleb On 06/12/2012 03:06 PM, Ludovic Dubost wrote:
Hi Devs,
I'd like to start a discussion about a Connector module for XWiki. XWiki is a great tool to create, edit and organize information. But while we can use macros to display external data, many XWiki's organization (such as the livetable), editing and viewing features can only be used for data stored in the XWiki System. However from time to time we've seen that it would be really interesting to:
- allow editing/viewing of data from external systems using the XWiki form system - using livetable and search on external data
I've been thinking for a while that we can achieve that in XWiki by transforming data read using an external API into XWiki Documents and then use the XWiki APIs to display that data. We can also implement livetable backends that would transform the livetable filters into query supported by the external tool. And if all this could be implemented as a "Connector Interface" most of the presentation code would be generic and we could develop and install connectors and magically you could make use of the XWiki development system to display, present and create forms to edit data.
I've made a prototype for Salesforce and JIRA that shows the concept and what the API could be (in this case in Groovy) and wrote a design document that explains the system and the open questions:
http://dev.xwiki.org/xwiki/bin/view/Design/ConnectorModule
In my view this system is super powerfull if we can build it in the XWiki Core so that the view/edit/save/search can be hooked to the connector so that from the user experience the system is transparent to the user.
WDYT ? Could the devs provide some insight of what the right architecture should be for such a feature.
Ludovic
Hi Ludovic, On Jun 12, 2012, at 9:06 PM, Ludovic Dubost wrote:
Hi Devs,
I'd like to start a discussion about a Connector module for XWiki. XWiki is a great tool to create, edit and organize information. But while we can use macros to display external data, many XWiki's organization (such as the livetable), editing and viewing features can only be used for data stored in the XWiki System. However from time to time we've seen that it would be really interesting to:
- allow editing/viewing of data from external systems using the XWiki form system - using livetable and search on external data
I've been thinking for a while that we can achieve that in XWiki by transforming data read using an external API into XWiki Documents and then use the XWiki APIs to display that data. We can also implement livetable backends that would transform the livetable filters into query supported by the external tool. And if all this could be implemented as a "Connector Interface" most of the presentation code would be generic and we could develop and install connectors and magically you could make use of the XWiki development system to display, present and create forms to edit data.
I've made a prototype for Salesforce and JIRA that shows the concept and what the API could be (in this case in Groovy) and wrote a design document that explains the system and the open questions:
http://dev.xwiki.org/xwiki/bin/view/Design/ConnectorModule
In my view this system is super powerfull if we can build it in the XWiki Core so that the view/edit/save/search can be hooked to the connector so that from the user experience the system is transparent to the user.
WDYT ? Could the devs provide some insight of what the right architecture should be for such a feature.
The idea is good. The best architecture solution for this is to implement one Store per external system. The features you describe are those of a store: load, save, delete. even create. Once you have a store implementation for an external system then all the rest of XWiki works without change. Regarding security XWiki already supports permissions so all that's needed is for the generated XWikiDocument to have the proper XWikiRights object set on them. The only main issue I can see is that we have a Cache Store in front of the real store for performance reason (and we'll need that for external systems too since accessing remote data is very very costly) but that stores assumes that the data is not modified from the outside. If data is modified from outside of XWiki then the cache will be stale. We can handle this in a various of manners. The best is probably to have some strategy to refresh our cache regularly for documents coming from outside systems and returning an error in save() if the data has been modified externally and within xwiki. Anyway this part is the hard one and we would need to brainstorm to find the best solution. What is sure is that we need a cache. Thanks -Vincent
Ludovic
The idea of using a store is nice. Indeed you get a cache. But there are some though calls: - XWQL mapping to the store's query language which is quite hard - Events to be informed that information is changed in the external store to allow Lucene to reindex the content - The complexity of cross engine querying or the lack of it which would make some features hard to implement (like the page finder or looking for pages in the wysiwyg editor) What is interesting is that you could not only map the external content to XWiki but also use the XWiki features on the same document by still having a local XWiki document (like comments and annotations). Ludovic 2012/6/13 Vincent Massol <[email protected]>:
Hi Ludovic,
On Jun 12, 2012, at 9:06 PM, Ludovic Dubost wrote:
Hi Devs,
I'd like to start a discussion about a Connector module for XWiki. XWiki is a great tool to create, edit and organize information. But while we can use macros to display external data, many XWiki's organization (such as the livetable), editing and viewing features can only be used for data stored in the XWiki System. However from time to time we've seen that it would be really interesting to:
- allow editing/viewing of data from external systems using the XWiki form system - using livetable and search on external data
I've been thinking for a while that we can achieve that in XWiki by transforming data read using an external API into XWiki Documents and then use the XWiki APIs to display that data. We can also implement livetable backends that would transform the livetable filters into query supported by the external tool. And if all this could be implemented as a "Connector Interface" most of the presentation code would be generic and we could develop and install connectors and magically you could make use of the XWiki development system to display, present and create forms to edit data.
I've made a prototype for Salesforce and JIRA that shows the concept and what the API could be (in this case in Groovy) and wrote a design document that explains the system and the open questions:
http://dev.xwiki.org/xwiki/bin/view/Design/ConnectorModule
In my view this system is super powerfull if we can build it in the XWiki Core so that the view/edit/save/search can be hooked to the connector so that from the user experience the system is transparent to the user.
WDYT ? Could the devs provide some insight of what the right architecture should be for such a feature.
The idea is good.
The best architecture solution for this is to implement one Store per external system. The features you describe are those of a store: load, save, delete. even create.
Once you have a store implementation for an external system then all the rest of XWiki works without change.
Regarding security XWiki already supports permissions so all that's needed is for the generated XWikiDocument to have the proper XWikiRights object set on them.
The only main issue I can see is that we have a Cache Store in front of the real store for performance reason (and we'll need that for external systems too since accessing remote data is very very costly) but that stores assumes that the data is not modified from the outside. If data is modified from outside of XWiki then the cache will be stale. We can handle this in a various of manners. The best is probably to have some strategy to refresh our cache regularly for documents coming from outside systems and returning an error in save() if the data has been modified externally and within xwiki. Anyway this part is the hard one and we would need to brainstorm to find the best solution. What is sure is that we need a cache.
Thanks -Vincent
Ludovic
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Ludovic Dubost Founder and CEO Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost
participants (3)
-
Caleb James DeLisle -
Ludovic Dubost -
Vincent Massol