12 Jun
2012
12 Jun
'12
9:06 p.m.
Hi Devs,
I'd like to start a discussion about a Connector module for XWiki.
XWiki is a great tool to create, edit and organize information. But
while we can use macros to display external data, many XWiki's
organization (such as the livetable), editing and viewing features can
only be used for data stored in the XWiki System.
However from time to time we've seen that it would be really interesting to:
- allow editing/viewing of data from external systems using the XWiki
form system
- using livetable and search on external data
I've been thinking for a while that we can achieve that in XWiki by
transforming data read using an external API into XWiki Documents and
then use the XWiki APIs to display that data. We can also implement
livetable backends that would transform the livetable filters into
query supported by the external tool. And if all this could be
implemented as a "Connector Interface" most of the presentation code
would be generic and we could develop and install connectors and
magically you could make use of the XWiki development system to
display, present and create forms to edit data.
I've made a prototype for Salesforce and JIRA that shows the concept
and what the API could be (in this case in Groovy) and wrote a design
document that explains the system and the open questions:
http://dev.xwiki.org/xwiki/bin/view/Design/ConnectorModule
In my view this system is super powerfull if we can build it in the
XWiki Core so that the view/edit/save/search can be hooked to the
connector so that from the user experience the system is transparent
to the user.
WDYT ? Could the devs provide some insight of what the right
architecture should be for such a feature.
Ludovic
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
13 Jun
13 Jun
7:34 a.m.
I had given this idea some thought when I was listening to how palantir
connects to outside sources and allows data to be mapped in, even if it
cannot be imported.
What came to my mind was some kind of user interface to create a mapping
between information in a database with an arbitrary schema and XDocuments
and XObjects and then "map in a space" so to speak.
Trying to imagine how the user would explain their schema in terms of
documents and objects is where I ran out of ideas, it would certainly
take a masterpiece of user interface.
The search matter is another mess since everything handles search
differently. If we're going to map in someone's legacy Oracle database
and their JIRA, we can probably make queries work okay. If we're going
to think big and map in Excel spreadsheets, directories full of Word
documents and sqlite email spools, we have to look at ways to soup up
Lucine to handle our XWQL searches in-house. Of course if these things
can be imported, we can use what we have but importing brings up issues
with synchronization and possibly authorization (where the wiki operator
might have permission to access a database but not have permission to
mirror it).
Thanks,
Caleb
On 06/12/2012 03:06 PM, Ludovic Dubost wrote:
Hi Devs,
I'd like to start a discussion about a Connector module for XWiki.
XWiki is a great tool to create, edit and organize information. But
while we can use macros to display external data, many XWiki's
organization (such as the livetable), editing and viewing features can
only be used for data stored in the XWiki System.
However from time to time we've seen that it would be really interesting to:
- allow editing/viewing of data from external systems using the XWiki
form system
- using livetable and search on external data
I've been thinking for a while that we can achieve that in XWiki by
transforming data read using an external API into XWiki Documents and
then use the XWiki APIs to display that data. We can also implement
livetable backends that would transform the livetable filters into
query supported by the external tool. And if all this could be
implemented as a "Connector Interface" most of the presentation code
would be generic and we could develop and install connectors and
magically you could make use of the XWiki development system to
display, present and create forms to edit data.
I've made a prototype for Salesforce and JIRA that shows the concept
and what the API could be (in this case in Groovy) and wrote a design
document that explains the system and the open questions:
http://dev.xwiki.org/xwiki/bin/view/Design/ConnectorModule
In my view this system is super powerfull if we can build it in the
XWiki Core so that the view/edit/save/search can be hooked to the
connector so that from the user experience the system is transparent
to the user.
WDYT ? Could the devs provide some insight of what the right
architecture should be for such a feature.
Ludovic
8:29 a.m.
Hi Ludovic,
On Jun 12, 2012, at 9:06 PM, Ludovic Dubost wrote:
Hi Devs,
I'd like to start a discussion about a Connector module for XWiki.
XWiki is a great tool to create, edit and organize information. But
while we can use macros to display external data, many XWiki's
organization (such as the livetable), editing and viewing features can
only be used for data stored in the XWiki System.
However from time to time we've seen that it would be really interesting to:
- allow editing/viewing of data from external systems using the XWiki
form system
- using livetable and search on external data
I've been thinking for a while that we can achieve that in XWiki by
transforming data read using an external API into XWiki Documents and
then use the XWiki APIs to display that data. We can also implement
livetable backends that would transform the livetable filters into
query supported by the external tool. And if all this could be
implemented as a "Connector Interface" most of the presentation code
would be generic and we could develop and install connectors and
magically you could make use of the XWiki development system to
display, present and create forms to edit data.
I've made a prototype for Salesforce and JIRA that shows the concept
and what the API could be (in this case in Groovy) and wrote a design
document that explains the system and the open questions:
http://dev.xwiki.org/xwiki/bin/view/Design/ConnectorModule
In my view this system is super powerfull if we can build it in the
XWiki Core so that the view/edit/save/search can be hooked to the
connector so that from the user experience the system is transparent
to the user.
WDYT ? Could the devs provide some insight of what the right
architecture should be for such a feature.
The idea is good.
The best architecture solution for this is to implement one Store per external system. The
features you describe are those of a store: load, save, delete. even create.
Once you have a store implementation for an external system then all the rest of XWiki
works without change.
Regarding security XWiki already supports permissions so all that's needed is for the
generated XWikiDocument to have the proper XWikiRights object set on them.
The only main issue I can see is that we have a Cache Store in front of the real store for
performance reason (and we'll need that for external systems too since accessing
remote data is very very costly) but that stores assumes that the data is not modified
from the outside. If data is modified from outside of XWiki then the cache will be stale.
We can handle this in a various of manners. The best is probably to have some strategy to
refresh our cache regularly for documents coming from outside systems and returning an
error in save() if the data has been modified externally and within xwiki. Anyway this
part is the hard one and we would need to brainstorm to find the best solution. What is
sure is that we need a cache.
Thanks
-Vincent
Ludovic
10:59 a.m.
The idea of using a store is nice. Indeed you get a cache.
But there are some though calls:
- XWQL mapping to the store's query language which is quite hard
- Events to be informed that information is changed in the external
store to allow Lucene to reindex the content
- The complexity of cross engine querying or the lack of it which
would make some features hard to implement (like the page finder or
looking for pages in the wysiwyg editor)
What is interesting is that you could not only map the external
content to XWiki but also use the XWiki features on the same document
by still having a local XWiki document (like comments and
annotations).
Ludovic
2012/6/13 Vincent Massol <vincent(a)massol.net>et>:
Hi Ludovic,
On Jun 12, 2012, at 9:06 PM, Ludovic Dubost wrote:
--
Ludovic Dubost
Founder and CEO
Blog: http://blog.ludovic.org/
XWiki: http://www.xwiki.com
Skype: ldubost GTalk: ldubost
Hi Devs,
I'd like to start a discussion about a Connector module for XWiki.
XWiki is a great tool to create, edit and organize information. But
while we can use macros to display external data, many XWiki's
organization (such as the livetable), editing and viewing features can
only be used for data stored in the XWiki System.
However from time to time we've seen that it would be really interesting to:
- allow editing/viewing of data from external systems using the XWiki
form system
- using livetable and search on external data
I've been thinking for a while that we can achieve that in XWiki by
transforming data read using an external API into XWiki Documents and
then use the XWiki APIs to display that data. We can also implement
livetable backends that would transform the livetable filters into
query supported by the external tool. And if all this could be
implemented as a "Connector Interface" most of the presentation code
would be generic and we could develop and install connectors and
magically you could make use of the XWiki development system to
display, present and create forms to edit data.
I've made a prototype for Salesforce and JIRA that shows the concept
and what the API could be (in this case in Groovy) and wrote a design
document that explains the system and the open questions:
http://dev.xwiki.org/xwiki/bin/view/Design/ConnectorModule
In my view this system is super powerfull if we can build it in the
XWiki Core so that the view/edit/save/search can be hooked to the
connector so that from the user experience the system is transparent
to the user.
WDYT ? Could the devs provide some insight of what the right
architecture should be for such a feature.
The idea is good.
The best architecture solution for this is to implement one Store per external system.
The features you describe are those of a store: load, save, delete. even create.
Once you have a store implementation for an external system then all the rest of XWiki
works without change.
Regarding security XWiki already supports permissions so all that's needed is for the
generated XWikiDocument to have the proper XWikiRights object set on them.
The only main issue I can see is that we have a Cache Store in front of the real store
for performance reason (and we'll need that for external systems too since accessing
remote data is very very costly) but that stores assumes that the data is not modified
from the outside. If data is modified from outside of XWiki then the cache will be stale.
We can handle this in a various of manners. The best is probably to have some strategy to
refresh our cache regularly for documents coming from outside systems and returning an
error in save() if the data has been modified externally and within xwiki. Anyway this
part is the hard one and we would need to brainstorm to find the best solution. What is
sure is that we need a cache.
Thanks
-Vincent
Ludovic
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
4575
days inactive
4576
days old
3 comments
3 participants
participants (3)
-
Caleb James DeLisle -
Ludovic Dubost -
Vincent Massol