On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea <[email protected]> wrote:

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special characters which breaks the URL in Tomcat for security reasons.

Badly configured Tomcat does not like slash but are you sure about backslash ?

...

This is based on the existing TemporaryResourceReference at:

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

For example:

http:// <server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

Note that in this example from the officeviewer macro the module-dependent resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

- generated image name from PPT

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

WDYT?

Thanks -Vincent

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- Thomas Mortagne

On 14 Apr 2016, at 18:46, Vincent Massol <[email protected]> wrote:

...

On 14 Apr 2016, at 17:55, Thomas Mortagne <[email protected]> wrote:

On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea <[email protected]> wrote:

...

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special characters which breaks the URL in Tomcat for security reasons.

Badly configured Tomcat does not like slash but are you sure about backslash ?

Yes, it’s both.

FTR http://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html#Security -Vincent

Thanks -Vincent

...

...

...

This is based on the existing TemporaryResourceReference at:

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

For example:

http:// <server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

Note that in this example from the officeviewer macro the module-dependent resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

- generated image name from PPT

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

WDYT?

Thanks -Vincent

IMO since Tomcat can be properly configured to behave as any decent application server should and just do what we tell it to do I don't think we care. On Thu, Apr 14, 2016 at 6:54 PM, Vincent Massol <[email protected]> wrote:

...

On 14 Apr 2016, at 18:46, Vincent Massol <[email protected]> wrote:

...

On 14 Apr 2016, at 17:55, Thomas Mortagne <[email protected]> wrote:

On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea <[email protected]> wrote:

...

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special characters which breaks the URL in Tomcat for security reasons.

Badly configured Tomcat does not like slash but are you sure about backslash ?

Yes, it’s both.

FTR http://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html#Security

-Vincent

...

Thanks -Vincent

...

...

...

This is based on the existing TemporaryResourceReference at:

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

For example:

http:// <server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

Note that in this example from the officeviewer macro the module-dependent resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

- generated image name from PPT

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

WDYT?

Thanks -Vincent

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- Thomas Mortagne

@Thomas: are you ok with the proposed format: http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path> ? Thanks -Vincent

On 14 Apr 2016, at 17:55, Thomas Mortagne <[email protected]> wrote:

On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea <[email protected]> wrote:

...

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special characters which breaks the URL in Tomcat for security reasons.

Badly configured Tomcat does not like slash but are you sure about backslash ?

...

...

This is based on the existing TemporaryResourceReference at:

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

For example:

http:// <server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

Note that in this example from the officeviewer macro the module-dependent resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

- generated image name from PPT

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

WDYT?

Thanks -Vincent

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

Hi, My impression was that it was too complicated, from a caller's POV, to manage the module, the document and then a module-based hierarchy when all you want is a temporary resource URL (for which you should not care about all of these things) and which you could easily resolve with some script service. The main thing is that it must resolve in the browser, for a while at least. Uniqueness would also be easily guaranteed IMO, since it would be a single (Java) service that would be in charge of this and that could take care of synchronization or DB access issues. Also, you`ve mentioned creating a file for the temporary resource. I was under the impression that this API would only take care of assigning and making accessible a temporary resource (URL) and it would be the caller's job to actually retrieve its content (since the resource could be stored in a file, in a wiki page, on some separate service maybe, etc.). Perhaps temporary files could be a default provided helper implementation (since it`s pretty common). Anyway, that was just my (non-binding) view on this which, for me, just feels simpler to use. Since you have the actual problem to fix, you probably know better the actual needs from this API than I got to understand from my quick reading. I personally hope we don`t get to proliferate this yet another document reference (escaping) syntax since we already have too many (dots vs slashes, w/o WebHome, etc. and now syntax vs URL) and our users might be already confused enough. Thanks, Eduard On Mon, Apr 18, 2016 at 7:24 PM, Vincent Massol <[email protected]> wrote:

ok I’m going to implement this format, using the URL serializer/resolver that I’ve committed.

Note that the URL format shouldn’t matter that much since it’s an internal format that we can change later on if we want.

Let me know quickly if you don’t agree.

Thanks -Vincent

...

On 15 Apr 2016, at 09:26, Vincent Massol <[email protected]> wrote:

@Thomas: are you ok with the proposed format:

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

?

Thanks -Vincent

...

On 14 Apr 2016, at 17:55, Thomas Mortagne <[email protected]> wrote:

On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea <[email protected]> wrote:

...

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special characters which breaks the URL in Tomcat for security reasons.

Badly configured Tomcat does not like slash but are you sure about backslash ?

...

...

This is based on the existing TemporaryResourceReference at:

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

...

...

For example:

http://

<server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

...

Note that in this example from the officeviewer macro the

module-dependent

...

resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

- generated image name from PPT

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

...

WDYT?

Thanks -Vincent

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

Like Edy, I'm not a big fan of the forced document based entry point since it might not makes any sense for some use cases. Now one job of the tmp resource is also to check access right so we need to pass it an entity reference on which to test the right when a right check is required. The alternative being to end up with the reference both in the path (to avoir collisions) and as some URL parameter which is not nice I guess what you propose it ok as long as empty reference is supported (i.e. don't test the right and just go return the file associated to the path) as in http://mydomain/xwiki/tmp/mymodule//I/don't/care/about/right.png Making the tmp resource generic enough to be just an entry point for calling some module which then do whatever it wants would just be a duplicate of resource handler framework but maybe we just don't really need this anymore central temp resource entry point since now that we have a generic resource handler framework ? On Fri, Apr 15, 2016 at 9:26 AM, Vincent Massol <[email protected]> wrote:

@Thomas: are you ok with the proposed format:

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

?

Thanks -Vincent

...

On 14 Apr 2016, at 17:55, Thomas Mortagne <[email protected]> wrote:

On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea <[email protected]> wrote:

...

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special characters which breaks the URL in Tomcat for security reasons.

Badly configured Tomcat does not like slash but are you sure about backslash ?

...

...

This is based on the existing TemporaryResourceReference at:

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

For example:

http:// <server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

Note that in this example from the officeviewer macro the module-dependent resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

- generated image name from PPT

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

WDYT?

Thanks -Vincent

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- Thomas Mortagne _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

-- Thomas Mortagne

On 19 Apr 2016, at 10:25, Vincent Massol <[email protected]> wrote:

...

On 19 Apr 2016, at 10:07, Thomas Mortagne <[email protected]> wrote:

Like Edy, I'm not a big fan of the forced document based entry point since it might not makes any sense for some use cases.

As I mentioned in my previous mail we don’t need to consider it a reference to a document. We could just make it a reference to anything. The module using the tmp service would know what type of reference it is and it could cast it to a document reference if it knows it’s that.

Actually this wouldn’t work since it would have one effect: we wouldn’t be able to check rights in the Tmp Handler which is a problem… We could decide to use a serialized Resource Reference and develop a Rights Checker accepting any Resource Reference (we would only implement check for Entity Resource Reference to start with). This means also inventing a serialization format for Resource References (something with a prefix representing the type as with Link Resource References). Since this is a bit complex we could start with an Entity Reference for now and implement this later on. Thanks -Vincent

BTW this means that my URL entity reference serializer/resolver are no useful for this ;) (they’d still be useful for the “reference” url scheme though). We could simply take a String an replace the “/“ and “\” with some other symbols and do the same when parsing the URL..

So the generic format would be:

http://<server>/<context>/tmp/<module id>/<serialized reference/id representing the resource>/<module-dependent resource path>

The <serialized reference/id representing the resource> wouldn’t be able to be empty though since <module-dependent resource path> is non-fixed length (e.g. “a/b/c”).

WDYT?

Thanks -Vincent

...

Now one job of the tmp resource is also to check access right so we need to pass it an entity reference on which to test the right when a right check is required. The alternative being to end up with the reference both in the path (to avoir collisions) and as some URL parameter which is not nice I guess what you propose it ok as long as empty reference is supported (i.e. don't test the right and just go return the file associated to the path) as in http://mydomain/xwiki/tmp/mymodule//I/don't/care/about/right.png

Making the tmp resource generic enough to be just an entry point for calling some module which then do whatever it wants would just be a duplicate of resource handler framework but maybe we just don't really need this anymore central temp resource entry point since now that we have a generic resource handler framework ?

On Fri, Apr 15, 2016 at 9:26 AM, Vincent Massol <[email protected]> wrote:

...

@Thomas: are you ok with the proposed format:

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

?

Thanks -Vincent

...

On 14 Apr 2016, at 17:55, Thomas Mortagne <[email protected]> wrote:

On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea <[email protected]> wrote:

...

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special characters which breaks the URL in Tomcat for security reasons.

Badly configured Tomcat does not like slash but are you sure about backslash ?

...

...

This is based on the existing TemporaryResourceReference at:

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

For example:

http:// <server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

Note that in this example from the officeviewer macro the module-dependent resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

- generated image name from PPT

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

WDYT?

Thanks -Vincent

On 19 Apr 2016, at 12:47, Thomas Mortagne <[email protected]> wrote:

On Tue, Apr 19, 2016 at 10:28 AM, Vincent Massol <[email protected]> wrote:

...

...

On 19 Apr 2016, at 10:25, Vincent Massol <[email protected]> wrote:

...

On 19 Apr 2016, at 10:07, Thomas Mortagne <[email protected]> wrote:

Like Edy, I'm not a big fan of the forced document based entry point since it might not makes any sense for some use cases.

As I mentioned in my previous mail we don’t need to consider it a reference to a document. We could just make it a reference to anything. The module using the tmp service would know what type of reference it is and it could cast it to a document reference if it knows it’s that.

Actually this wouldn’t work since it would have one effect: we wouldn’t be able to check rights in the Tmp Handler which is a problem…

Yes, which is why I said in my previous mail that we can't really get rid of it.

...

We could decide to use a serialized Resource Reference and develop a Rights Checker accepting any Resource Reference (we would only implement check for Entity Resource Reference to start with). This means also inventing a serialization format for Resource References (something with a prefix representing the type as with Link Resource References).

Since this is a bit complex we could start with an Entity Reference for now and implement this later on.

Extending what I started to express in my previous mail: IMO we should stop trying to make /tmp/ the ultimate generic temporary resource provider, we can live with something dedicated to providing a filesystem file associated to an entity and test access based on this entity. Any module that have another use case can easily bypass /tmp/ and provide its own resource reference handler. Making /tmp/ super generic just ends up reinventig a new (and more complex from what I can see in the proposals) resource reference handler framework. I'm even removing my comment about empty entity.

That means:

http://<server>/<context>/tmp/<entitytype>:<entity reference>/<module-dependent resource path>

It's easy to support any resource reference instead of the entity reference later just by changing the meaning of <entitytype> into <resourcetype> so we don't really need to deal with it now.

Thanks for the reply Thomas. How do we parse "<entitytype>:<entity reference>”? AFAIK we don’t have a parser for this. I’d prefer to have <serialized document reference> FTM since we have a parser for that and change that later on if need be. My goal was to implement quickly the tmp handler since I judged it was going to take about the same time as fixing the existing TmpAction to support Nested Spaces. This thread is starting to prove I was wrong and I’m very tempted to drop the topic and to go back to just fixing TempAction. That would be a pity IMO. (I’ve already spent a lot more time than I had planned with the SymbolScheme and URL serializer/resolver but at least they could serve for other needs).

The path is already module dependent so no need to add a module related element in the URL scheme IMO. We can indicate that a good practice for custom modules resources is to start the path with some module unique identifier but it does not really need to have any special meaning in the URL itself.

I don’t agree. It’s the same as typed api vs untyped API. If you don’t put the module id in the API then it’s up to best practices and we do know that best practices are hard to enforce. So I’m strongly in favor of having it in the API (i.e. in the URL).

Exactly like module that manipulate permanent and temporary directories don't get a dedicated folder, they just decide to put their stuff in some mymodule/ subforlder (or not).

That’s not right IMO. The temporary API should return an isolated location for a given module so that it has no risk of clashing with another module. It shouldn’t be the goal of a given module to ensure this. It should be a service of the temporary API. We could have 2 APIs but the main one that modules should use should be the one returning an isolated and safe location. Thanks -Vincent

Thanks

...

-Vincent

...

BTW this means that my URL entity reference serializer/resolver are no useful for this ;) (they’d still be useful for the “reference” url scheme though). We could simply take a String an replace the “/“ and “\” with some other symbols and do the same when parsing the URL..

So the generic format would be:

http://<server>/<context>/tmp/<module id>/<serialized reference/id representing the resource>/<module-dependent resource path>

The <serialized reference/id representing the resource> wouldn’t be able to be empty though since <module-dependent resource path> is non-fixed length (e.g. “a/b/c”).

WDYT?

Thanks -Vincent

...

Now one job of the tmp resource is also to check access right so we need to pass it an entity reference on which to test the right when a right check is required. The alternative being to end up with the reference both in the path (to avoir collisions) and as some URL parameter which is not nice I guess what you propose it ok as long as empty reference is supported (i.e. don't test the right and just go return the file associated to the path) as in http://mydomain/xwiki/tmp/mymodule//I/don't/care/about/right.png

Making the tmp resource generic enough to be just an entry point for calling some module which then do whatever it wants would just be a duplicate of resource handler framework but maybe we just don't really need this anymore central temp resource entry point since now that we have a generic resource handler framework ?

On Fri, Apr 15, 2016 at 9:26 AM, Vincent Massol <[email protected]> wrote:

...

@Thomas: are you ok with the proposed format:

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

?

Thanks -Vincent

...

On 14 Apr 2016, at 17:55, Thomas Mortagne <[email protected]> wrote:

On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea <[email protected]> wrote: > On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote: > >> Hi devs, >> >> I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the >> temporary resource concept inside the Resource module”) and I need to >> define a URL format for the new “tmp” resource type. >> >> I’m proposing the following: >> >> > >> http://<server>/<context>/tmp/<module id>/<serialized owner document >> reference>/<module-dependent resource path> >> > > Serialized document reference uses backslash to escape special characters > which breaks the URL in Tomcat for security reasons.

Badly configured Tomcat does not like slash but are you sure about backslash ?

> > >> >> This is based on the existing TemporaryResourceReference at: >> >> https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e... >> >> For example: >> >> http:// >> <server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg >> >> Note that in this example from the officeviewer macro the module-dependent >> resource path consists in: >> > > >> - base64(name of office attachment + hashcode(parameters)) >> > > See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I > was trying to avoid backslash (from the serialized attachment reference) in > the URL. > > >> - generated image name from PPT >> >> In this case, the implementation would generate the following file: >> >> >> [TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg >> >> WDYT? >> >> Thanks >> -Vincent

On 19 Apr 2016, at 14:23, Thomas Mortagne <[email protected]> wrote:

On Tue, Apr 19, 2016 at 1:14 PM, Vincent Massol <[email protected]> wrote:

...

...

On 19 Apr 2016, at 12:47, Thomas Mortagne <[email protected]> wrote:

On Tue, Apr 19, 2016 at 10:28 AM, Vincent Massol <[email protected]> wrote:

...

...

On 19 Apr 2016, at 10:25, Vincent Massol <[email protected]> wrote:

...

On 19 Apr 2016, at 10:07, Thomas Mortagne <[email protected]> wrote:

Like Edy, I'm not a big fan of the forced document based entry point since it might not makes any sense for some use cases.

As I mentioned in my previous mail we don’t need to consider it a reference to a document. We could just make it a reference to anything. The module using the tmp service would know what type of reference it is and it could cast it to a document reference if it knows it’s that.

Actually this wouldn’t work since it would have one effect: we wouldn’t be able to check rights in the Tmp Handler which is a problem…

Yes, which is why I said in my previous mail that we can't really get rid of it.

...

We could decide to use a serialized Resource Reference and develop a Rights Checker accepting any Resource Reference (we would only implement check for Entity Resource Reference to start with). This means also inventing a serialization format for Resource References (something with a prefix representing the type as with Link Resource References).

Since this is a bit complex we could start with an Entity Reference for now and implement this later on.

Extending what I started to express in my previous mail: IMO we should stop trying to make /tmp/ the ultimate generic temporary resource provider, we can live with something dedicated to providing a filesystem file associated to an entity and test access based on this entity. Any module that have another use case can easily bypass /tmp/ and provide its own resource reference handler. Making /tmp/ super generic just ends up reinventig a new (and more complex from what I can see in the proposals) resource reference handler framework. I'm even removing my comment about empty entity.

That means:

http://<server>/<context>/tmp/<entitytype>:<entity reference>/<module-dependent resource path>

It's easy to support any resource reference instead of the entity reference later just by changing the meaning of <entitytype> into <resourcetype> so we don't really need to deal with it now.

Thanks for the reply Thomas.

How do we parse "<entitytype>:<entity reference>”? AFAIK we don’t have a parser for this.

I’d prefer to have <serialized document reference> FTM since we have a parser for that and change that later on if need be. My goal was to implement quickly the tmp handler since I judged it was going to take about the same time as fixing the existing TmpAction to support Nested Spaces. This thread is starting to prove I was wrong and I’m very tempted to drop the topic and to go back to just fixing TempAction. That would be a pity IMO. (I’ve already spent a lot more time than I had planned with the SymbolScheme and URL serializer/resolver but at least they could serve for other needs).

Actually we do have a parser for this since this syntax already exist for String to EntityReference automatic conversion in Velocity scripts. See https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwik.... You could move this in a EnityReferenceResolver and use the resolver in the converter if you prefer.

Didn’t know about this one :) Now we still need to write a resolver (and serializer) and do it right. It might be better to implement it as a ResourceRefefenceResolver/Serializer if we want to be more generic though. I’ve also been wanting to refactor the link resource reference parser as a Resource module resolver.

You can't really decide later to add a prefix without breaking the URL scheme. When you already have a prefix you can add other prefixes without breaking anything.

Yes and I don’t think we care that much about breaking the URL scheme since it’s internal. Breaking the API to generate the URL is a bit more problematic but still acceptable if we keep it internal for now. Anyway let’s try to do it right. I’m just unsure that I’ll have the time to work on this any time soon but at least we now have this thread to know what we want when someone will next have the time to work on this.

The right API being already EntityReference based it's just about converting the String to an EntityReference instead of a DocumentReference.

...

...

The path is already module dependent so no need to add a module related element in the URL scheme IMO. We can indicate that a good practice for custom modules resources is to start the path with some module unique identifier but it does not really need to have any special meaning in the URL itself.

I don’t agree. It’s the same as typed api vs untyped API. If you don’t put the module id in the API then it’s up to best practices and we do know that best practices are hard to enforce. So I’m strongly in favor of having it in the API (i.e. in the URL).

As long as the choice is on module side it's never really fully safe but sure it's a bit safer. My point was more that for some core stuff we may not need any module id, producing simpler URL, but sure we can always find some module id that make sense.

...

...

Exactly like module that manipulate permanent and temporary directories don't get a dedicated folder, they just decide to put their stuff in some mymodule/ subforlder (or not).

That’s not right IMO. The temporary API should return an isolated location for a given module so that it has no risk of clashing with another module. It shouldn’t be the goal of a given module to ensure this. It should be a service of the temporary API. We could have 2 APIs but the main one that modules should use should be the one returning an isolated and safe location.

According to this logic it means we need to refactor the Environment API accordingly. Same thing for configurations and translations in which property names are up to each module which usually put itself as property name prefix but could forget, etc.

Yes potentially. As we can see in the translations, it’s hard to have a single naming convention that is respected. Thanks -Vincent

Thanks

...

-Vincent

...

Thanks

...

-Vincent

...

BTW this means that my URL entity reference serializer/resolver are no useful for this ;) (they’d still be useful for the “reference” url scheme though). We could simply take a String an replace the “/“ and “\” with some other symbols and do the same when parsing the URL..

So the generic format would be:

http://<server>/<context>/tmp/<module id>/<serialized reference/id representing the resource>/<module-dependent resource path>

The <serialized reference/id representing the resource> wouldn’t be able to be empty though since <module-dependent resource path> is non-fixed length (e.g. “a/b/c”).

WDYT?

Thanks -Vincent

...

Now one job of the tmp resource is also to check access right so we need to pass it an entity reference on which to test the right when a right check is required. The alternative being to end up with the reference both in the path (to avoir collisions) and as some URL parameter which is not nice I guess what you propose it ok as long as empty reference is supported (i.e. don't test the right and just go return the file associated to the path) as in http://mydomain/xwiki/tmp/mymodule//I/don't/care/about/right.png

Making the tmp resource generic enough to be just an entry point for calling some module which then do whatever it wants would just be a duplicate of resource handler framework but maybe we just don't really need this anymore central temp resource entry point since now that we have a generic resource handler framework ?

On Fri, Apr 15, 2016 at 9:26 AM, Vincent Massol <[email protected]> wrote: > @Thomas: are you ok with the proposed format: > > http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path> > > ? > > Thanks > -Vincent > >> On 14 Apr 2016, at 17:55, Thomas Mortagne <[email protected]> wrote: >> >> On Thu, Apr 14, 2016 at 4:52 PM, Marius Dumitru Florea >> <[email protected]> wrote: >>> On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote: >>> >>>> Hi devs, >>>> >>>> I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the >>>> temporary resource concept inside the Resource module”) and I need to >>>> define a URL format for the new “tmp” resource type. >>>> >>>> I’m proposing the following: >>>> >>>> >>> >>>> http://<server>/<context>/tmp/<module id>/<serialized owner document >>>> reference>/<module-dependent resource path> >>>> >>> >>> Serialized document reference uses backslash to escape special characters >>> which breaks the URL in Tomcat for security reasons. >> >> Badly configured Tomcat does not like slash but are you sure about backslash ? >> >>> >>> >>>> >>>> This is based on the existing TemporaryResourceReference at: >>>> >>>> https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e... >>>> >>>> For example: >>>> >>>> http:// >>>> <server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg >>>> >>>> Note that in this example from the officeviewer macro the module-dependent >>>> resource path consists in: >>>> >>> >>> >>>> - base64(name of office attachment + hashcode(parameters)) >>>> >>> >>> See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I >>> was trying to avoid backslash (from the serialized attachment reference) in >>> the URL. >>> >>> >>>> - generated image name from PPT >>>> >>>> In this case, the implementation would generate the following file: >>>> >>>> >>>> [TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg >>>> >>>> WDYT? >>>> >>>> Thanks >>>> -Vincent

On Thu, Apr 14, 2016 at 7:46 PM, Vincent Massol <[email protected]> wrote:

...

On 14 Apr 2016, at 16:52, Marius Dumitru Florea < [email protected]> wrote:

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special characters which breaks the URL in Tomcat for security reasons.

Yes but the same is true whether you have “A\.B.C” or "/A\.B/C”.

WDYM? The dot is escaped in the space name with a backslash only when the space name is serialized as a reference, which is not the case for the standard wiki page URL /xwiki/bin/view/Space.With.Dot/Page.With.Dot Having a slash or a backslash in the space or page name is less common than having a dot ("Version 1.2"). And the user might be willing to accept that having a backslash in the page (or attachment's) name can cause security issues with Tomcat, but I doubt he will accept to avoid dots.

That’s not a blocking issue anyway since we can easily transform them into other characters when we serialized and do the opposite when we parse the URL.

...

This is based on the existing TemporaryResourceReference at:

...

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

...

For example:

http://

<server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

...

Note that in this example from the officeviewer macro the

module-dependent

...

resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

Yes. However the image name “Company Presentation-slide0” could also contain slash or backlashes too.

It could but it's less common, especially because most Operating Systems are not very friendly with these characters when used in file or folder names.

Note that I wasn’t sure why you you didn’t compute the base64 of both the name of attachment + the parameters instead of having 2 directory levels consisting in the base64 of the attachment name + the hashcode of the parameters as different path segments. Need to check XWIKI-11528, maybe it’s there.

IMO we need to treat all path segments in the same way and convert slash and backslash into some other characters. I’m not sure we need the base64 solution. But anyway this is an implementation detail of the officeviewer module and not really related to the discussion of the generic Temporary URL format.

Thanks -Vincent

...

- generated image name from PPT

...

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

...

WDYT?

Thanks -Vincent

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

On Fri, Apr 15, 2016 at 11:52 AM, Vincent Massol <[email protected]> wrote:

...

On 15 Apr 2016, at 10:30, Marius Dumitru Florea < [email protected]> wrote:

On Thu, Apr 14, 2016 at 7:46 PM, Vincent Massol <[email protected]> wrote:

...

...

On 14 Apr 2016, at 16:52, Marius Dumitru Florea < [email protected]> wrote:

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special

characters

...

...

which breaks the URL in Tomcat for security reasons.

...

Yes but the same is true whether you have “A\.B.C” or "/A\.B/C”.

WDYM? The dot is escaped in the space name with a backslash only when the space name is serialized as a reference, which is not the case for the standard wiki page URL /xwiki/bin/view/Space.With.Dot/Page.With.Dot

Having a slash or a backslash in the space or page name is less common than having a dot ("Version 1.2"). And the user might be willing to accept that having a backslash in the page (or attachment's) name can cause security issues with Tomcat, but I doubt he will accept to avoid dots.

What do you propose? (I’ve sent another mail explaining why having the reference serialized as different path segments is an issue)

We could also implement a different document reference resolver/serializer for URLs so that the escape symbol is not “\”. Actually maybe this would be the best and would be useful in several places.

+1 Thanks, Marius

WDYT?

Thanks -Vincent

...

That’s not a blocking issue anyway since we can easily transform them into

...

other characters when we serialized and do the opposite when we parse the URL.

...

This is based on the existing TemporaryResourceReference at:

...

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

...

...

...

For example:

http://

<server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

...

...

...

Note that in this example from the officeviewer macro the

module-dependent

...

resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

Yes. However the image name “Company Presentation-slide0” could also contain slash or backlashes too.

It could but it's less common, especially because most Operating Systems are not very friendly with these characters when used in file or folder names.

...

Note that I wasn’t sure why you you didn’t compute the base64 of both

the

...

name of attachment + the parameters instead of having 2 directory levels consisting in the base64 of the attachment name + the hashcode of the parameters as different path segments. Need to check XWIKI-11528, maybe it’s there.

IMO we need to treat all path segments in the same way and convert slash and backslash into some other characters. I’m not sure we need the base64 solution. But anyway this is an implementation detail of the officeviewer module and not really related to the discussion of the generic Temporary URL format.

Thanks -Vincent

...

- generated image name from PPT

...

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

...

...

...

WDYT?

Thanks -Vincent

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs

On 15 Apr 2016, at 11:00, Eduard Moraru <[email protected]> wrote:

Hi,

Re http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

1. What determines this <module id>? What about collisions between extensions not knowing about eachother?

Not a real problem. Ofc each extension needs a unique name. They can use groupid:artifactif if they want. This is at the level of best practice IMO. FTR module id is what we currently use.

2. Are all these temporary resources bound to a document? What about the ones that are not? They would use some random doc reference there? Would that matter?

Right now it’s the case for all our temporary resources. It doesn’t matter if the doc exists or not. This can be just be considered as a unique ref inside the module. If we want to be generic I could just store it as a String in TemporaryResourceReference and have the module using it decide what it represents and do the resolving of it into a Java object (DocumentReference or something else). That would make is fully generic if we want that. Make the module using it a bit more complex (since they need to inject some resolver) but we could provide some helpers probably.

As far as I see it, after http://<server>/<context>/tmp/<module id>/ we can have pretty much anything (i.e. <module-dependent resource path>), depending on the app's needs.

Alternatively, this being a generic service, maybe we can assign a random ID on request and keep a mapping in a separate table.

That’s way too complex IMO. And for a mapping you’d still need to have the module id in that table anyway to be ;) A module needs to be able to construct the URL from its discrete data.

We could even drop the <module id> from the URL and store it in the table instead (as a column). The idea would be that we would not need to expose all these "coordinates" in the actual URL and pollute it with the logic of each individual module, but only use a generic random (unique) ID since it`s a temporary resource anyway. Also, being a separate DB table, the data will not be available on an export since it`s temporary anyway.

What is left, IMO, is a way to clean this temporary data. On the DB table side, we can always clean the table data on startup and the URLs will be gone, however we would need a way to also actually delete the corresponding temporary files and that may be dependant on the module`s logic.

Anyway, WDYT about this direction so far, with using simply http://<server>/<context>/tmp/<resource id>?

I don’t like it for a few reasons: * It makes the URL obscure. Right now I’v voluntarily proposed a scheme where you can still see the resource asked in the URL. Being obscure makes it really hard for debugging issues and understanding what’s happening * It’s too complex and fragile (you depend on the DB and you need the DB in sync with your FS). * You need a unique id generator based on discrete data which cannot be guaranteed Don’t forget that each module using the temporary resource reference need to: * Generate a temporary URL based on discrete elements (in the case I’ve shown: attachment name, current doc name, generated image name, etc) * Generate a temporary file on the Filesystem. If you put everything under the same directory then ** a) you face the OS limitations of the number of files in a single directory ** b) data is not neatly organized in subdirectories which makes it hard to know what is what when you debug So far I really prefer the scheme I proposed. Thanks -Vincent

Thanks, Eduard

On Fri, Apr 15, 2016 at 11:30 AM, Marius Dumitru Florea < [email protected]> wrote:

...

On Thu, Apr 14, 2016 at 7:46 PM, Vincent Massol <[email protected]> wrote:

...

...

On 14 Apr 2016, at 16:52, Marius Dumitru Florea < [email protected]> wrote:

On Thu, Apr 14, 2016 at 5:43 PM, Vincent Massol <[email protected]> wrote:

...

Hi devs,

I’m implementing http://jira.xwiki.org/browse/XWIKI-10375 ("Refactor the temporary resource concept inside the Resource module”) and I need to define a URL format for the new “tmp” resource type.

I’m proposing the following:

...

http://<server>/<context>/tmp/<module id>/<serialized owner document reference>/<module-dependent resource path>

Serialized document reference uses backslash to escape special

characters

...

...

which breaks the URL in Tomcat for security reasons.

...

Yes but the same is true whether you have “A\.B.C” or "/A\.B/C”.

WDYM? The dot is escaped in the space name with a backslash only when the space name is serialized as a reference, which is not the case for the standard wiki page URL /xwiki/bin/view/Space.With.Dot/Page.With.Dot

Having a slash or a backslash in the space or page name is less common than having a dot ("Version 1.2"). And the user might be willing to accept that having a backslash in the page (or attachment's) name can cause security issues with Tomcat, but I doubt he will accept to avoid dots.

...

That’s not a blocking issue anyway since we can easily transform them into other characters when we serialized and do the opposite when we parse the URL.

...

This is based on the existing TemporaryResourceReference at:

...

https://github.com/xwiki/xwiki-platform/blob/96caad053c14fc5546e9bc141bc284e...

...

...

...

For example:

http://

<server>/<context>/tmp/officeviewer/A.B.WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

...

...

...

Note that in this example from the officeviewer macro the

module-dependent

...

resource path consists in:

...

- base64(name of office attachment + hashcode(parameters))

See http://jira.xwiki.org/browse/XWIKI-11528 for the rationale behind it. I was trying to avoid backslash (from the serialized attachment reference) in the URL.

...

Yes. However the image name “Company Presentation-slide0” could also contain slash or backlashes too.

It could but it's less common, especially because most Operating Systems are not very friendly with these characters when used in file or folder names.

...

Note that I wasn’t sure why you you didn’t compute the base64 of both the name of attachment + the parameters instead of having 2 directory levels consisting in the base64 of the attachment name + the hashcode of the parameters as different path segments. Need to check XWIKI-11528, maybe it’s there.

IMO we need to treat all path segments in the same way and convert slash and backslash into some other characters. I’m not sure we need the base64 solution. But anyway this is an implementation detail of the officeviewer module and not really related to the discussion of the generic Temporary URL format.

Thanks -Vincent

...

- generated image name from PPT

...

In this case, the implementation would generate the following file:

[TMPDIR]/officeviewer/A/B/WebHome/Q29tcGFueSBQcmVzZW50YXRpb24ucHB0/Company+Presentation-slide0.jpg

...

...

...

WDYT?

Thanks -Vincent