Hi Vincent, Many thanks for your quick answer. I will list the wiki scope rights that we have set here. (I hope I get the translations right, I have the UI in German.)     Users: unregistered users: allow view, deny edit, deny delete, deny admin, allow register Groups: Redakteure: allow edit, allow delete,                         XWikiAdminGroup: allow edit, allow admin                        XWikiAllGroup: allow view, allow comment, allow register                        The latest destruction was done on the Blog space. I am not sure how the rights were set exactly before the trouble began, but I am abuolutely sure there was no right explicitly allowed to unregistered users (andd 99% sure none was explicityly denied). The option to hinder unregistered users from editing documents independently from page and space rights had not been ticked when we had the trouble last night. I have switched that on now. We also have some kind of trouble with XWiki.XwikiPreferences, which is not deleted entirely, but many (if not all) of the data in it is. These are the rights on the XWiki space: Users: unregistered users: allow view, deny edit, deny delete, deny admin Groups: XWikiAdminGroup: allow edit, allow delete, allow admin                        XWikiAllGroup: allow view                        Cheers, Olaf

________________________________ Von: Vincent Massol <[email protected]> An: XWiki Users <[email protected]> CC: O Voss <[email protected]> Gesendet: 8:33 Freitag, 6.April 2012 Betreff: Re: [xwiki-users] severe trouble with web crawlers

Hi Olaf,

On Apr 6, 2012, at 8:15 AM, O Voss wrote:

...

Hi,

a web crawler keeps deleting documents in our Wiki. Meanwhile we have a robots.txt, but it seems to ignore that.

Of course, you should never allow unregistered users to delete documents, but you know that already… ;)

...

In the global rights the unregistered user is explicitely denied edit, delete and admin rights. Nowhere are such rights set to allowed. What are we missing?

Personally I never use deny rights. I simply enable the rights for the user/group that should get the rights. The others are denied by default.

You should also check rights at 3 levels: - global - space level - page level

...

The deleted documents do not turn up in XWiki/DeletedDocuments. Is there any way to restore the deleted documents?

Only from a backup.

...

This is an absolute desaster atm, any hints greatly appreciated!

We'd need to see your full rights setup to help me.

You could check http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Access+Rights (not perfect doc though) to understand rights a bit more.

It won't help you but FYI we're planning to redesign the rights UI to make it even easier to use: http://incubator.myxwiki.org/xwiki/bin/view/Improvements/Rights

Thanks -Vincent

Hi I'm a companion of Olaf, the last week we exposed our xwiki to the googlebot to test our configuration. Yesterday we got the same problems with the crawler and we are lost like before. I will get more precise to the problem and quote some of our logs. My results of the analysis: - some critical actions (eg edit) redirects the googlebot to the login page with 302. The login page is 401, the googlebot's path stops here. fine! Log example: example.com - - 66.249.73.10 [27/Apr/2012:15:44:12 +0200] "GET /wiki/example.com/edit/XWiki/GadgetClass HTTP/1.1" 302 20 www.example.com - - 66.249.73.10 [27/Apr/2012:15:44:12 +0200] "GET /wiki/example.com/login/XWiki/XWikiLogin;jsessionid=1F380560FA9E3582D6DDB9B1D286B151?srid=yWSymcYq&xredirect=%2Fwiki%2Fexample.com%2Fedit%2FXWiki%2FGadgetClass%3Fsrid%3DyWSymcYq HTTP/1.1" 401 3004 ============================================================================ - some critical actions results in a OK (200). These are for example deletespace but also some edits: Log example: example.com - - 66.249.73.10 [27/Apr/2012:15:46:30 +0200] "GET /wiki/example.com/get/Hilfe/WebPreferences HTTP/1.1" 200 985 example.com - - 66.249.73.10 [27/Apr/2012:15:46:33 +0200] "GET /wiki/example.com/edit/Blog/WebPreferences HTTP/1.1" 200 5271 example.com - - 66.249.73.10 [27/Apr/2012:15:46:37 +0200] "GET /wiki/example.com/save/Blog/WebPreferences HTTP/1.1" 302 20 example.com - - 66.249.73.10 [27/Apr/2012:15:46:37 +0200] "GET /wiki/example.com/view/Blog/WebPreferences?resubmit=%2Fwiki%2Fexample.com%2Fsave%2FBlog%2FWebPreferences%3Fsrid%3Dn3Ake7tL&xback=%2Fwiki%2Fexample.com%2Fview%2FBlog%2FWebPreferences&xpage=resubmit HTTP/1.1" 200 3689 Here I see one part of the googlebot path. It triggers actions guests are not allowed to. According to the example: When I entered the /edit/Blog/WebPreferences I get a 302 redirect and a 401 login page: jloos@live:~$ curl -IL http:/example.com/wiki/example.com/edit/Blog/WebPreferences HTTP/1.1 302 Moved Temporarily Date: Sat, 28 Apr 2012 00:41:33 GMT Server: Apache/2.2.16 Set-Cookie: JSESSIONID=6819FB0D96E0695388E1AA2A1A92AF49; Path=/ Location: http://example.com/wiki/example.com/login/XWiki/XWikiLogin;jsessionid=6819FB... Content-Language: de Vary: Accept-Encoding Content-Type: text/html HTTP/1.1 401 Unauthorized Date: Sat, 28 Apr 2012 00:41:33 GMT Server: Apache/2.2.16 Pragma: no-cache Cache-Control: no-cache Expires: Wed, 31 Dec 1969 23:59:59 GMT Content-Language: de Content-Length: 13590 Vary: Accept-Encoding Content-Type: text/html;charset=utf-8 ============================================================================ - some actions with confirmation forms are delivered to the googlebot too Log example: example.com - - 66.249.71.33 [27/Apr/2012:16:03:05 +0200] "GET /wiki/example.com/deletespace/Start/WebHome HTTP/1.1" 200 3702 [...] example.com - - 66.249.71.33 [27/Apr/2012:16:43:03 +0200] "GET /wiki/example.com/deletespace/Start/WebHome?confirm=1&form_token=saMxN4MidDarWDBvxciU2w HTTP/1.1" 200 3001 So the googlebot gets a form with the csrf-token. Than it chose the yes in the delete confirmation. So our disaster is complete. ============================================================================ I can trace the googlebot actions very well with our logging. But I can't reproduce these actions as a guest in any way. I tried it with and without cookies in several browsers and with curl from the command-line. A wild guess: There seems to be some connections with other user-logins. The last googlebot disaster-actions occurs when a admin logged in and a crawl was in progress. My guess: Under some crazy circumstances, the sessions of a user flips or copied to the crawler. But I think its really far-fetched. The IP seems to be the googlebot: jloos@test:~$ host 66.249.71.33 33.71.249.66.in-addr.arpa domain name pointer crawl-66-249-71-33.googlebot.com. We are using XEM. The Master-Wiki is behind htaccess, and only the relating wiki is free accessible. I hope these detailed analysis isn't to detailed. And I can quote Olaf:

any hints greatly appreciated!

Greetings Jan -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com.

Well, this is indeed strange. Could you print the cookies sent with each request in the logs as well?

Thanks for the fast reply. First of all i have to activate cookie-logging, thanks for the idea ;). Then i have to import our backup and have to wait for the bot. I will reply soon as possible. Greetings Jan -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com.

In the last couple of weeks we had some small issues again, but no nearly complete wipe. Meanwhile, I switched on the logging of the cookies in our apache proxy and enabled the authentication logging in our XWiki as follows: added in /etc/xwiki/classes/logback.xml

<logger name="com.xpn.xwiki.user.impl.xwiki.XWikiAuthServiceImpl" level="info"/> <logger name="com.xpn.xwiki.user.impl.LDAP.XWikiLDAPAuthServiceImpl" level="info"/>

Sadly, the cookie and auth logging didn't give any new advice. The cookie section in the html-header for google-crawler requests is empty, especially for the critical edits of guest-protected sites. One really spooky incidence from this week: One of us got a e-mail notification that the AdminSheet (XWiki.XWikiPreferences) has been edited by himself (with his admin account). But this time, I couldn't find any request on the XWikiPreferences in the request-log, and according to the auth-log, no one was logged in. But during this edit, the google-crawler was active on our site. jerem wrote

For my wiki, we used to have a crawler like this (snuffing every links out there), then, due to the useless time spent trying to retrieve pages and checking rights, I wrote small scripts to provide an indexing page to the crawler.

We considered some preventions like this, but in my opinion it doesn't feel good to have such a hole on the site. What if other crawlera doesn't fetch the sitemap, or random visitors get the opportunity to delete content under some special circumstances. Currently, this crawler issue isn't such a performance problem for us. But it's on my agenda to have such a precomputed sitemap for our site, but this is not urgent for us. The crawler-issue causes headaches for me. Maybe we should try to update to a newer XWiki version, at least to get rid of the LDAP-Extension. But without a concrete run down of the real causes of this issues, I will keep a bad feeling. We aren't even able to reproduce the problem, this makes the problem-analysis nearly impossible for us. I hope to get any new hint and tips for things to try out. We are currently really loss. -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com.

Quick note. In the admin tools extension http://extensions.xwiki.org/xwiki/bin/view/Extension/Admin+Tools+Application There is a page allowing to report all rights from a wiki. The output of that page is very helpful to audit rights issues Ludovic Envoyé de mon iPad Le 6 juin 2012 à 23:15, Jan-Philip Loos <[email protected]> a écrit :

In the last couple of weeks we had some small issues again, but no nearly complete wipe. Meanwhile, I switched on the logging of the cookies in our apache proxy and enabled the authentication logging in our XWiki as follows:

added in /etc/xwiki/classes/logback.xml

...

<logger name="com.xpn.xwiki.user.impl.xwiki.XWikiAuthServiceImpl" level="info"/> <logger name="com.xpn.xwiki.user.impl.LDAP.XWikiLDAPAuthServiceImpl" level="info"/>

Sadly, the cookie and auth logging didn't give any new advice. The cookie section in the html-header for google-crawler requests is empty, especially for the critical edits of guest-protected sites.

One really spooky incidence from this week:

One of us got a e-mail notification that the AdminSheet (XWiki.XWikiPreferences) has been edited by himself (with his admin account). But this time, I couldn't find any request on the XWikiPreferences in the request-log, and according to the auth-log, no one was logged in. But during this edit, the google-crawler was active on our site.

jerem wrote

...

For my wiki, we used to have a crawler like this (snuffing every links out there), then, due to the useless time spent trying to retrieve pages and checking rights, I wrote small scripts to provide an indexing page to the crawler.

We considered some preventions like this, but in my opinion it doesn't feel good to have such a hole on the site. What if other crawlera doesn't fetch the sitemap, or random visitors get the opportunity to delete content under some special circumstances. Currently, this crawler issue isn't such a performance problem for us. But it's on my agenda to have such a precomputed sitemap for our site, but this is not urgent for us. The crawler-issue causes headaches for me.

Maybe we should try to update to a newer XWiki version, at least to get rid of the LDAP-Extension. But without a concrete run down of the real causes of this issues, I will keep a bad feeling.

We aren't even able to reproduce the problem, this makes the problem-analysis nearly impossible for us.

I hope to get any new hint and tips for things to try out. We are currently really loss.

-- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com. _______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users

Sorry, my fault: I misunderstood the part with the delete. You meant the rights, and not the 'delete' itself. But again to the rights hint: We double-checked the rights of the crucial sites, with different test-users and guests (without sessions). We testes in different browsers and even in VMs on different OS. I wrote even a small crawler to penetrate our site more regularly and controlled. We reproduced some guest-edits with our crawler. But we can't edit these sites when we are moving on our site as a guest manually. So we can't reproduce it directly and repeatable. We only know, edits and deletions are possible for (crawler-)guests sometimes under some unknown circumstances and causes, even if its not possible for 'us', when we are guests. I will test your extension 'Admin Tools Application' with CheckRights you posted on the reply later. Thanks! This will be very useful for our daily routine anyway. -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com.

Sergiu Dumitriu-2 wrote

One important question nobody asked so far: are you using any custom authenticator or rights implementation, like LDAP, Kerberos or another SSO?

We are using XEM with a couple of managed wikis. The authentication of the main-wiki (Manage-Wiki) and all sub-wikis is via mysql db but one wiki uses the XWiki build-in LDAP authentication with the LDAP Extension to ease the configuration. The rights management is the XWiki's default for all wikis. We are experiencing this crawler edits/deletions on all public wikis, the Manage-Wiki is behind a .htaccess, set via apache. Sergiu Dumitriu-2 wrote

Another possibility is that the servlet container or a frontend server is mangling sessions, putting the Google bot in the same session as a valid authenticated user. Can you give us more details about your setup? Like: Tomcat + Apache HTTPD + mod_proxy_http

our server architecture: apache2 is our web frontend server which sends requests for XWiki specific urls via mod_jk to the Tomcat servlet server. configuration details: apache2: - mainly debian default base-configuration (except specific site-configuration) - mod_jk for apache to tomcat connection tomcat6: - one AJP/1.3 connector only on localhost - only XEM running on tomcat XWiki 3.4 (xwiki.cfg): - custom "xwiki.authentication.validationKey" and "xwiki.authentication.encryptionKey" are set - "xwiki.authentication.authclass=com.xpn.xwiki.user.impl.LDAP.XWikiLDAPAuthServiceImpl" - "xwiki.authentication.ldap.server=127.0.0.1" and "xwiki.authentication.ldap.port=389" - no further LDAP configuration in xwiki.cfg (made direct in the specific wiki) The LDAP-Wiki: - LDAP Login-Matching like: "uid={0},ou=users,dc=example,dc=com" - Password-Matching: "{1}" - LDAP Base DN: "ou=users,dc=example,dc=com" - UID Attribute Name: "uid" - update from LDAP: yes - try local login: yes - group-mapping like: XWiki.XWikiAdminGroup=cn=admin,ou=groups,dc=example,dc=com - sync-mode: always OpenLDAP: - listens 127.0.0.1 and port 389 only Some additional notes: We are using LDAP to share one user-database with our public forum. So we are running phpBB and XWiki connected to the same LDAP directory. Thanks for your time Jan -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com.

On 07/13/2012 05:31 PM, Jan-Philip Loos wrote:

Hello,

I have an really important update for this issue.

Last month I installed the the "Admin Tools Application" as recommended by Ludovic Dubost. I ran the "Programming Rights Check" script and inspected the "User Rights Check" page. No right anomalies, as far as I can see.

A few minutes ago, our XWiki suffered under a new, but correlating, problem. I visited our Wiki and noticed that the Wiki was rendered with the default theme and with the default header image. Furthermore, as a guest, I had the opportunity to delete spaces/documents and visit restricted areas.

I inspected this issue with the Admin Tools: I checked the memory usage of our XEM: 1,5GB Free Active DB Connections: 13 of 50 Idle DB Connections: 5 of 5

After I tried the "Flush Cache" script of the "Admin Tools Application", the Wiki theme and header image was instant restored, and all rights seemed working.

I have a strong guess that this issue is connected to our initial problem. I guess this problem fixed itself a couple of times after a period of time. During this period, the Wiki is exposed and the crawler can trigger undesired actions.

I hope someone have an idea why our Wiki behaves like this.

Greetings

Jan

OK, this is a promising direction, at least I know what's going on. So, the problem is that the XWikiPreferences document gets in a very bad state in the document cache. The cache coherence in multiwiki mode has been improved a while ago, so it shouldn't happen with normal documents, but there's still http://jira.xwiki.org/browse/XWIKI-8046 which is open and which has top priority (it might get fixed in a few days for 4.1.4). Unfortunately, I don't have a quick solution for you at the moment. Watch that issue for more details. -- Sergiu Dumitriu http://purl.org/net/sergiu/

Hi This could have showed up because of. Changes a while ago that flush documents from the cache after save. So combined with a crawler trigerring saves it could trigger the behavior seen here where flushCache the restores everything. Maybe there is a reading failure which is not detected properly and the code falls back to a default class ? Envoyé de mon iPhone Le 14 juil. 2012 à 03:15, Sergiu Dumitriu <[email protected]> a écrit :

On 07/13/2012 05:31 PM, Jan-Philip Loos wrote:

...

Hello,

I have an really important update for this issue.

Last month I installed the the "Admin Tools Application" as recommended by Ludovic Dubost. I ran the "Programming Rights Check" script and inspected the "User Rights Check" page. No right anomalies, as far as I can see.

A few minutes ago, our XWiki suffered under a new, but correlating, problem. I visited our Wiki and noticed that the Wiki was rendered with the default theme and with the default header image. Furthermore, as a guest, I had the opportunity to delete spaces/documents and visit restricted areas.

I inspected this issue with the Admin Tools: I checked the memory usage of our XEM: 1,5GB Free Active DB Connections: 13 of 50 Idle DB Connections: 5 of 5

After I tried the "Flush Cache" script of the "Admin Tools Application", the Wiki theme and header image was instant restored, and all rights seemed working.

I have a strong guess that this issue is connected to our initial problem. I guess this problem fixed itself a couple of times after a period of time. During this period, the Wiki is exposed and the crawler can trigger undesired actions.

I hope someone have an idea why our Wiki behaves like this.

Greetings

Jan

OK, this is a promising direction, at least I know what's going on.

So, the problem is that the XWikiPreferences document gets in a very bad state in the document cache. The cache coherence in multiwiki mode has been improved a while ago, so it shouldn't happen with normal documents, but there's still http://jira.xwiki.org/browse/XWIKI-8046 which is open and which has top priority (it might get fixed in a few days for 4.1.4).

Unfortunately, I don't have a quick solution for you at the moment. Watch that issue for more details. -- Sergiu Dumitriu http://purl.org/net/sergiu/

_______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users

Hi Jan-Philip, On Sep 10, 2012, at 9:37 PM, Jan-Philip Loos <[email protected]> wrote:

Hello,

it's been a while since the last posting. We experienced the problem a couple of times, but without any serve impact.

I watched the jira issue mentioned by Sergiu Dumitriu. It's status is still open, major and unassigned, the resolution is unresolved. Any off-jira updates for this issue?

According to the roadmap, XWiki 4.2 will be released this month, and we are looking forward to this version. Any chances a fix is or will be in this release? Will be the hack posted by Jean-Vincent Drean in jira valid and free of any side-effects?

Actually we've made some progress on a similar issue: See http://jira.xwiki.org/browse/XWIKI-8160 I don't know if the problem is the same or not but it gives some direction. JV, WDYT? Thanks -Vincent

Greeting Jan

jira issue: http://jira.xwiki.org/browse/XWIKI-8046

-- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com. _______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users

We upgraded to the latest minor versions as they were released, first migration was from 3.1 to 4.2, followed by 4.2 -> 4.3 and finally 4.3 -> 4.4 The problem still occurred with 4.3 (4.4 is to new to figured it out now). In the last 3-4 month the problem occurred round about 3 times. I wrote a small script which compares the name of the header image. If the name is not the one customized in the XWiki.DefaultSkin it notifies me with an email. Notable: When the problem occurs its tends to occur on the same weekday - Wednesday, but not regular every Wednesday. I checked all system crons and xwiki scheduled jobs, no one was triggered at the same time the problems occurred. The rate seems to be lower than before, but I guess it's only because the crawler isn't as active as before and we have less visits per day. To outline it once more: - When I flush the cache manually with the Admin Tools extension, the subwiki is immediately restored. - The state (e.g. the header) is not the one from the main XEM-Wiki, its the default one - The content of the wiki is still present, "only" the styles and sadly the rights seems to be the default one We consider to switch now back to XE, in hope to get rid of the problem. But this seems not as easy as I hoped. Currently XEM is not super essential for us, but it will leave me sick behind not being able to solve the problem. -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com.

Hi Vincent, the last weeks I was busy with my studies, I will try the patch/workaround next week and keep you informed. Greetings Jan-Philip -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162... Sent from the XWiki- Users mailing list archive at Nabble.com.