Hi Vincent, Many thanks for your quick answer. I will list the wiki scope rights that we have set here. (I hope I get the translations right, I have the UI in German.)     Users: unregistered users: allow view, deny edit, deny delete, deny admin, allow register Groups: Redakteure: allow edit, allow delete,                         XWikiAdminGroup: allow edit, allow admin                        XWikiAllGroup: allow view, allow comment, allow register                        The latest destruction was done on the Blog space. I am not sure how the rights were set exactly before the trouble began, but I am abuolutely sure there was no right explicitly allowed to unregistered users (andd 99% sure none was explicityly denied). The option to hinder unregistered users from editing documents independently from page and space rights had not been ticked when we had the trouble last night. I have switched that on now. We also have some kind of trouble with XWiki.XwikiPreferences, which is not deleted entirely, but many (if not all) of the data in it is. These are the rights on the XWiki space: Users: unregistered users: allow view, deny edit, deny delete, deny admin Groups: XWikiAdminGroup: allow edit, allow delete, allow admin                        XWikiAllGroup: allow view                        Cheers, Olaf

Hi I'm a companion of Olaf, the last week we exposed our xwiki to the googlebot to test our configuration. Yesterday we got the same problems with the crawler and we are lost like before. I will get more precise to the problem and quote some of our logs. My results of the analysis: - some critical actions (eg edit) redirects the googlebot to the login page with 302. The login page is 401, the googlebot's path stops here. fine! Log example: example.com - - 66.249.73.10 [27/Apr/2012:15:44:12 +0200] "GET /wiki/example.com/edit/XWiki/GadgetClass HTTP/1.1" 302 20 www.example.com - - 66.249.73.10 [27/Apr/2012:15:44:12 +0200] "GET /wiki/example.com/login/XWiki/XWikiLogin;jsessionid=1F380560FA9E3582D6DDB9B1D286B151?srid=yWSymcYq&xredirect=%2Fwiki%2Fexample.com%2Fedit%2FXWiki%2FGadgetClass%3Fsrid%3DyWSymcYq HTTP/1.1" 401 3004 ============================================================================ - some critical actions results in a OK (200). These are for example deletespace but also some edits: Log example: example.com - - 66.249.73.10 [27/Apr/2012:15:46:30 +0200] "GET /wiki/example.com/get/Hilfe/WebPreferences HTTP/1.1" 200 985 example.com - - 66.249.73.10 [27/Apr/2012:15:46:33 +0200] "GET /wiki/example.com/edit/Blog/WebPreferences HTTP/1.1" 200 5271 example.com - - 66.249.73.10 [27/Apr/2012:15:46:37 +0200] "GET /wiki/example.com/save/Blog/WebPreferences HTTP/1.1" 302 20 example.com - - 66.249.73.10 [27/Apr/2012:15:46:37 +0200] "GET /wiki/example.com/view/Blog/WebPreferences?resubmit=%2Fwiki%2Fexample.com%2Fsave%2FBlog%2FWebPreferences%3Fsrid%3Dn3Ake7tL&xback=%2Fwiki%2Fexample.com%2Fview%2FBlog%2FWebPreferences&xpage=resubmit HTTP/1.1" 200 3689 Here I see one part of the googlebot path. It triggers actions guests are not allowed to. According to the example: When I entered the /edit/Blog/WebPreferences I get a 302 redirect and a 401 login page: jloos@live:~$ curl -IL http:/example.com/wiki/example.com/edit/Blog/WebPreferences HTTP/1.1 302 Moved Temporarily Date: Sat, 28 Apr 2012 00:41:33 GMT Server: Apache/2.2.16 Set-Cookie: JSESSIONID=6819FB0D96E0695388E1AA2A1A92AF49; Path=/ Location: http://example.com/wiki/example.com/login/XWiki/XWikiLogin;jsessionid=6819F… Content-Language: de Vary: Accept-Encoding Content-Type: text/html HTTP/1.1 401 Unauthorized Date: Sat, 28 Apr 2012 00:41:33 GMT Server: Apache/2.2.16 Pragma: no-cache Cache-Control: no-cache Expires: Wed, 31 Dec 1969 23:59:59 GMT Content-Language: de Content-Length: 13590 Vary: Accept-Encoding Content-Type: text/html;charset=utf-8 ============================================================================ - some actions with confirmation forms are delivered to the googlebot too Log example: example.com - - 66.249.71.33 [27/Apr/2012:16:03:05 +0200] "GET /wiki/example.com/deletespace/Start/WebHome HTTP/1.1" 200 3702 [...] example.com - - 66.249.71.33 [27/Apr/2012:16:43:03 +0200] "GET /wiki/example.com/deletespace/Start/WebHome?confirm=1&form_token=saMxN4MidDarWDBvxciU2w HTTP/1.1" 200 3001 So the googlebot gets a form with the csrf-token. Than it chose the yes in the delete confirmation. So our disaster is complete. ============================================================================ I can trace the googlebot actions very well with our logging. But I can't reproduce these actions as a guest in any way. I tried it with and without cookies in several browsers and with curl from the command-line. A wild guess: There seems to be some connections with other user-logins. The last googlebot disaster-actions occurs when a admin logged in and a crawl was in progress. My guess: Under some crazy circumstances, the sessions of a user flips or copied to the crawler. But I think its really far-fetched. The IP seems to be the googlebot: jloos@test:~$ host 66.249.71.33 33.71.249.66.in-addr.arpa domain name pointer crawl-66-249-71-33.googlebot.com. We are using XEM. The Master-Wiki is behind htaccess, and only the relating wiki is free accessible. I hope these detailed analysis isn't to detailed. And I can quote Olaf: Greetings Jan -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp744216… Sent from the XWiki- Users mailing list archive at Nabble.com.

request in the logs as well? Thanks for the fast reply. First of all i have to activate cookie-logging, thanks for the idea ;). Then i have to import our backup and have to wait for the bot. I will reply soon as possible. Greetings Jan -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp744216… Sent from the XWiki- Users mailing list archive at Nabble.com.

In the last couple of weeks we had some small issues again, but no nearly complete wipe. Meanwhile, I switched on the logging of the cookies in our apache proxy and enabled the authentication logging in our XWiki as follows: added in /etc/xwiki/classes/logback.xml Sadly, the cookie and auth logging didn't give any new advice. The cookie section in the html-header for google-crawler requests is empty, especially for the critical edits of guest-protected sites. One really spooky incidence from this week: One of us got a e-mail notification that the AdminSheet (XWiki.XWikiPreferences) has been edited by himself (with his admin account). But this time, I couldn't find any request on the XWikiPreferences in the request-log, and according to the auth-log, no one was logged in. But during this edit, the google-crawler was active on our site. jerem wrote We considered some preventions like this, but in my opinion it doesn't feel good to have such a hole on the site. What if other crawlera doesn't fetch the sitemap, or random visitors get the opportunity to delete content under some special circumstances. Currently, this crawler issue isn't such a performance problem for us. But it's on my agenda to have such a precomputed sitemap for our site, but this is not urgent for us. The crawler-issue causes headaches for me. Maybe we should try to update to a newer XWiki version, at least to get rid of the LDAP-Extension. But without a concrete run down of the real causes of this issues, I will keep a bad feeling. We aren't even able to reproduce the problem, this makes the problem-analysis nearly impossible for us. I hope to get any new hint and tips for things to try out. We are currently really loss. -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp744216… Sent from the XWiki- Users mailing list archive at Nabble.com.

Quick note. In the admin tools extension http://extensions.xwiki.org/xwiki/bin/view/Extension/Admin+Tools+Application There is a page allowing to report all rights from a wiki. The output of that page is very helpful to audit rights issues Ludovic Envoyé de mon iPad Le 6 juin 2012 à 23:15, Jan-Philip Loos <maxdaten(a)gmail.com> a écrit :

Sorry, my fault: I misunderstood the part with the delete. You meant the rights, and not the 'delete' itself. But again to the rights hint: We double-checked the rights of the crucial sites, with different test-users and guests (without sessions). We testes in different browsers and even in VMs on different OS. I wrote even a small crawler to penetrate our site more regularly and controlled. We reproduced some guest-edits with our crawler. But we can't edit these sites when we are moving on our site as a guest manually. So we can't reproduce it directly and repeatable. We only know, edits and deletions are possible for (crawler-)guests sometimes under some unknown circumstances and causes, even if its not possible for 'us', when we are guests. I will test your extension 'Admin Tools Application' with CheckRights you posted on the reply later. Thanks! This will be very useful for our daily routine anyway. -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp744216… Sent from the XWiki- Users mailing list archive at Nabble.com.

Sergiu Dumitriu-2 wrote We are using XEM with a couple of managed wikis. The authentication of the main-wiki (Manage-Wiki) and all sub-wikis is via mysql db but one wiki uses the XWiki build-in LDAP authentication with the LDAP Extension to ease the configuration. The rights management is the XWiki's default for all wikis. We are experiencing this crawler edits/deletions on all public wikis, the Manage-Wiki is behind a .htaccess, set via apache. Sergiu Dumitriu-2 wrote our server architecture: apache2 is our web frontend server which sends requests for XWiki specific urls via mod_jk to the Tomcat servlet server. configuration details: apache2: - mainly debian default base-configuration (except specific site-configuration) - mod_jk for apache to tomcat connection tomcat6: - one AJP/1.3 connector only on localhost - only XEM running on tomcat XWiki 3.4 (xwiki.cfg): - custom "xwiki.authentication.validationKey" and "xwiki.authentication.encryptionKey" are set - "xwiki.authentication.authclass=com.xpn.xwiki.user.impl.LDAP.XWikiLDAPAuthServiceImpl" - "xwiki.authentication.ldap.server=127.0.0.1" and "xwiki.authentication.ldap.port=389" - no further LDAP configuration in xwiki.cfg (made direct in the specific wiki) The LDAP-Wiki: - LDAP Login-Matching like: "uid={0},ou=users,dc=example,dc=com" - Password-Matching: "{1}" - LDAP Base DN: "ou=users,dc=example,dc=com" - UID Attribute Name: "uid" - update from LDAP: yes - try local login: yes - group-mapping like: XWiki.XWikiAdminGroup=cn=admin,ou=groups,dc=example,dc=com - sync-mode: always OpenLDAP: - listens 127.0.0.1 and port 389 only Some additional notes: We are using LDAP to share one user-database with our public forum. So we are running phpBB and XWiki connected to the same LDAP directory. Thanks for your time Jan -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp744216… Sent from the XWiki- Users mailing list archive at Nabble.com.

On 07/13/2012 05:31 PM, Jan-Philip Loos wrote: OK, this is a promising direction, at least I know what's going on. So, the problem is that the XWikiPreferences document gets in a very bad state in the document cache. The cache coherence in multiwiki mode has been improved a while ago, so it shouldn't happen with normal documents, but there's still http://jira.xwiki.org/browse/XWIKI-8046 which is open and which has top priority (it might get fixed in a few days for 4.1.4). Unfortunately, I don't have a quick solution for you at the moment. Watch that issue for more details. -- Sergiu Dumitriu http://purl.org/net/sergiu/

Hi This could have showed up because of. Changes a while ago that flush documents from the cache after save. So combined with a crawler trigerring saves it could trigger the behavior seen here where flushCache the restores everything. Maybe there is a reading failure which is not detected properly and the code falls back to a default class ? Envoyé de mon iPhone Le 14 juil. 2012 à 03:15, Sergiu Dumitriu <sergiu(a)xwiki.com> a écrit :

Hi Jan-Philip, On Sep 10, 2012, at 9:37 PM, Jan-Philip Loos <maxdaten(a)gmail.com> wrote: Actually we've made some progress on a similar issue: See http://jira.xwiki.org/browse/XWIKI-8160 I don't know if the problem is the same or not but it gives some direction. JV, WDYT? Thanks -Vincent

We upgraded to the latest minor versions as they were released, first migration was from 3.1 to 4.2, followed by 4.2 -> 4.3 and finally 4.3 -> 4.4 The problem still occurred with 4.3 (4.4 is to new to figured it out now). In the last 3-4 month the problem occurred round about 3 times. I wrote a small script which compares the name of the header image. If the name is not the one customized in the XWiki.DefaultSkin it notifies me with an email. Notable: When the problem occurs its tends to occur on the same weekday - Wednesday, but not regular every Wednesday. I checked all system crons and xwiki scheduled jobs, no one was triggered at the same time the problems occurred. The rate seems to be lower than before, but I guess it's only because the crawler isn't as active as before and we have less visits per day. To outline it once more: - When I flush the cache manually with the Admin Tools extension, the subwiki is immediately restored. - The state (e.g. the header) is not the one from the main XEM-Wiki, its the default one - The content of the wiki is still present, "only" the styles and sadly the rights seems to be the default one We consider to switch now back to XE, in hope to get rid of the problem. But this seems not as easy as I hoped. Currently XEM is not super essential for us, but it will leave me sick behind not being able to solve the problem. -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp744216… Sent from the XWiki- Users mailing list archive at Nabble.com.

Hi Vincent, the last weeks I was busy with my studies, I will try the patch/workaround next week and keep you informed. Greetings Jan-Philip -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp744216… Sent from the XWiki- Users mailing list archive at Nabble.com.