The problem is probably to dynamically update the robot.txt file when content (read new page) is added. Maybe it helkps the content is stored in a OBJECT and displayed only when a certain programmatically boolean is set On 16 nov 2008, at 17:33, Sergiu Dumitriu wrote:
Besides that, How can I control what content is indexed by external search engines (like Google, Yahoo) I have specific content that I would like to exclude from indexes, although it is on public accessible pages. In our system it is accessible by clcking a disclosure (because it contains medical advise)
I think you need to use a robots.txt file with the names of the pages you want to exclude from indexation.
Something like: http://platform.xwiki.org/robots.txt
-- Sergiu Dumitriu http://purl.org/net/sergiu/ _______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users