diff options
| author | David Cabo <david@calibea.com> | 2012-04-13 02:13:38 +0200 |
|---|---|---|
| committer | David Cabo <david@calibea.com> | 2012-04-13 02:13:38 +0200 |
| commit | ef33310d2442d0da93036a06c1bf469b01932b40 (patch) | |
| tree | 4c3c859a7238cce765b6ab009bdc3fd292535ff9 | |
| parent | 24269eb26a23818e539eff159ee9b3c36643b73f (diff) | |
Fix robots.txt to handle URLs with locales
| -rw-r--r-- | public/robots.txt | 23 |
1 files changed, 12 insertions, 11 deletions
diff --git a/public/robots.txt b/public/robots.txt index 6a8628c93..a187d0775 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -13,17 +13,18 @@ # http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-02.html User-agent: * -Disallow: /annotate/ -Disallow: /new/ -Disallow: /search/ -Disallow: /similar/ -Disallow: /track/ -Disallow: /upload/ -Disallow: /user/contact/ -Disallow: /feed/ -Disallow: /profile/ -Disallow: /signin -Disallow: /body/*/view_email$ +Disallow: */annotate/ +Disallow: */new/ +Disallow: */search/ +Disallow: */similar/ +Disallow: */track/ +Disallow: */upload/ +Disallow: */user/contact/ +Disallow: */feed/ +Disallow: */profile/ +Disallow: */signin +Disallow: */request/*/response/ +Disallow: */body/*/view_email$ # The following adding Jan 2012 to stop robots crawling pages # generated in error (see |