ad4451276d
* Add a `robots.txt` file to prevent crawlers from scraping the site * Added `ASSET_RIGHTS` entry to config.yaml to control whether `/robots.txt` is served or not * Always import robots.py, determine config in route function * Finish writing a comment * Remove unnecessary redundant import and config
21 lines
275 B
Plaintext
21 lines
275 B
Plaintext
User-agent: Googlebot
|
|
Disallow: /
|
|
|
|
User-agent: APIs-Google
|
|
Disallow: /
|
|
|
|
User-agent: AdsBot-Google-Mobile
|
|
Disallow: /
|
|
|
|
User-agent: AdsBot-Google-Mobile
|
|
Disallow: /
|
|
|
|
User-agent: Mediapartners-Google
|
|
Disallow: /
|
|
|
|
User-agent: Google-Safety
|
|
Disallow: /
|
|
|
|
User-agent: *
|
|
Disallow: /
|