morgan/Grinch-AP
1
0
Fork 0
mirror of https://github.com/MarioSpore/Grinch-AP.git synced 2025-10-21 20:21:32 -06:00
Code Issues Packages Projects Releases Wiki Activity

WebHost: Add robots.txt to WebHost (#3157)

Browse Source 
* Add a `robots.txt` file to prevent crawlers from scraping the site

* Added `ASSET_RIGHTS` entry to config.yaml to control whether `/robots.txt` is served or not

* Always import robots.py, determine config in route function

* Finish writing a comment

* Remove unnecessary redundant import and config
This commit is contained in:
Chris Wilson
2024-04-20 20:58:56 -04:00
committed by GitHub
parent 915ad61ecf
commit ad4451276d
5 changed files with 44 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
WebHostLib/static/robots.txt Normal file
View File

@@ -0,0 +1,20 @@
User-agent: Googlebot
Disallow: /
User-agent: APIs-Google
Disallow: /
User-agent: AdsBot-Google-Mobile
Disallow: /
User-agent: AdsBot-Google-Mobile
Disallow: /
User-agent: Mediapartners-Google
Disallow: /
User-agent: Google-Safety
Disallow: /
User-agent: *
Disallow: /