Commit Graph

2 Commits

Author SHA1 Message Date
Fabian Dill
ce37bed7c6 WebHost: fix accidental robots.txt capture (#3502) 2024-06-21 14:54:19 +02:00
Chris Wilson
ad4451276d WebHost: Add robots.txt to WebHost (#3157)
* Add a `robots.txt` file to prevent crawlers from scraping the site

* Added `ASSET_RIGHTS` entry to config.yaml to control whether `/robots.txt` is served or not

* Always import robots.py, determine config in route function

* Finish writing a comment

* Remove unnecessary redundant import and config
2024-04-20 20:58:56 -04:00