takahe/static/robots.txt

User-Agent: *

# Don't allow any bot to crawl tags.
Disallow: /tags/
Disallow: /tags/*

# Don't allow bots to crawl through the proxy
Disallow: /proxy/*
A quick robots.txt to tell bots to stop crawling tags. (#321) 2022-12-30 09:14:12 -08:00			`User-Agent: *`

			`# Don't allow any bot to crawl tags.`
			`Disallow: /tags/`
			`Disallow: /tags/*`
Instruct robots to ignore proxy (#413) 2023-01-14 10:15:18 -08:00
			`# Don't allow bots to crawl through the proxy`
			`Disallow: /proxy/*`