FreakU
/
takahe
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Instruct robots to ignore proxy (#413)

This commit is contained in:
Corry Haines 2023-01-14 10:15:18 -08:00 committed by GitHub
parent 2675ed69ee
commit ade954c2cd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
static/robots.txt
View File

@ -3,3 +3,6 @@ User-Agent: *
# Don't allow any bot to crawl tags. # Don't allow any bot to crawl tags.
Disallow: /tags/ Disallow: /tags/
Disallow: /tags/* Disallow: /tags/*
# Don't allow bots to crawl through the proxy
Disallow: /proxy/*