zhouzhongping/wiki
1
0
Fork 0
Code

项目：新增 robots.txt

Browse Source
This commit is contained in:
周中平 2022-09-09 17:33:28 +08:00
parent 2106e98c0d
commit f05b93e8c3
No known key found for this signature in database
GPG Key ID: B1DF9DD42D8E00DC
1 changed files with 31 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
static/robots.txt Normal file
View File

@ -0,0 +1,31 @@
User-agent: Baiduspider
Disallow:
User-agent: Sosospider
Disallow:
User-agent: sogou spider
Disallow:
User-agent: YodaoBot
Disallow:
User-agent: Googlebot
Disallow:
User-agent: Bingbot
Disallow:
User-agent: Slurp
Disallow:
User-agent: MSNBot
Disallow:
User-agent: googlebot-image
Disallow:
User-agent: googlebot-mobile
Disallow:
User-agent: yahoo-mmcrawler
Disallow:
User-agent: yahoo-blogs/v3.9
Disallow:
User-agent: psbot
Disallow:
User-agent: *
Disallow:
Crawl-delay: 60
Sitemap: https://wiki.7wate.com/sitemap.xml
Sitemap: http://wiki.7wate.com/sitemap.xml