2019年 大佬有话说 :
CF里显示GOOGLE BOT一小时爬10K的次数。。这正常吗
本帖最后由 2019年 于 2020-12-6 19:04 编辑
RT……………我做英文站的
顺便分享一下网上找的很好的屏蔽bad BOT的规则，这个适用做英文站的哈，因为屏蔽了中国的一些bot
(http.user_agent contains "Yandex") or (http.user_agent contains "muckrack") or (http.user_agent contains "Qwantify") or (http.user_agent contains "Sogou") or (http.user_agent contains "BUbiNG") or (http.user_agent contains "knowledge") or (http.user_agent contains "CFNetwork") or (http.user_agent contains "Scrapy") or (http.user_agent contains "SemrushBot") or (http.user_agent contains "AhrefsBot") or (http.user_agent contains "Baiduspider") or (http.user_agent contains "python-requests") or (http.user_agent contains "crawl" and not cf.client.bot) or (http.user_agent contains "Crawl" and not cf.client.bot) or (http.user_agent contains "bot" and not http.user_agent contains "bingbot" and not http.user_agent contains "Google" and not http.user_agent contains "推特" and not cf.client.bot) or (http.user_agent contains "Bot" and not http.user_agent contains "Google" and not cf.client.bot) or (http.user_agent contains "Spider" and not cf.client.bot) or (http.user_agent contains "spider" and not cf.client.bot)
下图是我的CF介面，好BOT(浅蓝allow）的流量远超BAD BOT
https://scrn.li/uploads/b/bb/bb8/bb82/bb82d/bb82d4/bb82d474-f49e-01d9-997e-300cae88a97f/V84Wh47cg0o1d2
has 大佬有话说 :
楼上的规则会屏蔽谷歌吗
2019年 大佬有话说 :
has 大佬有话说 : 2020-12-6 18:46
楼上的规则会屏蔽谷歌吗
不会的，我使用了，很好用，
contains "Google" and not cf.client.bot，这个是屏蔽假google bot
Lemon0 大佬有话说 :
为什么屏蔽 yandex sougo呢