3 ms·
PSA – ByteDance and Bytespider Bots – Recommend Blocking
- exar0815 3y agoAlso, interesting Video - albeit low content density - pointing out that those bots cause humongous amounts of traffic, maxing out most vps bandwidth plans and are not currently blocked by Cloudflare. For me, it looks like ByteDance is aggressively scraping the web for ML purposes.
- bediger4000 3y agoThe article is correct: Bytespider does not observe robots.txt directives. I set Apache to give Bytespider 404 on anything. Not even Ahrefsbot is this poorly behaved.