4 ms·
The future is all bot vs. bot
- burkaman 2y ago> Suleyman called content protected by robots.txt a "gray area ... that's going to work its way through the courts." robots.txt is not and has not ever been a matter for the courts. It's a social convention, and if you ignore it you get blocked. If you use your tech giant power to avoid a user-agent or IP-based block, then eventually everyone gets blocked, the content is behind a login or paywall, and you've made the internet worse for everyone and still not gotten what you want.