4 ms·
No they don't. There's no rule that you have to let anyone in who claims to be a web crawler.
by 0dayz 1y ago
No they don't.
There's no rule that you have to let anyone in who claims to be a web crawler.
- areyourllySorry 1y agowhich is why they will stop claiming to be one.
- chii 1y agoso what happened to competition fostering a better outcome for all then?
- realusername 1y agoSo who decides that you can be one? Right now it's Cloudflare, a litteral monopoly... The truth is that I sympathize with the people trying to use mobile connections to bypass such a cartel. What Cloudflare is doing now is worse than the web crawlers themselves and the legality of blocking crawlers with a monopoly is dubious at best.