4 ms·
That's basically asking to close the market in favor of the current actors. New actors have the right to emerge.
by realusername 2y ago
That's basically asking to close the market in favor of the current actors.
New actors have the right to emerge.
- 0dayz 2y agoNo they don't. There's no rule that you have to let anyone in who claims to be a web crawler.
- areyourllySorry 2y agowhich is why they will stop claiming to be one.
- chii 2y agoso what happened to competition fostering a better outcome for all then?
- realusername 2y agoSo who decides that you can be one? Right now it's Cloudflare, a litteral monopoly... The truth is that I sympathize with the people trying to use mobile connections to bypass such a cartel. What Cloudflare is doing now is worse than the web crawlers themselves and the legality of blocking crawlers with a monopoly is dubious at best.
- jeroenhd 2y agoThey have the right to try to convince me to let them scrape me. Most of the time they're thinly veiled data traders. I haven't seen any new company try to scrape my stuff since maybe Kagi. Kagi is welcome to scrape from their IP addresses. Other bots that behave are fine too (Huawei and various other Chinese bots don't and I've had to put an IP block on those).