4 ms·
If this does bypass their own (and others') anti-AI crawl measures, it'd basically mean that the only people who can't crawl are those without money. We're cre
by mdasen 7mo ago
If this does bypass their own (and others') anti-AI crawl measures, it'd basically mean that the only people who can't crawl are those without money.
We're creating an internet that is becoming self-reinforcing for those who already have power and harder for anyone else. As crawling becomes difficult and expensive, only those with previously collected datasets get to play. I certainly understand individual sites wanting to limit access, but it seems unlikely that they're limiting access to the big players - and maybe even helping them since others won't be able to compete as well.
- adi_kurian 7mo agoCommon Crawl has free egress