3 ms·
And a few decades ago, it would have been search engine scrapers instead.
by wvenable 1y ago
And a few decades ago, it would have been search engine scrapers instead.
- not2b 1y agoAnd that problem was largely solved by robots.txt. AI scrapers are ignoring robots.txt and beating the hell out of sites. Small sites that have decades worth of quality information are suffering the most. Many of the scrapers are taking extreme measures to avoid being blocked, like using large numbers of distinct IP addresses (perhaps using botnets).