26 ms·
Yes, I'm skeptical of the headline. Are they blocking a UA or what? It was going to be the LLM-scrapers ignoring robots.txt in order to promote some kind of new
by NoZebra120vClip 3y ago
Yes, I'm skeptical of the headline. Are they blocking a UA or what? It was going to be the LLM-scrapers ignoring robots.txt in order to promote some kind of new standard of discouraging them.
The scrapers are all rewriting their privacy policies to just say "all your text is belong to us" so I don't see much recourse for the media outlets, aside from being large with lots of aggressive IP attorneys.
And what becomes of all the archive.is pages we're feeding to them anyway?