4 ms·
Maybe GPTBot's distant cousin "No user agent" will come visit just a few minutes later.
by blitzar 3y ago
Maybe GPTBot's distant cousin "No user agent" will come visit just a few minutes later.
- NoZebra120vClip 3y agoYes, I'm skeptical of the headline. Are they blocking a UA or what? It was going to be the LLM-scrapers ignoring robots.txt in order to promote some kind of new standard of discouraging them. The scrapers are all rewriting their privacy policies to just say "all your text is belong to us" so I don't see much recourse for the media outlets, aside from being large with lots of aggressive IP attorneys. And what becomes of all the archive.is pages we're feeding to them anyway?