3 ms·
Why? Why is a random response code not good enough? Rather we need some AI useragent, that's enforced somehow on crawlers, so they can be filtered out. Just a
by not_your_vase 2y ago
Why? Why is a random response code not good enough?
Rather we need some AI useragent, that's enforced somehow on crawlers, so they can be filtered out.
Just a few days ago, an OSS project that I participate in had its wiki scraped, that looked like some random mac user agent, but suddenly from around the world, everyone looking at the wiki history, reading with crazy speed...
- gigaArpit 2y agoEach code has a defined meaning, which helps clients and servers understand the nature of the response.
- not_your_vase 2y agoI know very well what an http response code is. But what is the reason for wanting to extend the standard? What exactly makes 450 better than 404? Who benefits from it, and in what form?
- gigaArpit 2y agoThe idea behind introducing a new HTTP status code for blocking AI traffic should be to provide a clearer signal to both users and automated systems about the nature of the response. While a 404 indicates that a resource is not found, it doesn't convey the specific reason that the content is being restricted from AI access. While User-Agents can easily be spoofed by users and automated traffic, as you mentioned in your earlier reply, the HTTP status code would be a clear message from the server.
- toomuchtodo 2y agoIf it’s a clear message, they will simply avoid your AI identification heuristics. Better to feed them 404s so they eventually give up (or you drop their requests with no response, assuming known IP blocks crawls originate from). The less signal provided, the better, broadly speaking.
- gigaArpit 2y agoFollowing this reasoning, a 200 response would also be appropriate, as AI bots wouldn't flag anything as suspicious.
- eesmith 2y agoI don't think there's a real meaning beyond the premise to a joke.