3 ms·
The idea behind introducing a new HTTP status code for blocking AI traffic should be to provide a clearer signal to both users and automated systems about the n
by gigaArpit 2y ago
The idea behind introducing a new HTTP status code for blocking AI traffic should be to provide a clearer signal to both users and automated systems about the nature of the response. While a 404 indicates that a resource is not found, it doesn't convey the specific reason that the content is being restricted from AI access.
While User-Agents can easily be spoofed by users and automated traffic, as you mentioned in your earlier reply, the HTTP status code would be a clear message from the server.
- toomuchtodo 2y agoIf it’s a clear message, they will simply avoid your AI identification heuristics. Better to feed them 404s so they eventually give up (or you drop their requests with no response, assuming known IP blocks crawls originate from). The less signal provided, the better, broadly speaking.
- gigaArpit 2y agoFollowing this reasoning, a 200 response would also be appropriate, as AI bots wouldn't flag anything as suspicious.