4 ms·

"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare." I like the idea of "ai.txt" but those who eat resources rarely liste

"Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare." I like the idea of "ai.txt" but those who eat resources rarely listen to ToS. Frankly, I serve 503s to all identifiable bots, unless they are on my explicit allow list.

alwayslikethis 3y ago
Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts
- shanebellone 3y ago
  "Why not serve fake garbage indistinguishable from real content by a computer, like LLM output?" Serving more than the minimum wastes resources. Worse yet, a better solution would cost my time. "Sending errors just incentivizes bot owners to fix the identifiable parts" Sure, someone could make or configure their scraper perfectly. "Perfect" is now the table stakes though. Edit: My solution strives to cause an unproportional expense in order to circumvent. I want 10x on my time.
- twelve40 3y ago
  it'd be cool to be able to fingerprint that garbage, too. Like, sprinkle some hashes here and there (or something like that) so that you can later uniquely look up your own "content" being stolen by chatbots and which ones.
  - shanebellone 3y ago
    You can. I can't think of the appropriate term though. Hopefully someone else chimes in here with a link.
- ape4 3y ago
  I like this idea. Of course it would have to be only to robots that visit a page disallowed by the robots.txt
- dspillett 3y ago
  > Sending errors just incentivizes bot owners to fix the identifiable parts Nah. It'll just make them fake their identity so it is harder to tell the traffic is from a bot.
spc476 3y ago
It might be a better idea to serve up a 418 ("I'm a tea pot") with a line line text file saying "I'm not an HTTP server". That solved a problem I had with bots making HTTP requests to my gopher server [1]. Serving up a 503 informs the bot that there's a server issue and it may try again later. A 418 informs the bot that it made an erroneous request and such an odd error code might get someone to look into it and stop. [1] https://boston.conman.org/2019/09/30.2 https://boston.conman.org/2019/09/30.2
- shanebellone 3y ago
  This is very interesting. I've bookmarked the link. Thanks for sharing. I believe minimal is best and this might fit nicely within my larger system. Do you approach other problems with a similar mindset?

alwayslikethis 3y ago

Why not serve fake garbage indistinguishable from real content by a computer, like LLM output? Sending errors just incentivizes bot owners to fix the identifiable parts

shanebellone 3y ago
"Why not serve fake garbage indistinguishable from real content by a computer, like LLM output?" Serving more than the minimum wastes resources. Worse yet, a better solution would cost my time. "Sending errors just incentivizes bot owners to fix the identifiable parts" Sure, someone could make or configure their scraper perfectly. "Perfect" is now the table stakes though. Edit: My solution strives to cause an unproportional expense in order to circumvent. I want 10x on my time.
twelve40 3y ago
it'd be cool to be able to fingerprint that garbage, too. Like, sprinkle some hashes here and there (or something like that) so that you can later uniquely look up your own "content" being stolen by chatbots and which ones.
- shanebellone 3y ago
  You can. I can't think of the appropriate term though. Hopefully someone else chimes in here with a link.
ape4 3y ago
I like this idea. Of course it would have to be only to robots that visit a page disallowed by the robots.txt
dspillett 3y ago
> Sending errors just incentivizes bot owners to fix the identifiable parts Nah. It'll just make them fake their identity so it is harder to tell the traffic is from a bot.

spc476 3y ago

It might be a better idea to serve up a 418 ("I'm a tea pot") with a line line text file saying "I'm not an HTTP server". That solved a problem I had with bots making HTTP requests to my gopher server [1]. Serving up a 503 informs the bot that there's a server issue and it may try again later. A 418 informs the bot that it made an erroneous request and such an odd error code might get someone to look into it and stop. [1] https://boston.conman.org/2019/09/30.2 https://boston.conman.org/2019/09/30.2

shanebellone 3y ago
This is very interesting. I've bookmarked the link. Thanks for sharing. I believe minimal is best and this might fit nicely within my larger system. Do you approach other problems with a similar mindset?