3 ms·

The problem is the robots that do follow robots.txt its all the bots that don't. Robots.txt is largely irrelevant now they don't represent most of the traffic p

The problem is the robots that do follow robots.txt its all the bots that don't. Robots.txt is largely irrelevant now they don't represent most of the traffic problem. They certainly don't represent the bots that are going to hammer your site without any regard, those bots don't follow robots.txt.

_zagj 1y ago
Not sure why you were downvoted. I have zero confidence that OpenAI, Anthropic, and the rest respect robots.txt however much they insist they do. It's also clear that they're laundering their traffic through residential ISP IP addresses to make detection harder. There are plenty of third-parties advertising the service, and farming it out affords the AI companies some degree of plausible deniability.
- chneu 1y ago
  Nobody has any confidence in ai to not ddos. That's why there have been dozens of posts about how bandwidth is becoming an issue for many websites as bots continuously crawl their sites for new info. Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day. I think debian also recently spoke up about it.
- EPendragon 1y ago
  One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild west.
zarzavat 1y ago
That's what honeypots are for. Deny /honeypot in your robots.txt Add <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> to your index.html If an IP accesses that path, ban it.
- deleted 1y ago
  [deleted]
- kwar13 1y ago
  I like this. Adding now. Thanks!
- jpc0 1y ago
  > <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> Unrelated meta question but is the aria tag necessarily since display: none; should be removing the content from the flow?
  - zarzavat 1y ago
    I would hope that all screen readers would respect display:none. The aria-hidden is for CYA, banning even one blind user would be quite bad optics (as is this sentence now that I think about it).
- brewmarche 1y ago
  I wonder whether the path in the robots.txt (or maybe a <link> tag with a bogus rel attribute) would already be enough to make evil bots follow it. That would at least avoid accidental follows due to CSS/ARIA not properly working in weird constellations
- EPendragon 1y ago
  There is an interesting article about AI tarpits that addresses a similar issue: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/ https://arstechnica.com/tech-policy/2025/01/ai-haters-build-... The argument is basically to have them scrape your website indefinitely wasting their resources for the bots that decide to ignore your robots.txt (or any bot if you desire)
- fpoling 1y ago
  While this may work today, the bots now more and more use if not the full headless rendering then at least apply CSS rules not to fetch invisible content.

_zagj 1y ago

Not sure why you were downvoted. I have zero confidence that OpenAI, Anthropic, and the rest respect robots.txt however much they insist they do. It's also clear that they're laundering their traffic through residential ISP IP addresses to make detection harder. There are plenty of third-parties advertising the service, and farming it out affords the AI companies some degree of plausible deniability.

chneu 1y ago
Nobody has any confidence in ai to not ddos. That's why there have been dozens of posts about how bandwidth is becoming an issue for many websites as bots continuously crawl their sites for new info. Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day. I think debian also recently spoke up about it.
EPendragon 1y ago
One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild west.

zarzavat 1y ago

That's what honeypots are for. Deny /honeypot in your robots.txt Add <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> to your index.html If an IP accesses that path, ban it.

deleted 1y ago
[deleted]
kwar13 1y ago
I like this. Adding now. Thanks!
jpc0 1y ago
> <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> Unrelated meta question but is the aria tag necessarily since display: none; should be removing the content from the flow?
- zarzavat 1y ago
  I would hope that all screen readers would respect display:none. The aria-hidden is for CYA, banning even one blind user would be quite bad optics (as is this sentence now that I think about it).
brewmarche 1y ago
I wonder whether the path in the robots.txt (or maybe a <link> tag with a bogus rel attribute) would already be enough to make evil bots follow it. That would at least avoid accidental follows due to CSS/ARIA not properly working in weird constellations
EPendragon 1y ago
There is an interesting article about AI tarpits that addresses a similar issue: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/ https://arstechnica.com/tech-policy/2025/01/ai-haters-build-... The argument is basically to have them scrape your website indefinitely wasting their resources for the bots that decide to ignore your robots.txt (or any bot if you desire)
fpoling 1y ago
While this may work today, the bots now more and more use if not the full headless rendering then at least apply CSS rules not to fetch invisible content.