4 ms·

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, tha

by xyzal 2y ago

I think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent misalignment" for more info). A request rate too inhuman? Here, take those generated articles about positive effect of catching measles on performance in bed. And so on, and so forth ... Nepenthes is nice, but word salad can be detected easily. It needs a feature that pre-generates linguistically plausible but factically garbage text via open models

gdcbe 2y ago
That’s vaguely what https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/ is about
agilob 2y ago
Also, lower upload rate to 5kb/s
seper8 2y ago
I love this idea haha
usrnm 2y ago
Where are you going to get all that content? If it's static, it will get filtered out very fast, if it's dynamic and autogenerated, it might cost even more than just letting the crawler through
- hec126 2y ago
  Generate it once every few weeks with LLaMa and then serve as static content?
sigmoid10 2y ago
This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.
- hec126 2y ago
  You can sprinkle your site with almost-invisible hyperlinks. Bots will follow, humans will not.
  - rustc 2y ago
    This would be terrible for accessibility for users using a screen reader.
    - mostlysimilar 2y ago
      So would the site shutting down because AI bots are too much traffic.
    - MrResearcher 2y ago
      <a aria-hidden="true"> ... </a> will result in a link ignored by screen readers.
      - rustc 2y ago
        Removing elements that match `[hidden], [aria-hidden]` is the most trivial cleanup transform a crawler can do and I'm sure most crawlers already do that.
- soco 2y ago
  But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?
TonyTrapp 2y ago
We're affected by this. The only thing that would realistically work is the first suggestion. The most unscrupulous AI crawlers distribute their inhuman request rate over dozens of IPs, so every IP just makes 1-2 requests in total. And they use real-world browser user agents, so blocking those could lock out real users. However, sometimes they claim to be using really old Chrome versions, so I feel less bad about locking those out.
- sokoloff 2y ago
  > dozens of IPs, so every IP just makes 1-2 requests in total Dozens of IPs making 1-2 requests per IP hardly seems like something to spend time worrying about.
  - lucb1e 2y ago
    I'm also affected. I presume that this is per day, not just once, yet it's fewer requests than a human would often do so you cannot block it. I blocked 15 IP ranges containing 37 million IP addresses (most of them from Huawei's Singapore and mobile divisions, according to the IP address WHOIS data) because they did not respect robots.txt and didn't set a user agent identifier. This is not including several other scrapers that did set a user agent string but do not respect robots.txt (again, including Huawei's PetalBot). (Note, I've only blocked them from one specific service that proxies+caches data from a third party, which I'm caching precisely because the third party site struggled with load, so more load from these bots isn't helping) That's up to 37e6/24/60/60 = 430 requests per second if they all do 1 request per day on average. Each active IP address actually does more, some of them a few thousand per year, some of them a few dozen per year; but thankfully they don't unleash the whole IP range on me at once, it occasionally rotates through to new ranges to bypass blocks
  - TonyTrapp 2y ago
    If you are serving a Git repository browser and all of those IPs are hitting all the expensive endpoints such as git blame, it becomes something to worry about very quickly.
  - giantg2 2y ago
jajko 2y ago
Thats absolutely brilliant f*cked up idea, poisoning AI while fending them off. Gotta get my daily dose of bleach for enhanced performance, chatgpt said so.
lomonosov 2y ago
Russia is already doing poisoning with success, so it is a viable tactic! https://www.heise.de/en/news/Poisoning-training-data-Russian-propaganda-for-AI-models-10317581.html https://www.heise.de/en/news/Poisoning-training-data-Russian...
lucb1e 2y ago
> Is your user-agent too suspicious? Hello, it's me, your legitimate user who doesn't use one of the 4 main browsers. The internet gets more annoying every day on a simple android webview browser, I guess I'll have to go back to the fully-featured browser I migrated away from because it was much slower > A request rate too inhuman? I've run into this on five websites in the past 2 months, usually just from a single request that didn't have a referrer (because I clicked it from a chat, not because I block anything). When I email the owners, it's the typical "have you tried turning it off and on again" from the bigger sites, or on smaller sites "dunno but you somehow triggered our bot protection, should have expired by now [a day later] good luck using the internet further". Only one of the sites, Codeberg, actually gave a useful response and said they'd consider how to resolve when someone follows a link directly to a subpage and thus doesn't have a cookie set yet. Either way, blocks left and right are fun! More please!
- deleted 2y ago
  [deleted]
PeterStuer 2y ago
You have clearly no idea on the incompetence so many public administrations have in configuring robots.txt for data that is actually created for and specifically meant to be consumed programatically (think rss and atom feeds, REST api endpoints etc.). Half the time the person setting up the robots.txt just blankedly blocks everything, and does not even know (or care) to exclude those.
gchamonlive 2y ago
If I'm hosting my site independently, with a rented machine and a cloudflare CDN hosting my code on a self managed gitlab instance, how should I go about implementing this? Is there something plug and play I can drop into nginx that would do this work for me of serving bogus content and leaving my gitlab instance unscathed by bots?
j-bos 2y ago
This makes tons of sense, the AI trainers spend endless resources aligning their llms, the least we could do ia spend a few minutes aligning their owners. Fixing things at the incentive level.

by xyzal 2y ago

gdcbe 2y ago

That’s vaguely what https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/ is about

agilob 2y ago

Also, lower upload rate to 5kb/s

seper8 2y ago

I love this idea haha

usrnm 2y ago

Where are you going to get all that content? If it's static, it will get filtered out very fast, if it's dynamic and autogenerated, it might cost even more than just letting the crawler through

hec126 2y ago
Generate it once every few weeks with LLaMa and then serve as static content?

sigmoid10 2y ago

This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.

hec126 2y ago
You can sprinkle your site with almost-invisible hyperlinks. Bots will follow, humans will not.
- rustc 2y ago
  This would be terrible for accessibility for users using a screen reader.
  - mostlysimilar 2y ago
    So would the site shutting down because AI bots are too much traffic.
  - MrResearcher 2y ago
    <a aria-hidden="true"> ... </a> will result in a link ignored by screen readers.
    - rustc 2y ago
      Removing elements that match `[hidden], [aria-hidden]` is the most trivial cleanup transform a crawler can do and I'm sure most crawlers already do that.
soco 2y ago
But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?

TonyTrapp 2y ago

We're affected by this. The only thing that would realistically work is the first suggestion. The most unscrupulous AI crawlers distribute their inhuman request rate over dozens of IPs, so every IP just makes 1-2 requests in total. And they use real-world browser user agents, so blocking those could lock out real users. However, sometimes they claim to be using really old Chrome versions, so I feel less bad about locking those out.

sokoloff 2y ago
> dozens of IPs, so every IP just makes 1-2 requests in total Dozens of IPs making 1-2 requests per IP hardly seems like something to spend time worrying about.
- lucb1e 2y ago
  I'm also affected. I presume that this is per day, not just once, yet it's fewer requests than a human would often do so you cannot block it. I blocked 15 IP ranges containing 37 million IP addresses (most of them from Huawei's Singapore and mobile divisions, according to the IP address WHOIS data) because they did not respect robots.txt and didn't set a user agent identifier. This is not including several other scrapers that did set a user agent string but do not respect robots.txt (again, including Huawei's PetalBot). (Note, I've only blocked them from one specific service that proxies+caches data from a third party, which I'm caching precisely because the third party site struggled with load, so more load from these bots isn't helping) That's up to 37e6/24/60/60 = 430 requests per second if they all do 1 request per day on average. Each active IP address actually does more, some of them a few thousand per year, some of them a few dozen per year; but thankfully they don't unleash the whole IP range on me at once, it occasionally rotates through to new ranges to bypass blocks
- TonyTrapp 2y ago
  If you are serving a Git repository browser and all of those IPs are hitting all the expensive endpoints such as git blame, it becomes something to worry about very quickly.
- giantg2 2y ago

jajko 2y ago

Thats absolutely brilliant f*cked up idea, poisoning AI while fending them off. Gotta get my daily dose of bleach for enhanced performance, chatgpt said so.

lomonosov 2y ago

Russia is already doing poisoning with success, so it is a viable tactic! https://www.heise.de/en/news/Poisoning-training-data-Russian-propaganda-for-AI-models-10317581.html https://www.heise.de/en/news/Poisoning-training-data-Russian...

lucb1e 2y ago

> Is your user-agent too suspicious? Hello, it's me, your legitimate user who doesn't use one of the 4 main browsers. The internet gets more annoying every day on a simple android webview browser, I guess I'll have to go back to the fully-featured browser I migrated away from because it was much slower > A request rate too inhuman? I've run into this on five websites in the past 2 months, usually just from a single request that didn't have a referrer (because I clicked it from a chat, not because I block anything). When I email the owners, it's the typical "have you tried turning it off and on again" from the bigger sites, or on smaller sites "dunno but you somehow triggered our bot protection, should have expired by now [a day later] good luck using the internet further". Only one of the sites, Codeberg, actually gave a useful response and said they'd consider how to resolve when someone follows a link directly to a subpage and thus doesn't have a cookie set yet. Either way, blocks left and right are fun! More please!

deleted 2y ago
[deleted]

PeterStuer 2y ago

You have clearly no idea on the incompetence so many public administrations have in configuring robots.txt for data that is actually created for and specifically meant to be consumed programatically (think rss and atom feeds, REST api endpoints etc.). Half the time the person setting up the robots.txt just blankedly blocks everything, and does not even know (or care) to exclude those.

gchamonlive 2y ago

If I'm hosting my site independently, with a rented machine and a cloudflare CDN hosting my code on a self managed gitlab instance, how should I go about implementing this? Is there something plug and play I can drop into nginx that would do this work for me of serving bogus content and leaving my gitlab instance unscathed by bots?

j-bos 2y ago

This makes tons of sense, the AI trainers spend endless resources aligning their llms, the least we could do ia spend a few minutes aligning their owners. Fixing things at the incentive level.