4 ms·

Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.

atomic128 9mo ago
Poison Fountain: https://news.ycombinator.com/item?id=46577464 https://news.ycombinator.com/item?id=46577464
yakattak 9mo ago
Do you have a link to that?
- SchemaLoad 9mo ago
  https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/
bitbasher 9mo ago
Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!
- rester324 9mo ago
  You can implement this yourself, who is stopping you?
  - zzzeek 9mo ago
    Citation needed
    - GuinansEyebrows 9mo ago
      https://forge.hackers.town/hackers.town/nepenthes https://forge.hackers.town/hackers.town/nepenthes > Citation needed this reply kinda sucks :)
    - zimpenfish 9mo ago
      I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org https://iocaine.madhouse-project.org
      - doublerabbit 9mo ago
        Unfortunately and you kind of have to count this as the cost of the Internet. You've wasted 500Mb of bandwidth. I've had colocation for eight years+. My monthly b/w cost is now around 20-30Gb a month given to scrapers where I was only be using 1-2Gb a month, years prior. I pay for premium bandwidth (it's a thing) and only get 2TB of usable data. Do I go offline or let it continue?
        zimpenfish 9mo ago
        > You've wasted 500Mb of bandwidth. Yep, it sucks, but on the positive side, I'm feeding 500Mb of garbage into them every day and that feels like enough of a small win for me. > My monthly b/w cost is now around 20-30Gb a month given to scrapers [...] 1-2Gb a month That definitely sucks. > Do I go offline or let it continue? Might be time to start blocking entire IP ranges and ASNs and see if that helps.
      - chao- 9mo ago
        Can't seem to access this. It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux? EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?
        zephen 9mo ago
        Doesn't work on my firefox either. Friendly fire, I suppose.
        godelski 9mo ago
        Works on my Firefox. Mac and Linux
    - justkys 9mo ago
      [dead]
    - godelski 9mo ago
      One of the most popular ones is Anubis. It uses a proof of work and can even do poisoning: https://anubis.techaro.lol/ https://anubis.techaro.lol/ They even mention iocaine. I know, inconceivable!: https://iocaine.madhouse-project.org/ https://iocaine.madhouse-project.org/ There's also tons of HN posts on the topic with varying solutions: https://news.ycombinator.com/item?id=45935729 https://news.ycombinator.com/item?id=45935729 https://news.ycombinator.com/item?id=45711094 https://news.ycombinator.com/item?id=45711094 https://news.ycombinator.com/item?id=44142761 https://news.ycombinator.com/item?id=44142761 https://news.ycombinator.com/item?id=44378127 https://news.ycombinator.com/item?id=44378127
      - zzzeek 9mo ago
        Anubis is the only tool that claims to have heuristics to identify a bot, but my understanding is that it does this by presenting obnoxious challenges to all users. Not really feasible. Old school approaches like ip blocking or even ASN blocking are obsolete - these crawlers purposely spam from thousands of IPs, and if you block them on a common ASN, they come back a few days later from thousands of unique ASNs. So this is not really a "roll your own" situation, especially if you are running off the shelf software that doesn't have some straightforward means of building in these various approaches of endless page mazes (which I would still have to serve anyway).
- Aurornis 9mo ago
  You don’t need to do anything. You can use any number of solutions or roll your own. Someone shared an alternative. Must everything in AI threads be so negative and condescending?
  - thefz 9mo ago
    > Must everything in AI threads be so negative and condescending? Because if I own a website or a service and it is being degraded or slowed by some third party tool that wants to slurp its content for his own profit and don't even share, I tend to be irritated. And AI apologists/evangelists don't help when they try to justify the behavior.
  - dannersy 9mo ago
    Yes, they could roll their own, but you have no issues with this being necessary? I think the attitude of "just deal with it" is far more negative than someone expressing they are upset with the state of the internet, its controllers, and its abusers.
    - TitaRusell 9mo ago
      There's trillions invested in AI. Don't expect any introspective insight or criticism about it.
    - bs7280 9mo ago
      This is like saying "lets just get rid of all the guns" to solve gun violence and gun crime in the USA. The cat is out of the bag and no one can put it back. We live in a different world now and we have to figure it out.
ranger_danger 9mo ago
Modern scrapers are using headless chromium which will not see the invisible links, so I'm not sure how long this will be effective.
timpera 9mo ago
Unfortunately, Cloudflare often destroys the experience for users with shared connections, VPNs, exotic browsers… I had to remove it from my site after too many complaints.
- rudedogg 9mo ago
  Also iCloud Private Relay. CloudFlare is making it impossible to browse privately
  - acdha 9mo ago
    Cloudflare works fine with public relay - they and Fastly provide infrastructure for that service (one half of the blinded pair) so it’s definitely something they test.
- xorcist 9mo ago
  I am sure Cloudflare would have no problem selling you a VPN service. After all, it's not very far from hosting booters and selling DoS protection.
  - sadeshmukh 9mo ago
    Well... https://developers.cloudflare.com/warp-client/warp-modes/#warp-unlimited https://developers.cloudflare.com/warp-client/warp-modes/#wa... Price is $5/mo
inferiorhuman 9mo ago
Which is still a far worse experience than if Cloudflare's services weren't needed.
RobotToaster 9mo ago
Except for the scrapers that pay cloudflare to exempt them.
themafia 9mo ago
The solution, as always, is noise.
m463 9mo ago
cloudflare also blocks my human-is-driving browser all the time "enahble javascript and cookies to continue" also unsupported browser
loopback_device 9mo ago
Not sure "TLS added and removed here :)" as a Service is the right tool in the drawer for this.
taberiand 9mo ago
Savvy move by cloudflare, once they have enough sites behind their service they can charge the AI companies to access their cached copies on a back channel

atomic128 9mo ago

Poison Fountain: https://news.ycombinator.com/item?id=46577464 https://news.ycombinator.com/item?id=46577464

yakattak 9mo ago

Do you have a link to that?

SchemaLoad 9mo ago
https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/

bitbasher 9mo ago

Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!

rester324 9mo ago
You can implement this yourself, who is stopping you?
- zzzeek 9mo ago
  Citation needed
  - GuinansEyebrows 9mo ago
    https://forge.hackers.town/hackers.town/nepenthes https://forge.hackers.town/hackers.town/nepenthes > Citation needed this reply kinda sucks :)
  - zimpenfish 9mo ago
    I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org https://iocaine.madhouse-project.org
    - doublerabbit 9mo ago
      Unfortunately and you kind of have to count this as the cost of the Internet. You've wasted 500Mb of bandwidth. I've had colocation for eight years+. My monthly b/w cost is now around 20-30Gb a month given to scrapers where I was only be using 1-2Gb a month, years prior. I pay for premium bandwidth (it's a thing) and only get 2TB of usable data. Do I go offline or let it continue?
      - zimpenfish 9mo ago
        > You've wasted 500Mb of bandwidth. Yep, it sucks, but on the positive side, I'm feeding 500Mb of garbage into them every day and that feels like enough of a small win for me. > My monthly b/w cost is now around 20-30Gb a month given to scrapers [...] 1-2Gb a month That definitely sucks. > Do I go offline or let it continue? Might be time to start blocking entire IP ranges and ASNs and see if that helps.
    - chao- 9mo ago
      Can't seem to access this. It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux? EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?
      - zephen 9mo ago
        Doesn't work on my firefox either. Friendly fire, I suppose.
        godelski 9mo ago
        Works on my Firefox. Mac and Linux
  - justkys 9mo ago
    [dead]
  - godelski 9mo ago
    One of the most popular ones is Anubis. It uses a proof of work and can even do poisoning: https://anubis.techaro.lol/ https://anubis.techaro.lol/ They even mention iocaine. I know, inconceivable!: https://iocaine.madhouse-project.org/ https://iocaine.madhouse-project.org/ There's also tons of HN posts on the topic with varying solutions: https://news.ycombinator.com/item?id=45935729 https://news.ycombinator.com/item?id=45935729 https://news.ycombinator.com/item?id=45711094 https://news.ycombinator.com/item?id=45711094 https://news.ycombinator.com/item?id=44142761 https://news.ycombinator.com/item?id=44142761 https://news.ycombinator.com/item?id=44378127 https://news.ycombinator.com/item?id=44378127
    - zzzeek 9mo ago
      Anubis is the only tool that claims to have heuristics to identify a bot, but my understanding is that it does this by presenting obnoxious challenges to all users. Not really feasible. Old school approaches like ip blocking or even ASN blocking are obsolete - these crawlers purposely spam from thousands of IPs, and if you block them on a common ASN, they come back a few days later from thousands of unique ASNs. So this is not really a "roll your own" situation, especially if you are running off the shelf software that doesn't have some straightforward means of building in these various approaches of endless page mazes (which I would still have to serve anyway).
Aurornis 9mo ago
You don’t need to do anything. You can use any number of solutions or roll your own. Someone shared an alternative. Must everything in AI threads be so negative and condescending?
- thefz 9mo ago
  > Must everything in AI threads be so negative and condescending? Because if I own a website or a service and it is being degraded or slowed by some third party tool that wants to slurp its content for his own profit and don't even share, I tend to be irritated. And AI apologists/evangelists don't help when they try to justify the behavior.
- dannersy 9mo ago
  Yes, they could roll their own, but you have no issues with this being necessary? I think the attitude of "just deal with it" is far more negative than someone expressing they are upset with the state of the internet, its controllers, and its abusers.
  - TitaRusell 9mo ago
    There's trillions invested in AI. Don't expect any introspective insight or criticism about it.
  - bs7280 9mo ago
    This is like saying "lets just get rid of all the guns" to solve gun violence and gun crime in the USA. The cat is out of the bag and no one can put it back. We live in a different world now and we have to figure it out.

ranger_danger 9mo ago

Modern scrapers are using headless chromium which will not see the invisible links, so I'm not sure how long this will be effective.

timpera 9mo ago

Unfortunately, Cloudflare often destroys the experience for users with shared connections, VPNs, exotic browsers… I had to remove it from my site after too many complaints.

rudedogg 9mo ago
Also iCloud Private Relay. CloudFlare is making it impossible to browse privately
- acdha 9mo ago
  Cloudflare works fine with public relay - they and Fastly provide infrastructure for that service (one half of the blinded pair) so it’s definitely something they test.
xorcist 9mo ago
I am sure Cloudflare would have no problem selling you a VPN service. After all, it's not very far from hosting booters and selling DoS protection.
- sadeshmukh 9mo ago
  Well... https://developers.cloudflare.com/warp-client/warp-modes/#warp-unlimited https://developers.cloudflare.com/warp-client/warp-modes/#wa... Price is $5/mo

inferiorhuman 9mo ago

Which is still a far worse experience than if Cloudflare's services weren't needed.

RobotToaster 9mo ago

Except for the scrapers that pay cloudflare to exempt them.

themafia 9mo ago

The solution, as always, is noise.

m463 9mo ago

cloudflare also blocks my human-is-driving browser all the time "enahble javascript and cookies to continue" also unsupported browser

loopback_device 9mo ago

Not sure "TLS added and removed here :)" as a Service is the right tool in the drawer for this.

taberiand 9mo ago

Savvy move by cloudflare, once they have enough sites behind their service they can charge the AI companies to access their cached copies on a back channel