7 ms·

OpenAI's bot crushed this seven-person company's web site 'like a DDoS attack'

121 points by vednig 2y ago

ThrowawayTestr 2y ago
Has anyone been successfully sued for excess hosting costs due to scraping?
- neom 2y ago
  https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn makes it clear scraping publicly available data is generally not a CFAA violation. Certainly it would have to be a civil matter, but I doubt it would work (ianal)
  - ericholscher 2y ago
    We did get $7k out of one of the AI companies based on the massive bandwidth costs they caused us. https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...
    - neom 2y ago
      wow GOOD JOB!!! Were they relatively decent about it, is that why? I feel like normal businesses that are not super shady should be able to accept this kind of conversation and deal with the mistake and issue they causes for you. Good job perusing it tho, that's fantastic. (ps, big fan of your product, great work on that too!)
atleastoptimal 2y ago
Stuff like this will happen to all websites soon due to AI agents let loose on the web
- JohnMakin 2y ago
  https://cyberpunk.fandom.com/wiki/Blackwall https://cyberpunk.fandom.com/wiki/Blackwall
peterldowns 2y ago
I have little sympathy for the company in this article. If you put your content on the web, and don't require authentication to access it, it's going to be crawled and scraped. Most of the time you're happy about this — you want search providers to index your content. It's one thing if a company ignores robots.txt and causes serious interference with the service, like Perplexity was, but the details here don't really add up: this company didn't have a robots.txt in place, and although the article mentions tens/hundreds of thousands of requests, they don't say anything about them being made unreasonably quickly. The default-public accessibility of information on the internet is a net-good for the technology ecosystem. Want to host things online? Learn how. EDIT: They're a very media-heavy website. Here's one of the product pages from their catalog: https://triplegangers.com/browse/scans/full-body/sara-liang-body https://triplegangers.com/browse/scans/full-body/sara-liang-.... Each of the body-pose images is displayed at about 35x70px but is served as a 500x1000px image. It now seems like they have some cloudflare caching in place at least. I stand by my belief that unless we get some evidence that they were being scraped particularly aggressively, this is on them, and this is being blown out of proportion for publicity.
- JohnMakin 2y ago
  robots.txt as of right now is a complete honor system, so I think it's reasonable to make a conclusion that you shouldn't rely on it protecting you because odds are overwhelming that scraping behavior will become worse in the near to mid term future
- j45 2y ago
  It's less about sympathy and more about understanding that they might not be experts in things tech, relied on hired help that seemed to be good at what they did, and the most basic thing (setup a free cloudflare account or something) was missed. Learning how, is sometimes actually learning who's going to get you online in a good way. In this case when you have non-tech people building Wordpress sites, it's about what they can understand and do, and teh rate of learning doesn't always keep up relative to client work.
- fzeroracer 2y ago
  > If you put your content on the web, and don't require authentication to access it, it's going to be crawled and scraped. Most of the time you're happy about this — you want search providers to index your content > The default-public accessibility of information on the internet is a net-good for the technology ecosystem. Want to host things online? Learn how. These two statements are at odds, I hope you realize. You say public accessibility of information is a good thing, while blaming someone for being effectively DDOS'd as a result of having said information public.
ericholscher 2y ago
This keeps happening -- we wrote about multiple AI bots that were hammering us over at Read the Docs for >10TB of traffic: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... They really are trying to burn all their goodwill to the ground with this stuff.
- exe34 2y ago
  can you feed them gibberish?
  - Groxx 2y ago
    LLMs poisoned by https://git-man-page-generator.lokaltog.net/ https://git-man-page-generator.lokaltog.net/ -like content would be a hilarious end result, please do!
  - jcpham2 2y ago
    This would be my elegant solution, something like an endless recursion with a gzip bomb at the end if I can identify your crawler and it’s that abusive. Would it be possible to feed an abusing crawler nothing but my own locally-hosted LLM gibberish? But then again if you’re in the cloud egress bandwidth is going to cost for playing this game. Better to just deny the OpenAI crawler and send them an invoice for the money and time they’ve wasted. Interesting form of data warfare against competitors and non competitors alike. The winner will have the longest runway
    - actsasbuffoon 2y ago
      It wouldn’t even necessarily need to be a real GZip bomb. Just something containing a few hundred kb of seemingly new and unique text that’s highly compressible and keeps providing “links” to additional dynamically generated gibberish that can be crawled. The idea is to serve a vast amount of poisoned training data as cheaply as possible. Heck, maybe you could even make a plugin for NGINX to recognize abusive AI bots and do this. If enough people install it then you could provide some very strong disincentives.
griomnib 2y ago
I’ve been a web developer for decades as well as doing scraping, indexing, and analyzing million of sites. Just follow the golden rule: don’t ever load any site more aggressively than you would want yours to be. This isn’t hard stuff, and these AI companies have grossly inefficient and obnoxious scrapers. As a site owner those pisses me off as a matter of decency on the web, but as an engineer doing distributed data collection I’m offended by how shitty and inefficient their crawlers are.
- PaulHoule 2y ago
  I worked at one place where it probably cost us 100x (in CPU) more to serve content the way we were doing it as opposed to the way most people would do it. We could afford it because it was still cheap, but we deferred the cost reduction work for half a decade and went on a war against webcrawlers instead. (hint: who introduced the robots.txt standard?)
- mingabunga 2y ago
  We've had to block a lot of these bots as they slowed our technical forum to a crawl, but new ones appear every now and again. Amazons was the worst
  - griomnib 2y ago
    I really wonder if these dogshit scrapers are wholly built by LLM. Nobody competent codes like this.
- add-sub-mul-div 2y ago
  These people think they're on the verge of the most important invention in modern history. Etiquette means nothing to them. They would probably consider an impediment to their work a harm to the human race.
  - krapp 2y ago
    >They would probably consider an impediment to their work a harm to the human race. They do. Marc Andreeson said as much in his "techno-optimist manifesto," that any hesitation or slowdown in AI development or adoption is equivalent to mass murder.
PaulHoule 2y ago
First time I heard this story it was '98 or so and the perp was somebody in the overfunded CS department and the victim somebody in the underfunded math department on the other side of a short and fat pipe. (Probably running Apache httpd on a SGI workstation without enough ram to even run Win '95) In years of running webcrawlers I've had very little trouble, I've had more trouble in the last year than in the past 25. (Wrote my first crawler in '99, funny my crawlers have gotten simpler over time not more complex) In one case I found a site got terribly slow although I was hitting it at much less than 1 request per second. Careful observation showed the wheels were coming off the site and it had nothing to do with me. There's another site that I've probably crawled in it's entirety at least ten times over the past twenty years. I have a crawl from two years ago, my plan was to feed it into a BERT-based system not for training but to discover content that is like the content that I like. I thought I'd get a fresh copy w/ httrack (polite, respects robots.txt, ...) and they blocked both my home IP addresses in 10 minutes. (Granted I don't think the past 2 years of this site was as good as the past, so I will just load what I have into my semantic search & tagging system and use that instead) I was angry about how unfair the Google Economy was in 2013, in lines with what this blogger has been saying ever since http://www.seobook.com/blog http://www.seobook.com/blog (I can say it's a strange way to market an expensive SEO community but...) and it drives me up the wall that people looking in the rear view mirror are getting upset about it now. Back in '98 I was excited about "personal webcrawlers" that could be your own web agent. On one hand LLMs could give so much utility in terms of classification, extraction, clustering and otherwise drinking from that firehose but the fear that somebody is stealing their precious creativity is going to close the door forever... And entrench a completely unfair Google Economy. It makes me sad. ---- Oddly those stupid ReCAPTCHAs and Cloudflare CAPTCHAs torment me all the time as a human but I haven't once had them get in the way of a crawling project.
peebee67 2y ago
Greedy and relentless OpenAI's scraping may be, but that his web-based startup didn't have a rudimentary robots.txt in place seems inexcusably naive. Correctly configuring this file has been one of the most basic steps of web design for living memory and doesn't speak highly of the technical acumen of this company. >“We’re in a business where the rights are kind of a serious issue, because we scan actual people,” he said. With laws like Europe’s GDPR, “they cannot just take a photo of anyone on the web and use it.” Yes, and protecting that data was your responsibility, Tomchuck. You dropped the ball and are now trying to blame the other players.
- mystified5016 2y ago
  OpenAI will happily ignore robots.txt Or is that still my fault somehow? Maybe we should stop blaming people for "letting" themselves get destroyed and maybe put some blame on the people actively choosing to behave in a way that harms everyone else? But then again, they have so much money so we should all just bend over and take it, right?
  - peebee67 2y ago
    If they ignore a properly configured robots.txt and the licence also explicitly denies them use, then I'd guess they have a viable civil action to extract compensation. But that isn't the case here at all, and while there's reports of them doing so, they certainly claim to respect the convention. As for bending over, if you serve files and they request files, then you send them files, what exactly is the problem? That you didn't implement any kind of rate limiting? It's a web-based company and these things are just the basics.
ldehaan 2y ago
Fail2ban would have been all they needed. It's what I have, works great.
vzaliva 2y ago
From the article: "As Tomchuk experienced, if a site isn’t properly using robot.txt, OpenAI and others take that to mean they can scrape to their hearts’ content." The takeaway: check your robots.txt. The question of how much load requests robots can reasonably generate when allowed is a separate matter.
- krapp 2y ago
  Also probably consider blocking them with .htaccess or your server's equivalent, such as here: https://ethanmarcotte.com/wrote/blockin-bots/ https://ethanmarcotte.com/wrote/blockin-bots/ All this effort is futile because AI bots will simply send false user agents, but it's something.
  - Sesse__ 2y ago
    I took my most bothered page IPv6-only, the AI bots vanished in the course of a couple days :-) (Hardly any complaints from actual users yet. Not zero, though.)
more_corn 2y ago
Cloudflare bot detection? https://developers.cloudflare.com/bots/plans/free/ https://developers.cloudflare.com/bots/plans/free/
- 1oooqooq 2y ago
  they are problably hosting the bots
andrethegiant 2y ago
I'm working on fixing this exact problem[1]. Crawlers are gonna keep crawling no matter what, so a solution to meet them where they are is to create a centralized platform that builds in an edge TTL cache, respects robots.txt and retry-after headers out of the box, etc. If there is a convenient and affordable solution that plays nicely with websites, the hope is that devs will gravitate towards the well-behaved solution. [1] https://crawlspace.dev https://crawlspace.dev
1oooqooq 2y ago
is there a place with a list of aws servers these companies can block?
- peterldowns 2y ago
  OpenAI publishes a list of IP prefixes that its scraper uses, as well as the user-agents: https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots It does not document how the bot will respond to things like 429 response codes. It does not document how frequently their bots may crawl a given domain/url. It does not document any way to get in touch about abusive crawler practices.
  - layer8 2y ago
    From https://pod.geraspora.de/posts/17342163 https://pod.geraspora.de/posts/17342163 (discussed in https://news.ycombinator.com/item?id=42549624 https://news.ycombinator.com/item?id=42549624): “If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).”
OutOfHere 2y ago
Sites should learn to use HTTP error 429 to slow down bots to a reasonable pace. If the bots are coming from a subnet, apply it to the subnet, not to the individual IP. No other action is needed.
- Sesse__ 2y ago
  I've seen _plenty_ of user agents that respond to 429 by immediately trying again. Like, literally immediately; full hammer. I had to eventually automatically blackhole IP addresses that got 429 too often.
  - jcgrillo 2y ago
    It seems like it should be pretty cheap to detect violations of Retry-After on a 429 and just automatically blackhole that IP for idk 1hr. It could also be an interesting dataset for exposing the IPs those shady "anonymous scraping" comp intel companies use..
  - OutOfHere 2y ago
    That is just what a bot does by default. It will almost always give up after a few retries. The point of 429 is that you will not be using up your limited bandwidth sending the actual response, which will save you at least 99% of your bandwidth quota. It is not to find IPs to block, especially if the requestor gives up after a few requests. The IPs that you actually need to block are the ones that are actually DoSing you without stopping even after a few retries, and even then only temporarily.
    - Sesse__ 2y ago
      > That is just what a bot does by default. It will almost always give up after a few retries. I've had them go on for hours. (This is not mainly bots, but various crap desktop applications.) > The IPs that you actually need to block are the ones that are actually DoSing you without stopping even after a few retries, and even then only temporarily. Yup. They only get blocked until next reboot :-)
joelkoen 2y ago
> “OpenAI used 600 IPs to scrape data, and we are still analyzing logs from last week, perhaps it’s way more,” he said of the IP addresses the bot used to attempt to consume his site. The IP addresses in the screenshot are all owned by Cloudflare, meaning that their server logs are only recording the IPs of Cloudflare's reverse proxy, not the real client IPs. Also, the logs don't show any timestamps and there doesn't seem to be any mention of the request rate in the whole article. I'm not trying to defend OpenAI but as someone who scrapes data I think it's unfair to throw around terms "like DDOS attack" without providing basic request rate metrics. This seems to be purely based on the use of multiple IPs, which was actually caused by their own server configuration and has nothing to do with OpenAI.
- mvdtnz 2y ago
  Why should web store operators have to be so sophisticated to use the exact right technical language in order to have a legitimate grievance? How about this: these folks put up a website in order to serve customers, not for OpenAI to scoop up all their data for their own benefit. In my opinion data should only be made available to "AI" companies on an opt-in basis, but given today's reality OpenAI should at least be polite about how they harvest data.
jonas21 2y ago
It's "robots.txt", not "robot.txt". I'm not just nitpicking -- it's a clear signal the journalist has no idea what they're talking about. That and the fact that they're using a log file with the timestamps omitted as evidence of "how ruthelessly an OpenAI bot was accessing the site" makes the claims in the article a bit suspect. OpenAI isn't necessarily in the clear here, but this is a low-quality article that doesn't provide much signal either way.
- peterldowns 2y ago
  Well said, I agree with you.
  - ted_bunny 2y ago
    I wish they'd add a little arrow you could click.
    - peterldowns 2y ago
      Well said, I agree with you.
- Thoreandan 2y ago
  Hear hear. Poor article going out the door for publication with zero editorial checking.
  - joelkoen 2y ago
    Haha yeah just noticed they call Bytespider "TokTok's crawler" too
- ted_bunny 2y ago
  The best way to tell a journalist doesn't know their subject matter: check if they're a journalist.
dang 2y ago
Recent and related: AI companies cause most of traffic on forums - https://news.ycombinator.com/item?id=42549624 https://news.ycombinator.com/item?id=42549624 - Dec 2024 (438 comments)
methou 2y ago
I used to have some problem with some Chinese crawlers, first I told them no with robots.txt, then I see a swarm of of non-bot user-agents from cloud providers in China, so I blocked their ASN, and then I see another rise of IPs from some Chinese ISP, so I eventually I have to block the entire country_code = cn and show them a robots.txt
Hilift 2y ago
People who have published books recently on Amazon have noticed that immediately there are fraud knockoff copies with the title slightly changed. These are created by AI, and are competing with humans. A person this happened to was recently interviewed about their experience on BBC.
spwa4 2y ago
It's funny how history repeats. The web originally grew because it was a way to get "an API" into a company. You could get information, without a phone call. Then, with forms and credit cards and eventually with actual API's, you could get information, you could get companies to do stuff via an API. For a short while this was possible. Now everybody calls this abuse. And a lot of it is abuse, to be fair. Now that has been mostly blocked. Every website tries really hard to block bots (and mostly fail because Google funds their crawler millions of dollars while companies raise a stink over paying a single SWE), but it's still at the point that automated interactions with companies (through third-party services for example) are not really possible. I cannot give my credit card info to a company and have it order my favorite foods to my home every day, for example. What AI promises, in a way, is to re-enable this. Because AI bots are unblockable (they're more human than humans as far as these tests are concerned). For companies, and for users. And that would be a way to ... put API's into people and companies again. Back to step 1.
- afavour 2y ago
  I see it as different history repeating: VC capital inserting itself as the middleman between people and things they want. If all of our interactions with external web sites now go through ChatGPT that gives OpenAI a phenomenal amount of power. Just like Google did with search.
  - spwa4 2y ago
    Well, it's not just that. Every company insists on doing things differently and usually in annoying ways. Having a way to deal with companies while avoiding their internal policies (e.g. upselling, "retention team", ...) would be very nice. Yes, VCs want this because it's an opportunity for a double-sided marketplace, but I still want it too. I wonder to what extent is what these FANG businesses want with AI can be described as just "an API into businesses that don't want to provide an API".
readyplayernull 2y ago
> has a terms of service page on its site that forbids bots from taking its images without permission. But that alone did nothing. It's time to level up in this arms race. Let's stop delivering html documents, use animated rendering of information that is positioned in a scene so that the user has to move elements around for it to be recognizable, like a full site captcha. It doesn't need to be overly complex for the user that can intuitively navigate even a 3D world, but will take x1000 more processing for OpenAI. Feel free to come up with your creative designs to make automation more difficult.
- liamwire 2y ago
  Sounds entirely at odds with any accessibility requirements.
  - readyplayernull 2y ago
    Are 3D videogames at odds with accessibility?
tonetegeatinst 2y ago
What options exist if you want to handle this traffic and you own your hardware on prem? It seems that any router or switch over 100G is extremely expensive, and often requires some paid for OS. The pro move would be to not block these bots. Well I guess block them if you truly can't handle their throughput request (would an ASN blacklist work?) Or if you want to force them to slow down, start sending data, but only a random % of responses are sent (so say ignore 85% of the traffic they spam you with, and reply to the others at a super low rate or you could purposely send bad data) Or perhaps reachout to your peering partners and talk about traffic shaping these requests.
philipwhiuk 2y ago
I had the same problem with my club's website.
101008 2y ago
Cloudflare should provide a service (paid or free) to block AI crawlers.
- jgrahamc 2y ago
  We do: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...

121 points by vednig 2y ago

ThrowawayTestr 2y ago

Has anyone been successfully sued for excess hosting costs due to scraping?

neom 2y ago
https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn makes it clear scraping publicly available data is generally not a CFAA violation. Certainly it would have to be a civil matter, but I doubt it would work (ianal)
- ericholscher 2y ago
  We did get $7k out of one of the AI companies based on the massive bandwidth costs they caused us. https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...
  - neom 2y ago
    wow GOOD JOB!!! Were they relatively decent about it, is that why? I feel like normal businesses that are not super shady should be able to accept this kind of conversation and deal with the mistake and issue they causes for you. Good job perusing it tho, that's fantastic. (ps, big fan of your product, great work on that too!)

atleastoptimal 2y ago

Stuff like this will happen to all websites soon due to AI agents let loose on the web

JohnMakin 2y ago
https://cyberpunk.fandom.com/wiki/Blackwall https://cyberpunk.fandom.com/wiki/Blackwall

peterldowns 2y ago

I have little sympathy for the company in this article. If you put your content on the web, and don't require authentication to access it, it's going to be crawled and scraped. Most of the time you're happy about this — you want search providers to index your content. It's one thing if a company ignores robots.txt and causes serious interference with the service, like Perplexity was, but the details here don't really add up: this company didn't have a robots.txt in place, and although the article mentions tens/hundreds of thousands of requests, they don't say anything about them being made unreasonably quickly. The default-public accessibility of information on the internet is a net-good for the technology ecosystem. Want to host things online? Learn how. EDIT: They're a very media-heavy website. Here's one of the product pages from their catalog: https://triplegangers.com/browse/scans/full-body/sara-liang-body https://triplegangers.com/browse/scans/full-body/sara-liang-.... Each of the body-pose images is displayed at about 35x70px but is served as a 500x1000px image. It now seems like they have some cloudflare caching in place at least. I stand by my belief that unless we get some evidence that they were being scraped particularly aggressively, this is on them, and this is being blown out of proportion for publicity.

JohnMakin 2y ago
robots.txt as of right now is a complete honor system, so I think it's reasonable to make a conclusion that you shouldn't rely on it protecting you because odds are overwhelming that scraping behavior will become worse in the near to mid term future
j45 2y ago
It's less about sympathy and more about understanding that they might not be experts in things tech, relied on hired help that seemed to be good at what they did, and the most basic thing (setup a free cloudflare account or something) was missed. Learning how, is sometimes actually learning who's going to get you online in a good way. In this case when you have non-tech people building Wordpress sites, it's about what they can understand and do, and teh rate of learning doesn't always keep up relative to client work.
fzeroracer 2y ago
> If you put your content on the web, and don't require authentication to access it, it's going to be crawled and scraped. Most of the time you're happy about this — you want search providers to index your content > The default-public accessibility of information on the internet is a net-good for the technology ecosystem. Want to host things online? Learn how. These two statements are at odds, I hope you realize. You say public accessibility of information is a good thing, while blaming someone for being effectively DDOS'd as a result of having said information public.

ericholscher 2y ago

This keeps happening -- we wrote about multiple AI bots that were hammering us over at Read the Docs for >10TB of traffic: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... They really are trying to burn all their goodwill to the ground with this stuff.

exe34 2y ago
can you feed them gibberish?
- Groxx 2y ago
  LLMs poisoned by https://git-man-page-generator.lokaltog.net/ https://git-man-page-generator.lokaltog.net/ -like content would be a hilarious end result, please do!
- jcpham2 2y ago
  This would be my elegant solution, something like an endless recursion with a gzip bomb at the end if I can identify your crawler and it’s that abusive. Would it be possible to feed an abusing crawler nothing but my own locally-hosted LLM gibberish? But then again if you’re in the cloud egress bandwidth is going to cost for playing this game. Better to just deny the OpenAI crawler and send them an invoice for the money and time they’ve wasted. Interesting form of data warfare against competitors and non competitors alike. The winner will have the longest runway
  - actsasbuffoon 2y ago
    It wouldn’t even necessarily need to be a real GZip bomb. Just something containing a few hundred kb of seemingly new and unique text that’s highly compressible and keeps providing “links” to additional dynamically generated gibberish that can be crawled. The idea is to serve a vast amount of poisoned training data as cheaply as possible. Heck, maybe you could even make a plugin for NGINX to recognize abusive AI bots and do this. If enough people install it then you could provide some very strong disincentives.

griomnib 2y ago

I’ve been a web developer for decades as well as doing scraping, indexing, and analyzing million of sites. Just follow the golden rule: don’t ever load any site more aggressively than you would want yours to be. This isn’t hard stuff, and these AI companies have grossly inefficient and obnoxious scrapers. As a site owner those pisses me off as a matter of decency on the web, but as an engineer doing distributed data collection I’m offended by how shitty and inefficient their crawlers are.

PaulHoule 2y ago
I worked at one place where it probably cost us 100x (in CPU) more to serve content the way we were doing it as opposed to the way most people would do it. We could afford it because it was still cheap, but we deferred the cost reduction work for half a decade and went on a war against webcrawlers instead. (hint: who introduced the robots.txt standard?)
mingabunga 2y ago
We've had to block a lot of these bots as they slowed our technical forum to a crawl, but new ones appear every now and again. Amazons was the worst
- griomnib 2y ago
  I really wonder if these dogshit scrapers are wholly built by LLM. Nobody competent codes like this.
add-sub-mul-div 2y ago
These people think they're on the verge of the most important invention in modern history. Etiquette means nothing to them. They would probably consider an impediment to their work a harm to the human race.
- krapp 2y ago
  >They would probably consider an impediment to their work a harm to the human race. They do. Marc Andreeson said as much in his "techno-optimist manifesto," that any hesitation or slowdown in AI development or adoption is equivalent to mass murder.

PaulHoule 2y ago

First time I heard this story it was '98 or so and the perp was somebody in the overfunded CS department and the victim somebody in the underfunded math department on the other side of a short and fat pipe. (Probably running Apache httpd on a SGI workstation without enough ram to even run Win '95) In years of running webcrawlers I've had very little trouble, I've had more trouble in the last year than in the past 25. (Wrote my first crawler in '99, funny my crawlers have gotten simpler over time not more complex) In one case I found a site got terribly slow although I was hitting it at much less than 1 request per second. Careful observation showed the wheels were coming off the site and it had nothing to do with me. There's another site that I've probably crawled in it's entirety at least ten times over the past twenty years. I have a crawl from two years ago, my plan was to feed it into a BERT-based system not for training but to discover content that is like the content that I like. I thought I'd get a fresh copy w/ httrack (polite, respects robots.txt, ...) and they blocked both my home IP addresses in 10 minutes. (Granted I don't think the past 2 years of this site was as good as the past, so I will just load what I have into my semantic search & tagging system and use that instead) I was angry about how unfair the Google Economy was in 2013, in lines with what this blogger has been saying ever since http://www.seobook.com/blog http://www.seobook.com/blog (I can say it's a strange way to market an expensive SEO community but...) and it drives me up the wall that people looking in the rear view mirror are getting upset about it now. Back in '98 I was excited about "personal webcrawlers" that could be your own web agent. On one hand LLMs could give so much utility in terms of classification, extraction, clustering and otherwise drinking from that firehose but the fear that somebody is stealing their precious creativity is going to close the door forever... And entrench a completely unfair Google Economy. It makes me sad. ---- Oddly those stupid ReCAPTCHAs and Cloudflare CAPTCHAs torment me all the time as a human but I haven't once had them get in the way of a crawling project.

peebee67 2y ago

Greedy and relentless OpenAI's scraping may be, but that his web-based startup didn't have a rudimentary robots.txt in place seems inexcusably naive. Correctly configuring this file has been one of the most basic steps of web design for living memory and doesn't speak highly of the technical acumen of this company. >“We’re in a business where the rights are kind of a serious issue, because we scan actual people,” he said. With laws like Europe’s GDPR, “they cannot just take a photo of anyone on the web and use it.” Yes, and protecting that data was your responsibility, Tomchuck. You dropped the ball and are now trying to blame the other players.

mystified5016 2y ago
OpenAI will happily ignore robots.txt Or is that still my fault somehow? Maybe we should stop blaming people for "letting" themselves get destroyed and maybe put some blame on the people actively choosing to behave in a way that harms everyone else? But then again, they have so much money so we should all just bend over and take it, right?
- peebee67 2y ago
  If they ignore a properly configured robots.txt and the licence also explicitly denies them use, then I'd guess they have a viable civil action to extract compensation. But that isn't the case here at all, and while there's reports of them doing so, they certainly claim to respect the convention. As for bending over, if you serve files and they request files, then you send them files, what exactly is the problem? That you didn't implement any kind of rate limiting? It's a web-based company and these things are just the basics.

ldehaan 2y ago

Fail2ban would have been all they needed. It's what I have, works great.

vzaliva 2y ago

From the article: "As Tomchuk experienced, if a site isn’t properly using robot.txt, OpenAI and others take that to mean they can scrape to their hearts’ content." The takeaway: check your robots.txt. The question of how much load requests robots can reasonably generate when allowed is a separate matter.

krapp 2y ago
Also probably consider blocking them with .htaccess or your server's equivalent, such as here: https://ethanmarcotte.com/wrote/blockin-bots/ https://ethanmarcotte.com/wrote/blockin-bots/ All this effort is futile because AI bots will simply send false user agents, but it's something.
- Sesse__ 2y ago
  I took my most bothered page IPv6-only, the AI bots vanished in the course of a couple days :-) (Hardly any complaints from actual users yet. Not zero, though.)

more_corn 2y ago

Cloudflare bot detection? https://developers.cloudflare.com/bots/plans/free/ https://developers.cloudflare.com/bots/plans/free/

1oooqooq 2y ago
they are problably hosting the bots

andrethegiant 2y ago

I'm working on fixing this exact problem[1]. Crawlers are gonna keep crawling no matter what, so a solution to meet them where they are is to create a centralized platform that builds in an edge TTL cache, respects robots.txt and retry-after headers out of the box, etc. If there is a convenient and affordable solution that plays nicely with websites, the hope is that devs will gravitate towards the well-behaved solution. [1] https://crawlspace.dev https://crawlspace.dev

1oooqooq 2y ago

is there a place with a list of aws servers these companies can block?

peterldowns 2y ago
OpenAI publishes a list of IP prefixes that its scraper uses, as well as the user-agents: https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots It does not document how the bot will respond to things like 429 response codes. It does not document how frequently their bots may crawl a given domain/url. It does not document any way to get in touch about abusive crawler practices.
- layer8 2y ago
  From https://pod.geraspora.de/posts/17342163 https://pod.geraspora.de/posts/17342163 (discussed in https://news.ycombinator.com/item?id=42549624 https://news.ycombinator.com/item?id=42549624): “If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).”

OutOfHere 2y ago

Sites should learn to use HTTP error 429 to slow down bots to a reasonable pace. If the bots are coming from a subnet, apply it to the subnet, not to the individual IP. No other action is needed.

Sesse__ 2y ago
I've seen _plenty_ of user agents that respond to 429 by immediately trying again. Like, literally immediately; full hammer. I had to eventually automatically blackhole IP addresses that got 429 too often.
- jcgrillo 2y ago
  It seems like it should be pretty cheap to detect violations of Retry-After on a 429 and just automatically blackhole that IP for idk 1hr. It could also be an interesting dataset for exposing the IPs those shady "anonymous scraping" comp intel companies use..
- OutOfHere 2y ago
  That is just what a bot does by default. It will almost always give up after a few retries. The point of 429 is that you will not be using up your limited bandwidth sending the actual response, which will save you at least 99% of your bandwidth quota. It is not to find IPs to block, especially if the requestor gives up after a few requests. The IPs that you actually need to block are the ones that are actually DoSing you without stopping even after a few retries, and even then only temporarily.
  - Sesse__ 2y ago
    > That is just what a bot does by default. It will almost always give up after a few retries. I've had them go on for hours. (This is not mainly bots, but various crap desktop applications.) > The IPs that you actually need to block are the ones that are actually DoSing you without stopping even after a few retries, and even then only temporarily. Yup. They only get blocked until next reboot :-)

joelkoen 2y ago

> “OpenAI used 600 IPs to scrape data, and we are still analyzing logs from last week, perhaps it’s way more,” he said of the IP addresses the bot used to attempt to consume his site. The IP addresses in the screenshot are all owned by Cloudflare, meaning that their server logs are only recording the IPs of Cloudflare's reverse proxy, not the real client IPs. Also, the logs don't show any timestamps and there doesn't seem to be any mention of the request rate in the whole article. I'm not trying to defend OpenAI but as someone who scrapes data I think it's unfair to throw around terms "like DDOS attack" without providing basic request rate metrics. This seems to be purely based on the use of multiple IPs, which was actually caused by their own server configuration and has nothing to do with OpenAI.

mvdtnz 2y ago
Why should web store operators have to be so sophisticated to use the exact right technical language in order to have a legitimate grievance? How about this: these folks put up a website in order to serve customers, not for OpenAI to scoop up all their data for their own benefit. In my opinion data should only be made available to "AI" companies on an opt-in basis, but given today's reality OpenAI should at least be polite about how they harvest data.

jonas21 2y ago

It's "robots.txt", not "robot.txt". I'm not just nitpicking -- it's a clear signal the journalist has no idea what they're talking about. That and the fact that they're using a log file with the timestamps omitted as evidence of "how ruthelessly an OpenAI bot was accessing the site" makes the claims in the article a bit suspect. OpenAI isn't necessarily in the clear here, but this is a low-quality article that doesn't provide much signal either way.

peterldowns 2y ago
Well said, I agree with you.
- ted_bunny 2y ago
  I wish they'd add a little arrow you could click.
  - peterldowns 2y ago
    Well said, I agree with you.
Thoreandan 2y ago
Hear hear. Poor article going out the door for publication with zero editorial checking.
- joelkoen 2y ago
  Haha yeah just noticed they call Bytespider "TokTok's crawler" too
ted_bunny 2y ago
The best way to tell a journalist doesn't know their subject matter: check if they're a journalist.

dang 2y ago

Recent and related: AI companies cause most of traffic on forums - https://news.ycombinator.com/item?id=42549624 https://news.ycombinator.com/item?id=42549624 - Dec 2024 (438 comments)

methou 2y ago

I used to have some problem with some Chinese crawlers, first I told them no with robots.txt, then I see a swarm of of non-bot user-agents from cloud providers in China, so I blocked their ASN, and then I see another rise of IPs from some Chinese ISP, so I eventually I have to block the entire country_code = cn and show them a robots.txt

Hilift 2y ago

People who have published books recently on Amazon have noticed that immediately there are fraud knockoff copies with the title slightly changed. These are created by AI, and are competing with humans. A person this happened to was recently interviewed about their experience on BBC.

spwa4 2y ago

It's funny how history repeats. The web originally grew because it was a way to get "an API" into a company. You could get information, without a phone call. Then, with forms and credit cards and eventually with actual API's, you could get information, you could get companies to do stuff via an API. For a short while this was possible. Now everybody calls this abuse. And a lot of it is abuse, to be fair. Now that has been mostly blocked. Every website tries really hard to block bots (and mostly fail because Google funds their crawler millions of dollars while companies raise a stink over paying a single SWE), but it's still at the point that automated interactions with companies (through third-party services for example) are not really possible. I cannot give my credit card info to a company and have it order my favorite foods to my home every day, for example. What AI promises, in a way, is to re-enable this. Because AI bots are unblockable (they're more human than humans as far as these tests are concerned). For companies, and for users. And that would be a way to ... put API's into people and companies again. Back to step 1.

afavour 2y ago
I see it as different history repeating: VC capital inserting itself as the middleman between people and things they want. If all of our interactions with external web sites now go through ChatGPT that gives OpenAI a phenomenal amount of power. Just like Google did with search.
- spwa4 2y ago
  Well, it's not just that. Every company insists on doing things differently and usually in annoying ways. Having a way to deal with companies while avoiding their internal policies (e.g. upselling, "retention team", ...) would be very nice. Yes, VCs want this because it's an opportunity for a double-sided marketplace, but I still want it too. I wonder to what extent is what these FANG businesses want with AI can be described as just "an API into businesses that don't want to provide an API".

readyplayernull 2y ago

> has a terms of service page on its site that forbids bots from taking its images without permission. But that alone did nothing. It's time to level up in this arms race. Let's stop delivering html documents, use animated rendering of information that is positioned in a scene so that the user has to move elements around for it to be recognizable, like a full site captcha. It doesn't need to be overly complex for the user that can intuitively navigate even a 3D world, but will take x1000 more processing for OpenAI. Feel free to come up with your creative designs to make automation more difficult.

liamwire 2y ago
Sounds entirely at odds with any accessibility requirements.
- readyplayernull 2y ago
  Are 3D videogames at odds with accessibility?

tonetegeatinst 2y ago

What options exist if you want to handle this traffic and you own your hardware on prem? It seems that any router or switch over 100G is extremely expensive, and often requires some paid for OS. The pro move would be to not block these bots. Well I guess block them if you truly can't handle their throughput request (would an ASN blacklist work?) Or if you want to force them to slow down, start sending data, but only a random % of responses are sent (so say ignore 85% of the traffic they spam you with, and reply to the others at a super low rate or you could purposely send bad data) Or perhaps reachout to your peering partners and talk about traffic shaping these requests.

philipwhiuk 2y ago

I had the same problem with my club's website.

101008 2y ago

Cloudflare should provide a service (paid or free) to block AI crawlers.

jgrahamc 2y ago
We do: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...