11 ms·
AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
- breakyerself 1y agoThere's so much bullshit on the internet how do they make sure they're not training on nonsense?
- bgwalter 1y agoMuch of it is not training. The LLMs fetch webpages for answering current questions, summarize or translate a page at the user's request etc. Any bot that answers daily political questions like Grok has many web accesses per prompt.
- 8organicbits 1y agoIs an AI chatbot fetching a web page to answer a prompt a 'web scraping bot'? If there is a user actively promoting the LLM, isn't it more of a user agent? My mental model, even before LLMs, was that a human being present changes a bot into a user agent. I'm curious if others agree.
- bgwalter 1y agoThe Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load. I'm not sure how many websites are searched and discarded per query. Since it's the remote, proprietary LLM that initiates the search I would hesitate to call them agents. Maybe "fetcher" is the best term.
- ronsor 1y ago> The Register calls them "fetchers". They still reproduce the content of the original website without the website gaining anything but additional high load. So does my browser when I have uBlock Origin enabled.
- danaris 1y agoBut they're (generally speaking) not being asked for the contents of one specific webpage, fetching that, and summarizing it for the user. They're going out and scraping everything, so that when they're asked a question, they can pull a plausible answer from their dataset and summarize the page they found it on. Even the ones that actively go out and search/scrape in response to queries aren't just scraping a single site. At best, they're scraping some subset of the entire internet that they have tagged as being somehow related to the query. So even if what they present to the user is a summary of a single webpage, that is rarely going to be the product of a single request to that single webpage. That request is going to be just one of many, most of which are entirely fruitless for that specific query: purely extra load for their servers, with no gain whatsoever.
- snowwrestler 1y agoWhile it’s true that chatbots fetch information from websites in response to requests, the load from those requests is tiny compared to the volume of requests indexing content to build training corpuses. The reason is that user requests are similar to other web traffic because they reflect user interest. So those requests will mostly hit content that is already popular, and therefore well-cached. Corpus-building crawlers do not reflect current user interest and try to hit every URL available. As a result these hit URLs that are mostly uncached. That is a much heavier load.
- shikon7 1y agoBut surely there aren't thousands of new corpuses built every minute.
- bgwalter 1y agoWhy would the Register point out Meta and OpenAI as the worst offenders? I'm sure they do not continuously build new corpuses every day. It is probably the search function, as mentioned in the top comments.
- snowwrestler 1y agoIt says in the first sentence of the article that it is 80% bots (crawlers) and only 20% fetchers. Of course they are crawling every day to improve their training data. The goal is LLMs that know everything, but “everything” changes on a daily basis. Meta and OpenAI are simply the largest after Google, but Google has had ~20 more years to learn how to politely operate crawlers at full-Internet scale.
- prasadjoglekar 1y agoBy paying a pretty penny for non bullshit data (Scale Ai). That along with Nvidia are the shovels in this gold rush.
- danny_codes 1y agoMaking a lot of assumptions about the quality of scale AI.
- danaris 1y agoI mean...they don't. That's part of the problem with "AI answers" and such.
- shinycode 1y agoIn the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?
- pm215 1y agoSure, but if the fetcher is generating "39,000 requests per minute" then surely something has gone wrong somewhere ?
- miohtama 1y agoEven if it is generating 39k req/minute I would expect most of the pages already be locally cached by Meta, or served statically by their respective hosts. We have been working hard on catching websites and it has been a solved problem for the last decade or so.
- ndriscoll 1y agoCould be serving no-cache headers? Seems like yet another problem stemming from every website being designed as if it were some dynamic application when nearly all of them are static documents. nginx doing 39k req/min to cacheable pages on an n100 is what you might call "98% idle", not "unsustainable load on web servers". The data transfer, on the other hand, could be substantial and costly. Is it known whether these crawlers do respect caching at all? Provide If-Modified-Since/If-None-Match or anything like that?
- mrweasel 1y agoMany AI crawlers seems to go to great length to avoid caches, not sure why.
- andai 1y agoThey're not very good at web queries, if you expand the thinking box to see what they're searching for, like half of it is nonsense. e.g. they'll take an entire sentence the user said and put it in quotes for no reason. Thankfully search engines started ignoring quotes years ago, so it balances out...
- internet_points 1y agoThey mention anubis, cloudflare, robots.txt – does anyone have experiences with how much any of them help?
- nromiun 1y agoCDNs like Cloudflare are the best. Anubis is a rate limitor for small websites where you can't or won't use CDNs like Cloudflare. I have used Cloudflare on several medium sized websites and it works really well. Anubis's creator says the same thing: > In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you. Source: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis
- Nextgrid 1y agoAlternatively, go-away is also an option: https://git.gammaspectra.live/git/go-away https://git.gammaspectra.live/git/go-away, especially if you don't want cringy branding.
- bakugo 1y agorobots.txt is obviously only effective against well-behaved bots. OpenAI etc are usually well behaved, but there's at least one large network of rogue scraping bots that ignores robots.txt, fakes the user-agent (usually to some old Chrome version) and cycles through millions of different residential proxy IPs. On my own sites, this network is by far the worst offender and the "well-behaved" bots like OpenAI are barely noticeable. To stop malicious bots like this, Cloudflare is a great solution if you don't mind using it (you can enable a basic browser check for all users and all pages, or write custom rules to only serve a check to certain users or on certain pages). If you're not a fan of Cloudflare, Anubis works well enough for now if you don't mind the branding. Here's the cloudflare rule I currently use (vast majority of bot traffic originates from these countries): ip.src.continent in {"AF" "SA"} or ip.src.country in {"CN" "HK" "SG"} or ip.src.country in {"AE" "AO" "AR" "AZ" "BD" "BR" "CL" "CO" "DZ" "EC" "EG" "ET" "ID" "IL" "IN" "IQ" "JM" "JO" "KE" "KZ" "LB" "MA" "MX" "NP" "OM" "PE" "PK" "PS" "PY" "SA" "TN" "TR" "TT" "UA" "UY" "UZ" "VE" "VN" "ZA"} or ip.src.asnum in {28573 45899 55836}
- rco8786 1y agoOpenAI straight up DoSed a site I manage for my in-laws a few months ago.
- muzani 1y agoWhat is it about? I'm curious what kinds of things people ask that floods sites.
- average_r_user 1y agoI suppose that they just keep referring to the website in their chats, and probably they have selected the search function, so before every reply, the crawler hits the website
- rco8786 1y agoThe site is about a particular type of pipeline cleaning (think water/oil pipelines). I am certain that nobody was asking about this particular site or even the industry its in 15,000 times a minute 24 hours a day. It's much more likely that their crawler is just garbage and got stuck into some kind of loop requesting my domain.
- hereme888 1y agoI'm absolutely pro AI-crawlers. The internet is so polluted with garbage, compliments of marketing. My AI agent should find and give me concise and precise answers.
- lionkor 1y agoThe second I get hit with bot traffic that makes my server heat up, I would just slam some aggressive anti bot stuff infront. Then you, my friend, are getting nothing with your fancy AI agent.
- mediumsmart 1y agoso the fancy AI agent will have to get really fancy and mimic human traffic and all is good until the server heats up from all those separate human trafficionados - then what?
- lionkor 1y agoNobody is forcing anyone to share their knowledge. What then? Dead internet.
- discardedrefuse 1y agoAbsolutely yes. I guarantee you these megacorps are betting on a future where the open internet has been completely obliterated. And the only way to participate online is thru their portal; where everything you do feeds back into their AI. Because that is the only way to acquire fresh food for their beast.
- discardedrefuse 1y agoThe end of the open web. That's what. Sites will have to either shutdown or move behind a protection racket run by one of the evil megacorps. And TBH, shutting down is the better option. With clickthru traffic dead, whats even the point of putting anything online? To feed AIs so that someone else can profit at my (very literal) expense? No thanks. The knowledge dies with me. The internet dark age is here. Everyone, retreat to your fiefdom.
- exasperaited 1y agoXe Iaso is my spirit animal. > "I don't know what this actually gives people, but our industry takes great pride in doing this" > "unsleeping automatons that never get sick, go on vacation, or need to be paid health insurance that can produce output that superficially resembles the output of human employees" > "This is a regulatory issue. The thing that needs to happen is that governments need to step in and give these AI companies that are destroying the digital common good existentially threatening fines and make them pay reparations to the communities they are harming." <3 <3
- delfinom 1y agoI run a symbol server, as in, PDB debug symbol server. Amazon's crawler and a few others love requesting the ever loving shit out of it for no obvious reason. Especially since the files are binaries. I just set a rate-limit in cloudflare because no legitimate symbol server user will ever be excessive.
- ack_complete 1y agoI have a simple website consisting solely of static webpages pointing to a bunch of .zip binaries. Nothing dynamic, all highly cacheable. The bots are re-downloading the binaries over and over. I can see Bingbot downloading a .zip file in the logs, and then an hour later another Bingbot instance from a different IP in the same IP range downloading the same .zip file in full. These are files that were uploaded years ago and have never retroactively changed, and don't contain crawlable contents within them (executable code). Web crawlers have been around for years, but many of the current ones are more indiscriminate and less well behaved.
- xrd 1y agoIsn't there a class action lawsuit coming from all this? I see a bunch of people here indicating these scrapers are costing real money to people who host even small niche sites. Is the reason these large companies don't care because they are large enough to hide behind a bunch of lawyers?
- outside1234 1y agoYes. There are one set of rules for us and another set of rules for anything with more than a billion dollars.
- EgregiousCube 1y agoUnder what law? It's interesting because these are sites that host content for the purpose of providing it to anonymous network users. ebay won a case against a scraper back in 2000 by claiming that the server load was harming them, but that reasoning was later overturned because it's difficult to say that server load is actual harm. ebay was in the same condition before and after a scrape. Maybe some civil lawsuit about terms of service? You'd have to prove that the scraper agreed to the terms of service. Perhaps in the future all CAPTCHAs come with a TOS click-through agreement? Or perhaps every free site will have a login wall?
- buttercraft 1y agoIf you put measures in place to prevent someone from accessing a computer, and they circumvent those measures, is that not a criminal offense in some jurisdictions?
- integralid 1y agoOn the other hand, DDoS attacks are pretty clearly on the illegal side. I wonder how this would play out in practice.
- Nextgrid 1y agoIntention plays a part. (D)DoS is intentionally done to make a website unavailable to legitimate users. Scraping may do this as a side-effect (if you are incompetent and/or use the "cloud"), but isn't the intention.
- timsh 1y agoA bit off-topic but wtf is this preview image of a spider in the eye? It’s even worse than the clickbait title of this post. I think this should be considered bad practice.
- encrypted_bird 1y agoI fully agree, and speaking as someone macroinsectophobia (fear of large or many insect (or insect-like) creatures), seeing it really makes me uncomfortable. It isn't enough to send me into panic mode or anything, but damn if it doesn't freak me out.
- lostmsu 1y agoThis article and the "report" look like a submarine ad for Fastly services. At no point does it mention the human/bot/AI bot ratio, making it useless for any real insights.
- jasoncartwright 1y agoI recently, for pretty much the first time ever in 30 years of running websites, had to blanket ban crawlers. I now whitelist a few, but the rest (and all other non-UK visitors) have to pass a Cloudflare challenge [1]. AI crawlers were downloading whole pages and executing all the javascript tens of millions of times a day - hurting performance, filling logs, skewing analytics and costing too much money in Google Maps loads. Really disappointing. [1] https://developers.cloudflare.com/cloudflare-challenges/ https://developers.cloudflare.com/cloudflare-challenges/
- tehwebguy 1y agoThis is a feature! If half the internet is nuked and the other half put up fences there is less readily available training data for competitors.
- AutoDunkGPT 1y agoI love this for us!
- bwb 1y agoMy book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter. I wish there were a better way to solve this.
- skydhash 1y agoIf you're not updating the publicly accessible part of the database open, try to see if you can put some cache strategy up and let cloudflare take the hit.
- bwb 1y agoYep, all but one page type is heavily cached at multiple levels. We are working to get the rest and improve it further... just annoying as they don't even respect limits..
- weaksauce 1y agoput a honeypot link in your site that only robots will hit because it’s hidden. make sure it’s not in robots.txt or ban it if you can in robots.txt. setup a rule that any ip that hits that link will get a 1 day ban in your fail2ban or the like.
- bwb 1y agoGot a good link to something on github that does this? I have to make sure legit bots don't get hit, as a huge percent of our traffic which helps the project stay active is from google, etc.
- subscribed 1y agoI did it manually and got fail2ban to read the access log anyway. The it's the permanent iptables rule, but could be CF API call as well.
- weaksauce 1y agohere's one example using rack attack on a rails app: https://github.com/pinballmap/pbm/blob/302ac638850711878ac618847b323fa7c2d9e3e3/config/initializers/rack_attack.rb#L27 https://github.com/pinballmap/pbm/blob/302ac638850711878ac61... https://github.com/pinballmap/pbm/blob/302ac638850711878ac618847b323fa7c2d9e3e3/config/robots.production.txt#L63 https://github.com/pinballmap/pbm/blob/302ac638850711878ac61... but it only bans for 3 hours. if they don't respect the hidden link and robots.txt they get banned.
- sct202 1y agoI wonder how much of the rapid expansion of datacenters is from trying to support bot traffic.
- loeg 1y agoIn terms of CapEx, not much. The GPUs are much more expensive. Physical footprint? I don't know.
- pjc50 1y agoPlace alongside https://news.ycombinator.com/item?id=44962529 https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Linux kernel?". This is why. AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher electricity bills, because keeping humans warm is not as profitable as a machine which directly converts electricity into stock price rises.
- rnhmjoj 1y agoI'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.
- mostlysimilar 1y agoBut it didn't, and it's happening now, because of AI.
- kjkjadksj 1y agoPeople have been complaining about these crawlers for years well before AI
- PaulDavisThe1st 1y agoThe issue is 1 to 4 orders of magnitude worse than it was just a couple of years ago. This is not "crawlers suck". This is "crawlers are overwhelming us and almost impossible to fully block". It really isn't the same thing.
- 1y ago
- okasaki 1y agoI wonder if we're doing the wrong thing blocking them with invasive tools like cloudflare? If all you're concerned about is server load, wouldn't it be better to just offer a tar file containing all of your pages they can download instead? The models are months out of date, so a monthly dumb would surely satisfy them. There could even be some coordination for this. They're going to crawl anyway. We can either cooperate or turn it into some weird dark market with bad externalities like drugs.
- recallingmemory 1y agoYeah, I am in the opposing camp too - I don't use Cloudflare's bot fight tooling on any of our high traffic websites. I'm not seeing the issue with allowing bots to crawl our websites other than some additional spend for bandwidth. Agent mode is pretty powerful when paired with a website that cooperates, and if people want to use AI to interact with our data then what's wrong with that?
- masfuerte 1y agoA tar file would be better if the crawlers would use it, but even sites with well-publicised options for bulk downloads (like wikipedia) are getting hammered by the bots. The bot operators DNGAF.
- HankStallone 1y agoRight. I don't care if AI (or anything else) indexes or learns from my sites. That's what they're there for. But yesterday I blocked an IP that hit one of my sites 82000 times in an hour, or 22/second. And apparently it's a very stupid bot, because it kept redownloading CSS and other asset files every time it saw a link to them. There's no way the people behind that bot are going to follow any suggestions to make it behave better. After all, adding things like caching and rate-limiting to your web crawler might take a few hours, and who's got time for that.
- zzo38computer 1y agoI also do not like Cloudflare. If the crawlers were aware of these archive files, and would be willing to use it, then that would help, but it isn't. (It would also help to know which dynamic files are worthless for archiving and mirroring, but they will often ignore that.)
- mcpar-land 1y agoMy worst offender for scraping one of my sites was Anthropic. I deployed an ai tar pit (https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147) to see what it would do it with it, and Anthropic's crawler kept scraping it for weeks. I calculated the logs and I think I wasted nearly a year of their time in total, because they were crawling in parallel. Other scrapers weren't so persistent.
- Group_B 1y agoThat's hilarious. I need to set up one of these myself
- fleebee 1y agoFor me it was OpenAI. GTPBot hammered my honeypot with 0.87 requests per second for about 5 weeks. Other crawlers only made up 2% of the traffic. 1.8 million requests, 4 GiB of traffic. Then it just abruptly stopped for whatever reason.
- whatevaa 1y agoTar pits and serve fake but legitimate looking content. Poison it.
- neilv 1y agoDon't the companies in the headlines pay big bucks for people working on "AI"? Maybe they are paying big bucks for people who are actually very bad at their jobs? Why would the CEOs tolerate that? Do they think it's a profitable/strategic thing to get away with, rather than a sign of incompetence? When subtrees of the org chart don't care that they are very bad at their jobs, harmed parties might have to sue to get the company to stop.
- jgalt212 1y agoabout 18 months ago, our non-Google / Bing bot traffic went from single digits per cent to over 99.9% bot traffic. We tried some home-spun solutions at first, but eventually threw in the towel and put Cloudflare in front of all our publicly accessible pages. On a long term basis, this was probably the right move for us, but we felt forced into this. And the Cloudflare Managed Ruleset definitely blocks some legit traffic such that it requires a fair amount of manual tuning.
- levleontiev 1y agoThat's why I am building a Rate Limiter as a service. Seems that it has its niche.
- s_ting765 1y agoCan confirm META's bots aggressively scraping some of my internet-facing services have but they do respect robots.txt.
- vkou 1y agoWhy is this not a violation of the CFAA, and why aren't SWEs and directors going to prison over it? As long as I have an EULA or a robots.txt or even a banner that forbids this sort of access, shouldn't any computerized access be considered abuse? Something, something, scraping JSTOR?
- deleted 1y ago[deleted]
- greatgib 1y agoThat's the moment that you remember that years ago in self hosted you could have sustained millions request per second on a single low end server for quite nothing. But now you are "on the cloud", with lambdas because "who cares" and hiring a proper part-time sysadmin is too complicated and so now you are pounded with crazy costs for moderate loads...
- p3rls 1y agoI am under attack mode right now because I am being attacked from hundreds of chinese proxied bots taking my PHP response time from its usual 0.2 to 2+second response times. fucking ridiculous.
- BrenBarn 1y agoOne thing I don't fully understand in all this is how the IP address stuff works. Like I keep hearing people saying somebody can get 10 gazillion residential IPs so they become unblockable, but how? This article also mentions crawlers should publish there IP ranges. Like, yeah? What if using more than X number of IPs to crawl was a criminal offense unless you got a permit, which would require you to identify and publish all those IPs up front?
- defrost 1y ago> But How? Typically by hiring them .. although their provenance might be sketchy.. eg: https://www.cybersecuritydive.com/news/us-charges-oregon-man-botnet-for-hire/758293/ https://www.cybersecuritydive.com/news/us-charges-oregon-man... ^ Oregon man arrested for running ~70,000 device DDOS-for-Hire botnet; the distributed attacking computers were mostly compromised IoT gadgets - fridges routers, toasters, doorbells, etc. with weak security that were probably scanned and p0wned via Shodan (or similar device mapping project). More legally there are many "free software" deals that offer services for people via installed software that comes with a side order of background web crawling in the fine print of the WALL-O'-TEXT Terms Of Uses agreement. Enterprising middle people gather up bots and offer them for hire to web crawlers, large scale companies will farm their own bots via their existing user base.
- BrenBarn 1y agoOkay, but are OpenAI and Meta straight up buying botnets on the black market?
- defrost 1y agoUnlikely. There are many ways, at their scale they (Meta at least) probably have edge servers in ISP's across the planet and can easily mix their crawlers with residential IP addresses rotationally assigned by domestic ISP's they co-mingle with. Or some other way, legal but somewhat obfuscated.
- diggan 1y ago