9 ms·
AI crawlers need to be more respectful
- troupo 2y agoWhat amazes me that none of this is surprising, all this behavior (not just what's described in the post) is on par with what the companies are doing, and have been doing for decades... And yet there will be many people, including here on HN, who will just cheer these companies on because they spit out an "opensource model" or a 10-dollars-a-month subscription
- influx 2y agoDo you feel the same way about Google spidering for their commercial search engine?
- BadHumans 2y agoGoogle sent traffic to your website. You were at least getting something in return.
- ToucanLoucan 2y agoEmphasis on were, since they've made their search such utter shit in the quest for ad revenue that they're now going to have their own AI sum up your results (badly) instead to attempt to solve the problem they created.
- hmry 2y agoYeah, and when Google stopped sending traffic, people sued them, as they should.
- deleted 2y ago[deleted]
- LukeShu 2y agoI don't. Just 3 AI spiders put more load on our servers than all search engine spiders and all human traffic combined. Some numbers I have handy from before I blocked the bots: ClaudeBot drove more requests through our Redmine in a month than it saw in the combined 5 years prior to ClaudeBot. Bytespider accounted for 59% of the total traffic to our Git server. Amazonbot accounted for 21% of the total traffic to our Git server. Google has never even been close to breaking out of the single-digit-percentages of any metric.
- DamonHD 2y agoGenerally Googlebot is well behaved and efficient these days, though I have discovered that it is currently horribly broken around 429 / 503 response codes... And pays no attention to Retry-After either... Also Google-Podcast which is meant to have been turned off!
- o11c 2y agoSomeone needs to start adding all these AI's homepages to the browser "malware" lists.
- Saris 2y agoNot originally since they just sent people to the site. But these days where they just rip content from the site to give people as answers, completely depriving the site of traffic, yeah that seems basically just as bad as the AI bots.
- talldayo 2y agoThe situation didn't change when it was search index crawlers being called-out. At the end of the day, this sort of "abuse" is native to the world of the internet; like you said, it's decades old at this point. HN will cheer on a lot of things that are counter-intuitive to their wellbeing; open-weight models doesn't feel like one of them. You can't protest AI (or search engines) because after long enough people can't do their job without them. The correct course of action is to name-and-shame, not write pithy engineering blogs begging people to stop. People won't stop.
- troupo 2y ago> HN will cheer on a lot of things that are counter-intuitive to their wellbeing; open-weight models doesn't feel like one of them. Except for the fact that they come from undisclosed sources from a company that does this: https://x.com/Tantacrul/status/1794863603964891567 https://x.com/Tantacrul/status/1794863603964891567
- talldayo 2y agoAnd I'll be damned if Tanta isn't having his tweets used for training Elon's AI. The parochial circle regresses as it goes around, I'm done acknowledging the make-believe barriers we pretended the internet clung to. You post it, others consume it. Same as it ever was.
- ToucanLoucan 2y agoExploitation of the common man being a key ingredient of a product has never once inspired any actual consumer revolt. Fundamentally, no matter what they say, as long as they get their fleeting hit of dopamine from buying/using a thing, people just don't care. As Squidward says: nobody gives a care for the fate of labor as long as they get their instant gratifications.
- croemer 2y agoJust 2 buggy crawlers seems not that many, sure they each had large impact, but given that there are likely hundreds if not thousands of such crawlers out there it's a rather small number. It seems that most crawlers are actually respectful.
- Centigonal 2y agoHow many foundation AI companies are there? 2 is a pretty big chunk of that pie.
- PaulHoule 2y agoI used to run a site with a huge number of pages that had high running costs but low revenue. The only web crawler that did anything for me was Google, as Google sent an appreciable amount of traffic. Referrers from Bing were almost undetectable: the joke among my black hat SEO friends at the time was that you could rank for money keywords like "buy wow gold" and get 10 hits. Then there were the Chinese crawlers like Baidu that would crawl at 10x the rate of Google but send zero referrers. And then there were crawlers looking for copyrighted images that cost me money to accommodate even if they never sent me cease and desist letters. As much as I hate the Google monopoly I couldn't afford having my site crawled like that without any benefit to me. It's an awful situation for the long term though because it prevents new entrants. Right now I am thinking about a new search engine for a vertical where a huge number of products are available from different vendors and when you do find results from Google they are sold out at least 70% of the time. I hate to think it's going to get harder to make something.
- xyst 2y agoSo much waste. Even worse than the digital currency rush.
- simonw 2y ago"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.
- exe34 2y agoat that scale, they could just ship some usb disks and a prepaid return postage.
- mschuster91 2y agoThe more interesting thing for me is: the crawler didn't detect it on its own that it racked up 10TB from one site in one day. If I would design a crawler, I'd keep at least some basic form of tracking, if only to check for people deliberately trolling me by delivering me an infinite chain of garbage.
- bastawhiz 2y agoI've been running guthib.mattbasta.workers.dev for years, and in the past few months it's hit the free limit for requests every day. Some AI company is filling their corpus with exactly that: infinite garbage.
- deleted 2y ago[deleted]
- stusmall 2y agoRespect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.
- ryandrake 2y agoWhy is it "classy" to not name names when a business (which likely holds it self out there as reputable) behaves badly, especially when it behaves in a way that costs you money? Everyone is so vague and coy. These companies are being abusive and reckless. Name and shame!
- guhcampos 2y agoWe all know wheret his is headed. In 5 years every useful content in the Web will be behind a paywall.
- surfingdino 2y agoServed as word vomit.
- deleted 2y ago[deleted]
- RecycledEle 2y agoI suspect the AI companies are as careless with their training as they are with their web scraping.
- winddude 2y agoNot only that, even commoncrawl had issues (about a year ago) where AWS couldn't keep up with the demand for downloading the WARCs. As someone who written a lot of crawling infrastructure and managed large scale crawling operations, respectful crawling is important. That being said it always seems like google has had a massively unfair advantage for crawling not only with budget but with brandname, and perceived value. It sometimes felt hard to reach out to websites and ask them to allow our crawlers, and grey tactics were often used. And I'm always for a more open internet. I think regular releases of content in a compressed format would go a long way, but there would always be a race for having the freshest content. What might be better is offering the content in machine format, XML or JSON or even SOAP. Which is usually better for what the sites crawling want to achieve, cheaper for you to serve, and cheaper and less resource intensive compared to crawling. (Have them "cache" locally by enforcing rate limiting and signup)
- apantel 2y ago> That being said it always seems like google has had a massively unfair advantage for crawling not only with budget but with brandname, and perceived value. VCs and other startup culture evangelists are always challenging founders to figure out what their ‘unfair advantage’ is. That’s the name of the game.
- int3 2y agoShouldn't all sites have some kind of bandwidth / cost limiting in place? Not to say that AI crawlers shouldn't be more careful, but there are always malicious actors on the internet, seems foolish not to have some kind of defense in place
- DamonHD 2y agoIt's harder to do right then you think. The first dynamic bandwidth (and concurrent connection) limiter that I wrote was to protect a site against Google in part!
- jsheard 2y agoThe big three cloud providers (AWS/GCP/Azure) have collectively decided that you don't want to set a spending limit actually, so they simply don't let you.
- bo1024 2y agoThey say this: > We have IP-based rate limiting in place for many of our endpoints, however these crawlers are coming from a large number of IP addresses, so our rate limiting is not effective. Do you have something else in mind? Just shut down the whole site after a certain limit?
- shrimp_emoji 2y ago[flagged]
- pants2 2y agoWhile the crawling is disrespectful, it seems RTD could find a cheaper host for their files. At my work we have a 10G business fiber line and serve >1PB per month for around $1,500. Takes 90% of the load off our cloud services. Took me just a couple weeks to set up everything.
- Joel_Mckay 2y agoHad a conversation with a firm that wanted a distributed scraper built, and they really did not care about site usage policies. You would be fooling yourselves if you think such a firm cared about robots.txt or page tags. We warned them they would be sued eventually, to contact the site owners for legal access to the data, and issued a hard pass on the project. Probably they assumed if the indexing process was out of another jurisdiction their domestic firm wouldn't be liable for theft of service or copyright infringement. It was my understanding AI/ML does not change legal obligations in business, but the firm probably found someone to build that dubious project eventually... Spider traps and rate-limiting are good options too. =3
- reaperman 2y agoRobots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from crawling our site”. Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.
- Joel_Mckay 2y agoThe posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)
- echoangle 2y agoLegally enforceable as in “you can block people who don’t follow the policy” or enforceable as in “you can sue them for money”?
- Joel_Mckay 2y agoIf I recall it is considered theft-of-service if you bypass the posted site usage terms with an agent like a spider, and certainly a copyright violation for unauthorized content usage (especially in the context of a commercial venture.) One may be sued, but not because you parsed robots.txt wrong =3
- DamonHD 2y agoNot just AI: here is my current side-quest: https://www.earth.org.uk/RSS-efficiency.html https://www.earth.org.uk/RSS-efficiency.html Over 99% of the bandwidth (and CPU) taken by the biggest podcast / music services simply on polling feeds is completely unnecessary. But ofc pointing this out to them gets some sort of "oh this is normal, we don't care" response because they are big enough to know that eg podcasters need them.
- immibis 2y agoCan you insert a podcast item that says "you are spamming our server - please stop"?
- DamonHD 2y agoNot easily on a static server and not without the risk of annoying an actual real human listener! If you look at the "Defences" section: https://www.earth.org.uk/RSS-efficiency.html#Hints https://www.earth.org.uk/RSS-efficiency.html#Hints you'll see there are some things that can be done, such as randomly rejecting a large fraction of requests that don't allow compression (gzip is madly effective on many feed files: it's rude for a client not to allow it). But all these measures take effort to set up, and don't stop the bad bots making the request 100s of times too often. Just responding to each stupid request forces a flurry of packets and wakes up and uses CPU...
- bastawhiz 2y agoI run pinecast.com. If there was a leaderboard for hn users serving XML, I'd almost certainly be in the top five. I don't disagree with your post. But: RSS downloads are at an all time low, and that's a bad thing. They're at an all time low because Spotify and Apple both fetch feeds from centralized servers. 1000 subscribers no longer means 24000ish daily feed fetches, it means 48. With keep alive or H2, these services simply don't reconnect. The number of IPs that hit me from Apple, for instance, is probably only double digits. Since Apple and Spotify both sit between me and the listeners, they eliminate the privacy that listeners would otherwise enjoy. It also forces podcasters to go to them to find out how many people are subscribed, which means lots of big databases instead of one database that I host for my customers. Centralization of feed checking carries huge risks, in my opinion, especially as both Apple and Spotify make moves to also become the hosting providers.
- Venn1 2y agoI blocked Microsoft/OpenAI a few weeks ago for (semi) childish reasons. Seven months later, Bing still refuses to index my blog, despite scraping it daily. The AI scrapers and crawlers toggle on Cloudflare did the trick.
- mikae1 2y agoHellPot: https://github.com/yunginnanet/HellPot https://github.com/yunginnanet/HellPot > Clients (hopefully bots) that disregard robots.txt and connect to your instance of HellPot will suffer eternal consequences. HellPot will send an infinite stream of data that is just close enough to being a real website that they might just stick around until their soul is ripped apart and they cease to exist.
- levkk 2y agoThis doesn't solve their bandwidth costs which is their real problem with these bots.
- everforward 2y agoI'm curious if there's a point where a crawler is misconfigured so badly that it becomes a violation of the CFAA by nature of recklessness. They say a single crawler downloaded 73TB of zipped HTML files in a month. That averages out to ~29 MB/s of traffic, every second, for an entire month. Averaging 30 megabytes a second of traffic for a month is crossing into reckless territory. I don't think any sane engineer would call that normal or healthy for scraping a site like ReadTheDocs; Twitter/Facebook/LinkedIn/etc, sure, but not ReadTheDocs. To me, this crosses into "recklessly negligent" territory, and I think should come with government fines for the company that did it. Scraping is totally fine to me, but it needs to be done either a) at a pace that will not impact the provider (read: slowly), or b) with some kind of prior agreement that the provider is accepting responsibility to provide enough capacity. While I agree that putting content out into the public means it can be scraped, I don't think that necessarily implies scrapers can do their thing at whatever rate they want. As a provider, there's very little difference to me between getting DDoSed and getting scraped to death; both ruin the experience for users.
- 5757rurt 2y ago[flagged]
- asdasdsddd 2y agoWhy can't you just rate limit non-browser user agents very aggressively, if your primary audience is human.
- bakugo 2y agoThe words "AI" and "respectful" don't belong in the same sentence. The mere concept of generative AI is disrespectful.
- apantel 2y agoThe concept of generative AI is not disrespectful. You could have generative AI trained on licensed data - would that be disrespectful?
- surfingdino 2y agoSites need to start suing crawler operators for bandwidth costs.
- echoangle 2y agoHow is that supposed to work? Under which law will you force me to pay for using your public website, especially if I am not in your country? Just put up a captcha and block crawlers, you’re not going to get them to pay you
- mateozaratefw 2y agoTik Tok crawler fucked us up by taking a product-name (e-commerce) and inserting it into the search bar recursively with the results page. Respect the game, but not respecting the robots.txt crawl delay is awful.
- Pesthuf 2y agoHave capitalists ever stopped just because their actions (that make them money) hurt others? Because the consequences of the damage they cause might in the end hurt them, too? Hoping they just stop seems futile…
- deleted 2y ago[deleted]
- bo1024 2y agoIt would be nice to have more of a peer-to-peer infrastructure (torrent inspired) for serving big resources.
- immibis 2y agoNever going to happen due to misaligned incentives. In 2024 everyone wants to keep their data behind lock and key. The commons is gone. Just look at the Google/Reddit deal.
- joshu 2y agoeverything old is new again. i remember when someone at google started aggressively crawling del.icio.us from a desktop machine and i ended up blocking all employees...
- exhaze 2y agoHaving built an AI crawler myself for first party data collection: 1. I intentionally made sure my crawler was slow (I prefer batch processing workflows in general, and this also has the effect of not needing a machine gun crawler rate) 2. For data updates, I made sure to first do a HEAD request and only access the page if it has actually been changed. This is good for me (lower cost), the site owner, and the internet as a whole (minimizes redundant data transfer volume) Regarding individual site policies, I feel there’s often a “tragedy of the commons” dilemma for any market segment subject to aggregator dominance: - individual sites often aggressively hide things like pricing information and explicitly disallow crawlers from accessing them - humans end up having to access them: this results in a given site either not being included at all, or accessed once but never reaccessed, causing aggregator data to go stale - aggregators often outrank individual sites due to better SEO and likely human preference of aggregators, because it saves them research time - this results in the original site being put at a competitive disadvantage in SEO, since the their product ends up not being listed, or listed with outdated/incorrect information - that sequence of events leads to negative business outcomes, especially for smaller businesses who often already have a higher chance of failure Therefore, I believe it’s important to have some sort of standard policy that is implemented and enforced at various levels: CDNs, ISPs, etc. The policy should be carefully balanced to consider all these factors as well as having a baked in mechanism for low friction amendment based on future emergent effects. This would result in a much better internet, one that has the property of GINI regulation, ensuring well-distributed outcomes that are optimized for global socioeconomic prosperity as a whole. Curious to hear others’ perspectives about this idea and how one would even kick off such an ambitious effort.
- andrei_says_ 2y agoThey will be once we legislate that respect. Not a second earlier.
- johneth 2y ago> "One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Invoice the abusers. They're rolling in investor hype money, and they're obviously not spending it on competent developers if their bots behave like this, so there should be plenty left to cover costs.