5 ms·
"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to
by simonw 2y ago
"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler."
Wow. That's some seriously disrespectful crawling.
- exe34 2y agoat that scale, they could just ship some usb disks and a prepaid return postage.
- mschuster91 2y agoThe more interesting thing for me is: the crawler didn't detect it on its own that it racked up 10TB from one site in one day. If I would design a crawler, I'd keep at least some basic form of tracking, if only to check for people deliberately trolling me by delivering me an infinite chain of garbage.
- bastawhiz 2y agoI've been running guthib.mattbasta.workers.dev for years, and in the past few months it's hit the free limit for requests every day. Some AI company is filling their corpus with exactly that: infinite garbage.
- deleted 2y ago[deleted]
- stusmall 2y agoRespect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.
- ryandrake 2y agoWhy is it "classy" to not name names when a business (which likely holds it self out there as reputable) behaves badly, especially when it behaves in a way that costs you money? Everyone is so vague and coy. These companies are being abusive and reckless. Name and shame!
- WhyCause 2y agoIn the article, they mention that they are working with the crawling company to be reimbursed for the download costs. Naming and shaming the company while you're trying to work with them is a real good way to not get what you want.
- deleted 2y ago[deleted]
- OtherShrezzing 2y agoNaming names isn't really required. The hosts have a $5,000 bandwidth fee, but so do the consumers. There's maybe 10 companies with the financial & compute resources to let a $5,000-per-month-per-website bug run rampant before taking the harvesting service offline. Meta/Google/Whoever may benefit from economies of scale, so they're not seeing the full $5,000 their side, but they're hitting tens of thousands of sites with that crawler.
- energy123 2y agoI thought data egress is much more expensive than downloading it?
- echoangle 2y agoYou know you can hit the data rate they were complaining about by using a residential fiber connection, right? 10 TB per day is about 1 Gigabit continuous if I’m not mistaken. There are probably millions of people that could to this if they wanted to.
- OtherShrezzing 2y agoThere's millions of people who could do that to an individual website. There are remarkably few organisations who could do that simultaneously across the top 100,000 or so sites on the internet, which is how readthedocs has encountered this issue.
- Joel_Mckay 2y agoIt is 100% likely a cloud provider IP range. They are a persistent source of spam email servers, scrapers, and bot probes. The simple reason is the operators quickly dump a host, and the next user is left wondering why their legitimate site is instantly spelunking spam ban lists. It is the degenerative nature of cloud services... and unsurprisingly we end up often banning most parts of Digital Ocean, Amazon, Azure, Google, and Baidu. Have a wonderful day, =3
- immibis 2y agoIt's the degenerative nature of assuming an IP corresponds to a user. They have not corresponded to users for over a decade. I once discovered I'm banned on my mobile phone connection from at least one app which doesn't know that CGNAT exists (a very poor assumption for mobile phone apps in particular). If you must block IPs, do it as a last resort, make it based on some observable behavior, quickly instated when that behavior occurs, and quickly uninstated when it does not.
- Joel_Mckay 2y agoReally depends on the use-case, but yeah the response happens in a proportional manner. We also follow the tit-for-tat forgiveness policy to ensure old bans are given a second chance. Mostly, we want the nuisance to sometimes randomly work, as it wastes more of their time fixing bugs. And note, if a server is compromised and persistently causing a problem... we won't hesitate to black hole an entire country along with the active Tor exit nodes and known proxies lists (the hidden feature in context cookies). Have a great day friend, =3
- croemer 2y agoWhy are you ending all your messages with =3 ?
- Joel_Mckay 2y agohttps://www.jpl.nasa.gov/images/pia22092-arp-142-the-penguin-and-the-egg https://www.jpl.nasa.gov/images/pia22092-arp-142-the-penguin... Don't worry about it friend =3
- Joel_Mckay 2y agoTry page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha. If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days. 73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.) Good luck =3
- DamonHD 2y agoWhen the bots come out of cloud services then the IPs are all over the place: it's much harder to do right these days.
- Joel_Mckay 2y agoRate limiting firewalls and spider traps also work well... There is page referral monitoring, context cookies, and dynamically created link chaff with depth-charge rules. One can dance all day friend, but a black-hole is coming for the entire IP block shortly. And unlike many people, some never remove a providers leased block and route until it is re-sold. =3
- immibis 2y agoI mean, blocking random IP ranges is your prerogative if you don't want to have customers. The scrapers will find ways around, while actual users will be unable to use your site. Residential proxies are something like $5 for 1000.
- Joel_Mckay 2y agoTrue, note domestic ISP IP ranges are published, and unless one deals internationally... don't bother serving people that will never buy anything from your firm anyways. Domestic "Users" functioning as proxies will be tripping usage limits, and getting temporarily banned. Google does this by the way, try hammering their services and find out what happens. Context cookies also immediately flag egregious multi-user routes, and if it is a ISP IP you know its a problem user. If it is over 15 users an hour per IP, than you can be 100% sure its a Tor proxy. we ban over 243000+ IPs, and saw zero impact to our bottom line. Have a nice day, =)
- Faaak 2y ago5000$ for 73TiB seems excessive tough? Some EU cloud providers would price it 10x cheaper
- immibis 2y agoHetzner is $1 per TB and that's only if they decide your overall usage is excessive and needs to be billed for.
- deleted 2y ago[deleted]
- immibis 2y agoThat's also $73 worth of bandwidth on another server host. Please stop using extremely expensive hosts and then blaming other people for the consequences of your decision.
- gnfargbl 2y ago10TB/day is, roughly, a single saturated 1Gbit link. In technical bandwidth terms, that is the square root of fuck all. The crazy thing here is that the target site is content to pay ~$700 for the volume of traffic that you can move through a single teeny-tiny included-at-no-extra-charge cat5 Ethernet link in one single day. And apparently, they're going to continue doing so.
- Saris 2y agoYeah that's an insane price for bandwidth, they need to move providers ASAP if that's the kind of fees they get.
- bastawhiz 2y agoHosting documentation shouldn't need that much bandwidth. It's text and zip files full of text. Without bots, that's a very small cost even if the bytes are relatively costly.