32 ms·
Let us serve you, but don't bring us down
- mintaka5 3y agohungry OpenAI crawling. getting all the data it can steal!!!
- radq 3y agoSounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.
- p-e-w 3y agoIndeed. Tens of thousands of requests per second from just 64 hosts? So they allow individual hosts to make hundreds of requests per second, sustained? That sounds crazy. Even for a burst limit hundreds per second would be extremely high.
- akiselev 3y agoArchive.org is a core utility for the web to the point where Wikipedia and many other sites would collapse without it in the sense that many if not most of their outbound links would be dead forever. I’m pretty sure it would even impact the US justice system [1]. Obviously judges aren’t going to have to worry about reasonable rate limits but if these DDoSes are rare, I’d much rather they dealt with them on a case by case basis. Without some complex dynamic rate limit that scales based on available compute and demand, rate limiting would be a blunt solution that will necessarily generate false positives. [1] https://www.theregister.com/2018/09/04/wayback_machine_legit/ https://www.theregister.com/2018/09/04/wayback_machine_legit...
- Defletter 3y agoProbably not just the US justice system. I also guess that academia would take a hit too. Not every citation is of a published paper.
- p-e-w 3y agoRate limiting is (usually) done on a per-host basis. The only users who would be adversely affected are those who perform hundreds of requests per second from a single system. Virtually every major website has per-host inbound request limits. This is completely standard practice and Archive.org is the odd one here. And DDoS isn't the only concern. Legitimate users that run poorly written Python scripts that make insane numbers of requests can hog server resources, and rate limits with appropriate error messages linking to resources documenting efficient access patterns can improve the experience for everyone, and drastically cut costs for the service operators.
- phkx 3y agoIt‘s rather per public IP, such that e.g. behind a corporate proxy you may experience rate limiting even for regular use, just because you’re sharing a few IP addresses with a large number of users. Better hope that you get an increased quota for your external IPs in that case.
- microtherion 3y ago> I’m pretty sure it would even impact the US justice system What an old fashioned concern! Don't you know that the US justice system uses ChatGPT as an archive retrieval system nowadays? /s
- hackernewds 3y agoPrecisely. What kinds of legitimate processes would require this kind of utility from Archive.org?
- deleted 3y ago[deleted]
- TekMol 3y agoWhat is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that? They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place. How have they not been sued out of existance yet?
- ddtaylor 3y agoDetails?
- crazydoggers 3y agoSame way a library does. The information is made available to anyone who wants it, the copyrights are maintained intact, and they don’t attempt to profit from the material. Copyright law specifically allows for libraries and archives to make copies of copyrighted material. Without such laws, without libraries, knowledge could not be guaranteed to be shared freely among the public, resulting in ever growing knowledge and education gaps between those with means and those without. Edit: Since you asked for the legal ground, here it is specifically: https://www.law.cornell.edu/uscode/text/17/108 https://www.law.cornell.edu/uscode/text/17/108 And here’s is further discussion by the copyright office itself: https://www.copyright.gov/policy/section108/discussion-document.pdf https://www.copyright.gov/policy/section108/discussion-docum... Congress and the copyright office have made it an important part of copyright law to protect archival, library and fair use doctrine.
- 3y ago
- pfooti 3y agoI run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, because we are hard-capped at 100 QPS to our provider, beyond that and they start dropping our traffic (and it is an outside provider, bundling domain registries like verisign and stuff), which makes regular users break if their traffic gets unlucky. Anyway, after a week of 40qps, they start spiking to 200+, and we pull the plug on the whole thing: now each request to our endpoint requires a recaptcha token. This is not great (more friction for legit users = more churn) but it is successful. IF they had only kept their QPS low, nobody would have cared. I wanted to send some kind of response code like, "nearing quota". FTR before people ask: it was quite difficult to stop this particular attack, since it worked like a DDOS, smeared across a _large_ number of ipv4 and ipv6 requesters. 50 QPS just isn't enough quota to do stuff like reactively banning IP numbers if the attacker has millions of IPs available.
- pierat 3y agoQPS seems to be "queries per second". I first thought it was something like "quadrillion bytes per second" or some newer over-the-top data measurement :)
- vaylian 3y agoI haven't heard of that abbreviation before. But there is a stub on Wikipedia if anyone cares about improving that article: https://en.wikipedia.org/wiki/Queries_per_second https://en.wikipedia.org/wiki/Queries_per_second
- sbierwagen 3y agoTemporarily add 500ms of latency to all ipv6 users, backoff timers for ipv4 addresses. Since there's only 4 billion v4 addresses, it's easy enough to just track them all in a sqlite db.
- mholt 3y ago> "Error establishing database connection" Well, that's ironic. ... Anyone have an archive.org link of the page?
- deleted 3y ago[deleted]
- behnamoh 3y agoPretty sure it was an attack, not legitimate bulk use of the website.
- copperx 3y agoI'm not an expert in networking, but isn't this scenario Cloudflare's bread and butter? Is there any reason to not outsource this problem?
- crote 3y agoIt depends. A CDN-like approach wouldn't really work because all requests are different and legit, so plain old caching isn't going to cut it. On the other hand, Cloudflare has quite some experience with DDoS protection, so they definitely have the right infrastructure in place to stop this kind of abuse. What is the bill going to be, though?
- ygouzerh 3y agoIt will definitely help yes, I don't know as well why they are not using it. And probably work on auto-scaling as well, in case the API Requests are legit. 10,000 requests/seconds is nowadays an average number for load testing in the companies I work with, for websites / mobile apps deployed at this scale
- aetherspawn 3y ago[flagged]
- fbdab103 3y agoHuh, meanwhile, I recently wanted to grab ~200 pages from the archive where I put a 1-2 minute delay between the requests and had a bunch of repeated failures.
- fragmede 3y agoThis is a job for HTTP 429!
- jszymborski 3y agoOr 420 :P
- jszymborski 3y agoI'm getting a lot of down votes so I suspect people might not be aware of the (unofficial) Enhance Your Calm HTTP status code that Twitter coined https://en.wikipedia.org/wiki/List_of_HTTP_status_codes https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
- toyg 3y agoI recently fought and won 429... On a website ran by the very company I work for. They use a vendor for certain services, and such vendor sells a (terrible) API access for bulk operations, so I whipped out Selenium instead; which meant I, the site "admin", was being throttled with 429. A few pauses here and there, and all was well.
- x-complexity 3y agoUnpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
- userbinator 3y agoand loudly point towards downloading the files via torrents. By "loudly", perhaps actually redirect to the .torrent instead, for those who trip the rate limit?
- x-complexity 3y ago...that can work.
- remram 3y agoA Link header would be more appropriate (maybe with rel="alternate"?)
- dan_wood 3y ago[flagged]
- legends2k 3y agoTorrents aren't illegal inherently; the content downloaded are often copyright violations, yes, but using torrenting protocol itself isn't illegal. Most Linux distributions provide a torrent option to download their OS image file.
- chaxor 3y agoTorrents are the backbone of a lot of academic research downloads. They're the absolute best way to distribute neural network weights, or large models used in physical science studies. That or IPFS. If you use huggingface or dockerhub for anything like this, seriously consider switching, as it will be extremely useful when those centralized systems inevitably fail.
- tastysandwich 3y agoI have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.
- Alifatisk 3y agoWish more people did like you!
- BlueTemplar 3y agoIt's not just about "wishing", (D)DoSing is a punishable offence in many jurisdictions, and considering the damage taking down such a popular and essential website causes, I hope tIA will report them to law enforcement if it happens a third time.
- bazmattaz 3y agoYes, this is the way. When I was purchasing a car last year I scraped a popular used car website for cars I liked to keep track of deals. I added a random sleep in between each page so the entire script took a few hours to run.
- thangngoc89 3y agoYes. I did the same. My side project takes 4-5 days for the whole routine because there are random wait in between requests and only 2 active requests at any given moment.
- perryprog 3y agoWhat’s the benefit of waiting a random amount of time between requests?
- Lalabadie 3y agoSome zealous systems will infer a very regular request rate as coming from automated services and block them, no matter how gentle the rate.
- blitzar 3y agoSomeone somewhere must be training a new "Open" Ai
- markus_zhang 3y agoHas anyone/any company asked you to buy their anti-DdoS service recent days/weeks?
- porbelm 3y agoOCRs of PD material? Is someone about to train a new AI model, perhaps? I'd think the Archive would be happy to arrange for receiving hard drives, filling them up, and sending them back. They seem like a very helpful bunch.
- wly_cdgr 3y agoIt's ChatGPT itself....learning....evolving...:)
- deleted 3y ago[deleted]
- Jamie9912 3y agoHow could they not have rate limiting?
- Aachen 3y agoPremature optimization, perhaps?
- yyyk 3y agoArchive.org is a bit of a special case, you need to call them repeatedly to archive a website. They do have a rate limit there, it's pretty aggressive* to the point you could trip it by manually using the site. They must have forgotten to limit the OCR files download. * If they had a better API (a simple non-synchronous API would be enough, one where we could send a list of URLs would be even better), one could have made a lot less calls.
- Retr0id 3y agoLast time I wanted to bulk-archive a bunch of urls, I asked about it, and sent a txt file full of URLs to someone and they put it in the archival queue.
- llui85 3y agoThey have a Google Sheets "API" which I've used and works reasonably well: https://archive.org/services/wayback-gsheets/ https://archive.org/services/wayback-gsheets/
- krackers 3y agoThis has been broken for the past month (just stuck on waiting for workers for several days), did they fix it?
- abbe98 3y agoI believe you can upload WRAC files to IA and ask them to index the content. Saves them the need to do the archiving and you won't be rate limited on their end.
- account42 3y agoSo they just trust you that your archives are not manipulated?
- Roark66 3y agoI just wonder, is archive.org getting any government grant money? If they aren't they should. And I'm not even talking about just the US. All sorts of countries (US, UK, Germany - to name the few) and international organisations like EU pour hundreds of millions into "cultural projects" of very questionable value. How about they actually fund something really worthy of preservation? Of course it is archive.org role to reach out first. For example EU could fund an archive.org mirror in the EU (with certain throughout etc). Of course opponents of public/government funding have a very good point in that many organisations when they get public money, they find a way to burn through all of it in a lot less efficient way. This can be mitigated by attaching concrete conditions to the grants. One example is a mirror in a specific location.
- brylie 3y agoThe Internet Archive lists some of their major donors on the About IA page: https://archive.org/about/ https://archive.org/about/ Since they are a U.S. 501(c)(3), they also publish annual reports, which can be downloaded e.g., at the ProPublica Nonprofit Explorer https://projects.propublica.org/nonprofits/organizations/943242767 https://projects.propublica.org/nonprofits/organizations/943...
- tkgally 3y agoThe Internet Archive has many smaller donors, too. I use the Archive frequently for a variety of purposes and derive immense value from it, so I donate to it every year.
- mellosouls 3y agoThere are various underfunded digital archives in the EU and elsewhere that have legal duties to preserve online content relating to their specific countries. They may be "of very questionable value" to you but your solution to remove funding from them to channel it to a much wealthier organisation in a wealthier country is neither ethical, legal or practical.
- hutzlibu 3y ago
- simonmales 3y agoWe have a service that was often crawled by bots on cloud providers, luckily we can block AWS IPs without to much hassle.
- mdaniel 3y agoI also recently learned they publish changes to their IP netblocks almost daily as part of their new "please use more SNS" blog post: https://news.ycombinator.com/item?id=35056973 https://news.ycombinator.com/item?id=35056973 For example: https://aws-new-features.s3.us-east-1.amazonaws.com/update/2023-05-26.json https://aws-new-features.s3.us-east-1.amazonaws.com/update/2... (see the `NEW_ip_prefixes` and `ip_prefixes_REMOVED` keys)
- buro9 3y agoI'm waiting for Cloudflare to open source their web server as promised. But failing that I'm likely to implement a web proxy that utilizes the wirefilter library that is already open sourced by Cloudflare (but no longer updated). The tools to stop these attacks are reasonably trivial. Most of the art of stopping them is observability. When you can see every dimension of a TCP/UDP connection, the TLS handshake, the HTTP communication... Then the dimension by which an attack is being conducted is glaringly obvious. Once you have the very obvious correlation, then you only need a blunt instrument of a tool that can block/deny/nullroute the attack. What's hard isn't the block rules, What's hard is instrumenting enough observability to see the obvious correlation of an attack. Even on my personal nginx a single request logs about 20 things. But there are literally hundreds of properties one can log if you have access to the entirety of the stack, and if you have access to log on it, then you have the ability to carry that context to a point in the code where you can block it. Also, 10k qps really isn't a lot. So this should be treated as a warning sign, this attack was low volume even for amateur booter services.
- alentred 3y agoJust yesterday there was a comment here on HN [1] about https://jsonip.com https://jsonip.com, which is essentially supported by a single person (all operational costs included) and gets abused in a somewhat similar manner. I am not even sure what to think: do the folks not understand what they do, or are they just bluntly ignorant of it? [1] https://news.ycombinator.com/item?id=36092417 https://news.ycombinator.com/item?id=36092417
- raverbashing 3y ago> o the folks not understand what they do, or are they just bluntly ignorant of it? They don't care. And yes I've heard stories of people "finding a service" that does something basic and just dumping their whole traffic onto it (of course they play the victim once they're found out)
- BlueTemplar 3y agoThe beatings (via judge) shall continue until they learn netiquette.
- marginalia_nu 3y agoYeah I'm getting this shit too with Marginalia Search. I'm getting about 2-2.5M queries per day that are definitely from bots that would 100% sink my server if they somehow go through the bot mitigation. It peaks at hundreds of search queries per second. To be clear these are search queries, and search queries typically trigger disk reads of about ~10-20 Mb. I get about 20,000 queries per day that may be human.
- lagniappe 3y agodoes your server have its own isp connection or is that your home connection
- marginalia_nu 3y agoI'm on my home connection. Not really anywhere near saturating it.
- netfortius 3y agoI would [have] reach[ed] to AWS techies. The services of archive.org are known and important enough for the AWS folks to chase down the sources and/or initiate some action against the perpetrators, malicious or not as they might be.
- MeteorMarc 3y agoWhat an incredibly well versed notice from archive.org! Clear, no assumptions, no irony, no accusations, but only reaching out as they request from us.
- dougb5 3y agoVery few people have been more dedicated to the original spirit of service of the World Wide Web than Brewster Kahle. It's as though he's been personally monitoring his server logs for the past 30 years.
- mellosouls 3y agoWe've had various web-crawling businesses promoting their crawl-etiquette-busting technologies to approval and acclaim on HN; undermining of services for the public good like this is exactly the end result we are supporting when we do that.
- isoos 3y agoProposal for web-scrapers to self-coordinate: monitor the response latencies and rate limit yourself based on that. (1) Limit your load / parallelism: do not use more than 4 threads, and if the site takes more than a few seconds to respond, use fewer. (2) Limit aggregate load: after each request, sleep/wait at least the amount that the previous request took to get served. (3) If you need more than that, ask the site owners for direct channel. This way if multiple crawler happen to crawl the same resource-limited site, the site may have a fighting chance.
- Tade0 3y agoI wish there was a way to just get a whole archived site N links deep as a .zip file - I tried a few tools for this and none of them worked as good as going over it manually and recording a .har file with all the contents. I recall tripping the rate limiter at least once by accident - I was clicking around the archived site for reference while working on creating a restored version.
- luizfelberti 3y agoThe fact that IA is so liberal with their scraping policies is laudable, because on the other end of the spectrum lies Wikimedia, where you're lucky if you manage to download their dumps at 500kbps, which makes their dumps pretty much impossible to obtain...
- shakesbeard 3y agoThey provide monthly torrents. Can't imagine that to be too slow unless there's literally no one seeding.
- hgsgm 3y agoI just tried it. over 50MB in 1minute (6MB/hr). http://dumps.wikimedia.your.org/enwiki/20220820/ http://dumps.wikimedia.your.org/enwiki/20220820/ They also seed torrents
- hgsgm 3y ago(er, 6MB/sec)
- Hello71 3y ago$ curl https://dumps.wikimedia.org/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 2 20.4G 2 476M 0 0 4387k 0 1:21:38 0:01:51 1:19:47 4393k^C $ curl https://wikidata.aerotechnet.com/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 20.4G 0 45.9M 0 0 1953k 0 3:03:23 0:00:24 3:02:59 2257k^C $ curl https://mirror.clarkson.edu/wikimedia/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 1 20.4G 1 344M 0 0 35.2M 0 0:09:56 0:00:09 0:09:47 36.8M^C maybe you should check your own internet connection first?
- elzbardico 3y agohttps://support.aws.amazon.com/#/contacts/report-abuse https://support.aws.amazon.com/#/contacts/report-abuse It should teach them a lesson to have their AWS account banned.
- grumple 3y agoFor actual spammers this won't do much. You can create a new account with a new email and presumably use one of those gift cards for payments (or use Apple wallet to regenerate card numbers).
- rickreynoldssf 3y agoThere's no reason for Archive.org to allow any traffic from AWS, Azure or Google Cloud. Archive.org should just block the CIDRs for all those networks. Most traffic that comes from them is scraping traffic and usually malicious.
- 8organicbits 3y agoSome users have virtual desktops in the cloud. Or use browser security sandboxes that run in the cloud.
- rickreynoldssf 3y ago"Sorry we cannot support your network" The needs of the many outweigh the needs of the very few...
- hgsgm 3y agoScrapers scraping scrapers.
- crazygringo 3y agoIs there an open-source rate limiter that works well for sites large and small? It just strikes me as surprising that sites are still dealing with problems like this in 2023. The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point. And I don't think different sites have particularly different needs here... basic pattern-matching heuristics can identify IP addresses/blocks (plus things like HTTP headers) that are suddenly ramping up requests, and use CAPTCHAs to allow legitimate users through when one IP address is doing NAT for many users behind it. Really the main choice is just whether to block spidering as much as possible, always allow it as long as it's at a reasonable speed, or limit it to a standard whitelist of known orgs (Google, Bing, Facebook, Internet Archive, etc.). It just strikes me as odd that when you follow a basic tutorial for installing Apache on Linux on Digital Ocean, rate limiting isn't part of it. It seems like it should be almost as basic as an HTTPS certificate by this point.
- klysm 3y agoI think rate limiting is the wrong approach. If there’s too much load you need to manage the queue
- deleted 3y ago[deleted]
- remram 3y agoWhat does "manage the queue" mean? And how would that be better than dropping requests from the source of the abuse?
- thayne 3y agoIt isn't that hard to set up naive rate limiting per ip address. It's a few lines in haproxy, and there is documentation on how to do it. There are a couple of problems that make it more complicated though. The first is that with NATs, you can have a lot of users behind a single IP address, which can result in legitimate requests getting blocked. The second is that, while it can help against a DoS, it doesn't help that much against a DDoS, because the requests are coming from a large number of distinct IP addreesses.
- syngrog66 3y agoreminds me of a general pattern I've observed in life: 1. whatever interface you expose to the public, some will abuse it 2. rules/laws/countermeasures are then added and enforced, often universally (though sometimes inconsistently) 3. which then bites, annoys, insults or otherise burdens or adds to the prices paid by ALL the OTHER non-abusing users pf that same interface always. eventually. every time
- lastangryman 3y agoIt's a digital Tragedy of the Commons: https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons
- syngrog66 3y agoagreed
- costco 3y agoWhy are there always super obnoxious scrapers out there? It doesn't even make sense from the scrapers perspective because it raises the chance of being blocked and having annoying anti bot measures introduced. It especially makes no sense in this case because archive.org would be happy to give them the data for probably less than whatever they were going to spend on AWS. There's no manners anymore.
- phendrenad2 3y agoI use the Internet Archive a lot, and I get rate limited after opening (manually!) a few dozen pages now. The message is clear: archive.org is meant for one-off lookups, if you're doing real historical research, you're not welcome. I thought creating an account and even donating money might help me, but nope.
- johnklos 3y agoIt's a shame that AWS can be used to do this and they'll either do nothing or take their sweet time doing anything, and half the time they reply saying that the abuse is no longer happening from the reported IPs (because they've already moved to new IPs). It's infuriating.