21 ms·
The largest DDoS attack to date, peaking above 398M rps
- mjan22640 3y agoAs elections are getting closer, the public needs to be primed for adversary attacks and disinformation.
- nancyp 3y agoIs there a chance that these cloud ddos service providers are attacking each other or themselves to create PR opportunities?
- jruohonen 3y agoSo is this a new world record?
- taway_6PplYu5 3y agoNew headline suggestion: Unnamed group creates larges DDoS attack to date.
- mensetmanusman 3y agoGuinness was there with their clipboards, so yes.
- deleted 3y ago[deleted]
- adzm 3y agoLinked in this article is more info on the rapid reset feature of HTTP2 which was used at part of the ddos https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack https://cloud.google.com/blog/products/identity-security/how...
- oldtownroad 3y agoAt a previous company, we were subject to semi-frequent attacks (of a much smaller scale). The operating assumption internally was that it’s a competitor trying to undermine us but it remains a mystery. Anyone involved in these type of attacks (at internet-infrastructure scale or targeting specific companies) brave/crazy enough to create a throwaway account and tell hn about the motivations?
- logdahl 3y agoThe universities in Sweden were attacked by "Turkey" after the big quran-burning scandal. They had some twitter account bragging about it. Was pretty evident it was Russia.
- codezero 3y agoI had a customer that was getting DDoSed by competitors, the competitors likely didn't know they were doing a DDoS, as they were aggressively scraping product listings but just doing so without any delay/rate limiting and it effectively DDoSed them. They weren't trying to make the target site slower, but were trying to get data at a rate that made the target's servers uneconomical to their actual paying customers. This kind of attack is nothing like the actual DDoS attacks, but it's a lot more common in my experience, but also relatively easy to mitigate with something like Cloudflare or Akamai (which is what I'd recommend to my customers).
- OsrsNeedsf2P 3y agoWe had a similar issue and assumed it was script kiddies having fun. Turns out someone (raises hand) wrote a really bad microservice who's inefficient queries sometimes triggered all our alerts.
- saltminer 3y agoReminds me of a support ticket I had to investigate recently. Performance metrics were taking a dive and triggered automated alerts - average response time jumped from 200ms to 2000, 8000, and eventually began approaching 15000, at which point requests were timing out all over the place. At first I was wondering if my recently-deployed MR was responsible, but upon further investigation, it was from one developer on another team doing two very dumb things with their application prototype: - Constantly retrying the same query with no filtering and setting the maximum allowed page size - Sending massive quantities of small queries for specific individuals (50-100 per second) which quickly hit the rate limit, and immediately resuming after the rate limit expired Some developer outreach was required...
- dduarte 3y agoSame attack on Cloudflare https://blog.cloudflare.com/zero-day-rapid-reset-http2-record-breaking-ddos-attack/ https://blog.cloudflare.com/zero-day-rapid-reset-http2-recor...
- Aissen 3y agoThe technical article (linked in the post) has more interesting details: https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/ https://blog.cloudflare.com/technical-breakdown-http2-rapid-...
- H8crilA 3y agoThis should be the top comment. TL;DR: HTTP/2 is internally concurrent, can handle multiple streams. It is possible in HTTP/2 to send a nasty request that looks like so: - GET x1 - GET x2 - GET x3 - ... - GET x100 - Actually, cancel all of the above (uses multiple RST_STREAM frames) - GET x101 - GET x102 - (...) - GET x200 - Actually, cancel all of the above (uses multiple RST_STREAM frames) - (...) This can be repeated a lot of times. The problem is that the endpoint, which typically is a reverse proxy, might start dispatching the requests before it reads about their cancellation. And sure it will cancel them, but by the time of cancellation it will already have resulted in some resource usage downstream. Such requests are accepted because at no point the client has opened more than 100 streams, which is the typical concurrency limit. The example from the blog manages to squeze in a single packet 1000 GETs (i.e. 1000 HEADERS) correctly interleaved with RST_STREAM. Maybe it's just me, but it's always fun to see such creative and simple abuses of protocols/code.
- ryanisnan 3y agoThat’s pretty fascinating. This is a naive solution, but couldn’t the protocol have supported limits of requests per packet? I get that it is antithetical, but for most sites, this type of request pattern seems highly unusual.
- AtNightWeCode 3y agoIf this is true than the design is problematic. What makes it even worse is that cancellation of requests typically does not work in cloud environments. It is a bit laughable that Azure for instance recommend the use of cancellation tokens but in reality you never get them for web requests.
- sph 3y agoHow does DDoS mitigation work? When people say "I put my website behind Cloudflare to mitigate DDoSes", what does it mean exactly? Is it only about having a large enough ingress pipe that you can weather however many Gb/s you are being bombarded with, and still having some spare capacity for legitimate traffic?
- jart 3y agoYou use algorithms like token bucket.
- dharmab 3y agoOften it means automatically recognizing DDoS requests and handling them in a way that is less costly, without impacting legitimate users. In this case, it might mean recognizing when a client rapidly resets streams, and either moving that traffic to a slow lane or filtering it entirely.
- richardwhiuk 3y agoCloudflare, and other companies, can detect that requests are DDoS and either drop, throttle, or verify the traffic, instead of forwarding it your server. You configure your server to drop all traffic which wasn't set by cloudflare, which is efficient.
- SteveNuts 3y agoWhen the ddos attack is volumetric, the only way to mitigate it is to have a fat enough network to handle the traffic while you work with ISPs to start blocking the traffic upstream. Not all ddos attacks are based on volume though, some are exploiting native features of a protocol, like the slow loris attack https://www.cloudflare.com/learning/ddos/ddos-attack-tools/slowloris/ https://www.cloudflare.com/learning/ddos/ddos-attack-tools/s...
- dijit 3y agothat's not the only way. The way we used to do it is have "filter boxes" with a real anycast IP address's which reverse connect to your origin. This helps a lot because it keeps a lot of traffic localised instead of allowing it to collect in one place. Anycast should also mean you have a failover mechanism; but if it fails then you're only down in one section of the world where the most bots are anyway, which is usually not as bad as being down globally.
- ricardobeat 3y agoNo word on the origin of these attacks? This must require massive amounts of hardware, you’d imagine it to be easily traceable unless some kind of botnet.
- danpalmer 3y agoOne could imagine that, given the size, it could be politically or legally sensitive to announce the origin.
- ricardobeat 3y agoThe silence is actually already giving it away then, one of four options.
- blagie 3y agoEnumerate, please.
- KolmogorovComp 3y agoI assume China, Iran, North-Korea or Russia (in alphabetical order).
- lionkor 3y agoUS enemy one, two, three and four (whoever is trendy to blame right now)
- persedes 3y agoLooking at the scale of those that's what I figured too, but one of the previous largest ones (mirai) was targeting a minecraft server (...). Krebs has some interesting write ups on Mirai.
- knorker 3y agoCloudflare explicitly says it's an unknown threat actor: https://blog.cloudflare.com/zero-day-rapid-reset-http2-record-breaking-ddos-attack/ https://blog.cloudflare.com/zero-day-rapid-reset-http2-recor...
- mihaic 3y agoThe fact that large cloud providers can handle huge DDoS attacks I think in the long run leads to a worse internet. It forces botnets to up their game and for websites the only solutions available are to pay Google, Amazon or Cloudflare a protection tax. I honestly don't see any other options, but I'd really wish for them to come through some community coordinated list of botnet infected IPs or something.
- elwell 3y agoLeave it to HN to find the fly in the ointment when Google is mentioned.
- rs999gti 3y ago> Cloudflare a protection tax $NET gives away DDOS protection for free for non-businesses
- falcor84 3y agoNothing "forces botnets to up their game", they just want to make money (or in some cases, "watch the world burn"); I don't see how any coordination whatsoever would diminish these motivations.
- grotorea 3y agoSo the email spam solution? Doesn't that come with its own list of problems? Also, stupid question from someone not that familiar with DDoS, can't you flood the target with requests even if the source address will be rejected? Or even if the IP packet has a falsified source address?
- tmpX7dMeXU 3y agoYes.
- kijin 3y agoA spamhaus-like blacklist for botnet IPs is an interesting idea. What if Google and Cloudflare collectively reverse-DoSed all the infected IPs, not by sending them any traffic, but simply by refusing to accept any connections from them to any part of their infrastructure? Whoever is on those IPs will suddenly find that half the internet doesn't work anymore. Which is probably a good enough incentive for them to replace their router, format their PC, or whatever else is necessary to disinfect themselves. In many parts of the world, landline IP allocations tend to be stable enough for this to have a real effect. Phones are a different story, but phones are also much less likely to be useful in a DDoS botnet. (The owner would immediately notice the sudden heat and data usage.) If we're going to live in a world where a small number of companies own half the internet, at least they could use their power to do some good.
- Aissen 3y agoInterestingly, the CVE is still "reserved": https://nvd.nist.gov/vuln/detail/CVE-2023-44487 https://nvd.nist.gov/vuln/detail/CVE-2023-44487 .
- narag 3y agoAny clue who was the target? Can such an attack be performed with botnets?
- endisneigh 3y agoWonder how you could even handle this if you weren’t using a big cloud provider and didn’t have a lot of money to spend.
- ivanjermakov 3y agoPatch your HTTP server to mitigate the Rapid Reset attack: > To mitigate against the non-cancelling variant of this attack, we recommend that HTTP/2 servers should close connections that exceed the concurrent stream limit. This can be either immediately or after some small number of repeat offenses. https://cloud.google.com/blog/products/identity-security/how-it-works-the-novel-http2-rapid-reset-ddos-attack https://cloud.google.com/blog/products/identity-security/how...
- knorker 3y agoHow small a server setup do you mean? This was apparently 20k machines: https://blog.cloudflare.com/zero-day-rapid-reset-http2-record-breaking-ddos-attack/ https://blog.cloudflare.com/zero-day-rapid-reset-http2-recor... Most web services don't need clever protocol attacks to be downed by 20k machines.
- tristan9 3y agoYou can’t. If your webserver receives 400m rps it dies, end if story. Mitigations are just that, mitigations. They are as effective as buying a better door lock to protect your apartment from a nuke.
- deleted 3y ago[deleted]
- ChumpGPT 3y agoCut China, North Korea, Iran and Russia off from the Internet and problem solved. It will be difficult but it can be done.
- deleted 3y ago[deleted]
- supertrope 3y agoWhat about infected computers and cheap VPS within the United States?
- ilyt 3y agoIt's easy to attack thru VPNs or leased cloud VMs.
- codedokode 3y agoSuch attacks are possible because ISPs do not want to adopt a protocol that would allow any host to send a special packet to block malicious traffic on the upstream provider or even at the source network. In this case networks like Cloudflare would become unnecessary.
- throwawayqqq11 3y agoISPs could enshitty-sell it though.
- supertrope 3y agoAltruism is not profitable.
- ilyt 3y agoThat costs a lot of money to implement. They are in business of selling pipes, not pipe filters
- Egrodo 3y agoIf it becomes this easy to block traffic couldn't malicious applications really mess up a user by spamming out reject packets for common IP?
- deleted 3y ago[deleted]
- Ajedi32 3y agoI think it would have to be something like "Block traffic from <offending IP> intended for <my IP>. <TTL>. <Cryptographic signature verifying that I control my IP>."
- codedokode 3y agoThe intermediate routers can send back a confirmation code, and you must send a new reject packet with this code to confirm the ban.
- 3y ago
- MuffinFlavored 3y agoHow large of a “botnet” did it take to cause 398M rps?
- lapcat 3y agoApparently only 20K: https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/ https://blog.cloudflare.com/technical-breakdown-http2-rapid-...
- MuffinFlavored 3y ago19.9k requests per second per machine That seems high/impressive I guess we don't have a good understanding if somebody had control of a blend of 10k VMs in the cloud, 10k infected PCs, etc.
- tommica 3y agoSorry, I forgot to break out of my while loop
- luckystarr 3y agoShould Google actually provide the DDoS initiators with debug information like this graph? Just thinking...
- zelon88 3y agoThe graph shows the initiators nothing they don't already know. What would be truly terrifying is if Google kept this a secret and let it sneak up on the world. I have zero faith in Google to do the right thing, and I have zero confidence in their impartiality, and zero confidence in them being the gatekeeper for internet standards. And that's WITH all these silly open self-congratulating blog posts. Google needs people to trust them in order for them to try to be gatekeeper of best practices and web standards. They want nothing more than to absorb the W3C. That will never happen if they can't convince the "professionals" to parrot everything they say. And they can't get anyone to parrot unless they write these self-congratulatory blog posts. Likewise, they need all the open-source developers who basically wrote Google's codebase to fix their code for them, for free of course. They won't do that unless Google tells them what's broken.
- tuananh 3y agoCloudflare blog: https://blog.cloudflare.com/zero-day-rapid-reset-http2-record-breaking-ddos-attack/ https://blog.cloudflare.com/zero-day-rapid-reset-http2-recor...
- icing0 3y agoYou can send thousands of such Request+RST to an Apache httpd per connection, but it will only work on 2 of them at a time. We are an old server. You have to be nice to us before we do more for you. -.-
- SigmundurM 3y agoAWS blog: https://aws.amazon.com/blogs/security/how-aws-protects-customers-from-ddos-events/ https://aws.amazon.com/blogs/security/how-aws-protects-custo...
- dominicdoty 3y agoCouldn't cloudflare show a page to the next handful of http requests from an IP informing the user that "something on your network is participating in DDoS attacks". All the big providers could do this, just inject a little turnstile like page in front of the next cloudflare site you visit. I would love to know if there's a compromised device on my network, and I don't have any real monitoring set up to detect it. It's not a full solution, but at least informing users there is a problem is a good start.
- BitPirate 3y agoIn the age of CGNAT? Not a good idea.
- averageRoyalty 3y ago> All the big providers could do this, just inject a little turnstile like page in front of the next cloudflare site you visit. Oh good. We can go back to the pre-HTTPS days where ISPs injected ads into HTML. Except this time we normalise it for the CDN provider.
- pythonguython 3y agoWho has an incentive to carry out these DDos attacks? Why would anyone be willing to spend large amounts of money and develop a sophisticated attack against corporate cloud infrastructure? It seems like the only reasonable answer is foreign governments. But still what is the result - you inconvenience American tech companies and their customers for a few hours? This happens all the time, so clearly someone finds it worthwhile. Can anyone help me understand?
- GartzenDeHaes 3y agoDDOS protection is a billion dollar a year industry.
- gloosx 3y agofew hours of inconvenience in corporate cloud infrastructure has a result also - large amounts of money lost. So most obvious incentive is simply ransom
- anonacct37 3y agoPR. Attack Google or cloudflare. Wait for them to publish a blog post about the biggest attack ever seen, then tell potential customers of your botnet that you can launch a bigger attack than anyone else and point to the above blog post.
- Jeff_Brown 3y agoThe botnet is probably the critical thing. Even if the PR (or "avenge the global south", or whatever) value might not be enormous, the cost to a bad actor of having other peoples' computers do something is almost negligible.
- endergen 3y agoDoesn't using your botnet expose your botnet IP addresses/devices?
- mrweasel 3y ago
- qaq 3y agoHmm wasn't there like 3.47Tbps attack on MS that's prob about same magnitude
- datadeft 3y ago398 million requests per second is really the largest DDOS attack to date?
- mgaunard 3y ago398M rps means a request every 2.5ns. Most likely the figure is incorrect, or at least misleading.
- sidcool 3y agoI am impressed from both parties, from a tech perspective. The attackers who exploited HTTP2, and Google that mitigated it. How did Google mitigate it exactly? Is there a technical deep dive?
- chillax 3y agoNot Google, but there is one over at Cloudflare - https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/ https://blog.cloudflare.com/technical-breakdown-http2-rapid-...
- theanonymousone 3y agoNovice here. Should I seek the patch from the web application framework I use (e.g. Jetty, Spring Boot)?
- AtNightWeCode 3y agoThis is just Google bs. There is no way in hell they can't mitigate anything at the edge of this nature. If this was a real problem it most likely originated from within GCP. The article does not even state where the traffic comes from. EDIT: Ok, so this was a 0-day issue. Then it all makes more sense. Sorry.
- dang 3y agoRelated ongoing threads: The novel HTTP/2 'Rapid Reset' DDoS attack - https://news.ycombinator.com/item?id=37830987 https://news.ycombinator.com/item?id=37830987 HTTP/2 Zero-Day Vulnerability Results in Record-Breaking DDoS Attacks - https://news.ycombinator.com/item?id=37830998 https://news.ycombinator.com/item?id=37830998
- metalrain 3y agoSo given that minimum size of HTTP request is something under 32 bytes, some sources say 18 bytes but let's be generous here HTTP can be complex. In theory I could send 4M requests in second with 1 Gbps connection. 1 Tbps total connection capacity could do close to 4.3B rps given perfect distribution. I imagine distributing the load becomes harder/more expensive the bigger the scale. In real network you probably have to find the paths that don't slow you down and then strike for short time before you are banned. Cloudflare has made post about mitigating 2 Tbps DDOS in 2021, so in real attacks request sizes aren't quite that optimal.
- 2OEH8eoCRo0 3y agoWho was the target?!
- glenngillen 3y ago> We noticed these attacks at the same time two other major industry players — Google and AWS — were seeing the same. Curious if there's anyone in the HN crowd that works at this level in one of the major vendors. What happens during an attack of this scale? Are there people from Cloudflare + Google + AWS on a live videoconference call co-ordinating with each other in real-time to mitigate it? Or is each vendor mostly observing from a distance what is happening elsewhere, and solely focussed on sorting their own problems out?
- menscher 3y agoWe typically fight our own fires, but if one of us sees something interesting/new we often ask others (after the fire is out) if anyone else saw a similar attack (which could be a new botnet, a new attack method, or whatever). In this case we realized we were all looking at the same thing (which could have huge impact on smaller targets), so collaborated on understanding the problem and coordinated the security response with all webserver vendors.
- deleted 3y ago[deleted]
- phendrenad2 3y agoAny info on where these attacks originate? I'm guessing it's hacked routers and IoT devices. I wonder if it's time for governments to start giving IoT vendors "DDoS scores" and mandating that they appear on the label.
- 1vuio0pswjnm7 3y ago"In the end, H2 [HTTP/2] is not much robust but each implementation has certain possibilities to cover some of the limitations and these differ due to many architectural constraints." "The good point in this is that this will probably make more people want to reconsider H3/QUIC [HTTP/3] if they don't trust their products anymore :-)" https://www.mail-archive.com/haproxy@formilux.org/msg44136.html https://www.mail-archive.com/haproxy@formilux.org/msg44136.h...