40 ms·
Ask HN: How to survive indefinite DDoS attack?
Say I have a website that is under DDOS attack. Assume it will be going on indefinitely. Assume my hosting provider has DDOS protection that will protect my servers but at the same time users will be unable to access the website.
How can I survive being targeted like this? Cycling IP addresses and updating DNS ad infinitum does not seem like viable solution... because in the end, it's the network attacks that get you, not the application level attacks.
- LinuxBender 3y agoHow to survive indefinite DDoS attack? That probably depends on who your customers are. I can chime in one one scenario but maybe this does not help your case. If you are B2B then you could just shut off all public IP's and set up some Anycast VPN WAN accelerated meshes with your customers and use a wildcard sub-domain for each customer to access. They won't like having to set that up on their end but then they will have more resilience, privacy and security than they had before in that they can route around internet problems that would have otherwise caused outages or degraded services. Your company would have to provide professional services to help them with the setup and the customers would need discounts to offset the costs of the hardware assuming they do not already have it. The customers end-user devices would join their network over their VPN, not yours. For some B2B arrangements this isn't even a rounding error and removes scrubbing center costs and removes some SLA breach credits. Give this text to your principal network engineers, approve the spend and they will know what to do. For the customers that do not accept this option, update the contracts and SLA's, then give them a public end-point that has a "best effort" SLA or use a cheap CDN if your contracts permit sending your sensitive data through them. Incentivize them to move to your Anycast VPN WAN accelerated mesh with tighter SLA's and improved security posture by keeping sensitive data off third party data processors like CDN's. [Edit] This would also be a good time to use one of your companies sub-divisions or set up a new legal entity and get a new AS number with some dedicated IPv6 space and let your attackers just keep attacking the old address space. Announce the old IPv4 addresses on a throw away circuit to gather stats for law enforcement and network mailing lists. Try to do as much mesh peering with that new IPv6 space as you can. This is not security through obscurity so to speak, but rather a small piece of The Art of War. The marketing team will not be happy but they can distribute their sites to a myriad of CDN's on marketing specific domains for your company. Gotta keep 'em separated, set some smart cache control headers and hope for the best. An alternative to this semi-complex setup is to pay DDoS scrubbing centers to handle the routing of your datacenter traffic but that can hurt latency sensitive applications. This may be useful for the occasional extortion attack but would be very expensive and sub-optimial(c) in a never-ending indefinite DDoS attack.
- Thaxll 3y agoPay couple of $ for Cloudflare be done with it. https://www.cloudflare.com/ddos/ https://www.cloudflare.com/ddos/ Don't forget to have a fresh IP before doing the move so that your origin is unknown.
- fvv 3y agobut is it far-sighted that to keep the internet as free as possible there are no ways to mitigate a ddos, and at the same time this freedom is completely lost when the companies that protect against ddos can be counted on the fingers of one hand?
- djbusby 3y agoI hate that this is the right answer. I hate that I use it too. I hate that I don't have time/cash to explore alternatives.
- fvv 3y agobecause alternatives usually cost several thousands $/months.. what has the internet become? Why not create blacklists (even on a timed basis) with the IPs to block at BGP level? and these blacklists managed by bodies, - okay they are bodies..., but at least they can be open foundations to which it is not mandatory to join (as in the case of cloudflare)- available to BGP backbones and routed on a network, castrate the blacklisted traffic? maybe even at host level be able to make one IP available for normal traffic and possibly one for blacklist traffic using BGP routings to which the blacklist tables are applied?
- tutfbhuf 3y agoHave you considered placing your website behind Cloudflare? This solution resolved my DDOS issues. Previously, I was regularly dealing dealing with over 20 Gbps and 5M pps.
- StevenWaterman 3y agoAssuming this is an API and not a website, my approach would be (other than just paying cloudflare to handle it). Application servers that aren't listed on DNS and their IPs kept secret. A _lot_ of proxy servers that take requests, check the auth, and forward the valid requests to the application servers. For invalid requests, just let them time out. If checking the auth is expensive enough to cause problems, require some proof-of-work in the request too, and only check the auth if the PoW is there. In other words, keep all the DDOS traffic off the application servers. Have sacrificial proxies. Make it cheap to handle the DDOS traffic and expensive to generate the DDOS traffic. And then just process the traffic. Spread the proxies across multiple cloud providers if you need to - but ideally ones with free ingress.
- pjc50 3y agoThat's "building your own cloudflare". > Make it cheap to handle the DDOS traffic and expensive to generate the DDOS traffic The traffic is from compromised devices, so it's very cheap.
- 0xEFF 3y agoEven so, the suggestion is a good one. Modern proxies have gotten much better at handling auth and processing the traffic as the GP describes. Service meshes have these features built in now when a few years ago we had to roll our own solutions with haproxy or nginx.
- StevenWaterman 3y agoIf you can make it take 10x as long to send a request, the DDOS attack becomes 1/10 the size.
- pjc50 3y agoOnly for some categories - SYN floods aren't limited by responses at all.
- dewey 3y agoThat sounds fun for an engineer but if it’s a business then you should focus on your core business and not trying to become a anti-ddos service.
- claudiulodro 3y agoCloudflare is the simplest solution, but for the sake of treating this like a system design exercise, can you reasonably assume that there is not an infinite supply of devices with which to conduct the attack, fingerprint the devices, and block them?
- pjc50 3y agoDepending on the weight of DDoS, you may simply find it saturates your upstream router so very little valid traffic even reaches you. Or your ISP kicks in their own protection and drops all your traffic. Edit: https://www.cloudflare.com/learning/ddos/famous-ddos-attacks/ https://www.cloudflare.com/learning/ddos/famous-ddos-attacks... largest ever seems to have been about 180k devices. Getting your upstream to accept that many firewall entries is a big problem.
- bityard 3y agoThe first "D" in "DDos" means "distributed," which means there _effectively is_ an infinite supply of devices with which to conduct the attack. It's a hard problem, which is why DDoS attacks are so prevalent and difficult to defend against without sophisticated mitigation devices and services. You have to block the "bad" traffic by some characteristic and many DDoS attackers have gotten _very_ good at making their attack traffic look almost identical to legitimate traffic.
- bityard 3y agoI work for a company in the DDoS mitigation space and there is not nearly enough information in your question for anyone to offer any kind of sensible response for your particular situation. What is your business? How much traffic "normal" do you get? What is the size of the attack? What is the bandwidth of your upstream connection? Who are your customers? Where is it hosted? What are your acceptable thresholds for false negatives and positives? Do you know who is attacking you and why? Most every hosting provider will have some sort of DDoS monitoring and mitigation on their networks already. Their response to sustained or repeated attacks might range from scrubbing the bad traffic before it gets to you and not notifying you at all, to reaching out to you to work with you on both ends of the issue, to cancelling your account. If you just have a fairly simple website that you host yourself, Cloudflare likely a fine option. If you have more advanced needs, you should talk to a more comprehensive DDoS solution vendor.
- bosch_mind 3y agoWho would be more advanced than Cloudflare?
- rattlyy 3y agoDDoS-Guard, it's basically cloudflare russia edition that is used by controversial websites like Parler, 8chan etc that will get ddosed indefinitely.
- thesh4d0w 3y agoI think it's reasonable to call them "less ethical" rather than "more advanced" Technology wise, cloudflare is probably the leader.
- jackconsidine 3y agoPeople mention Cloudflare. Google Cloud Armor is another option (we looked into both). A colleague was DDoS attacked and was able to move the service behind Cloud Armor in a few hours, effectively ending a ridiculous barrage of network requests. I think CA charges a minuscule amount per request, and can be capped with a fix monthly cost which limits the DDoS liability.
- hknmtt 3y agobut all how are all these services able to let in valid traffic? won't they simply cut me off altogether to protect my infrastructure at the cost of no access for my users?
- pjc50 3y agoThe entire business model is in not cutting you off, but cutting off the requests. E.g. by forwarding them to the cloudflare captcha first. This does mean occasional false positives.
- kijin 3y agoUnfortunately, the captcha is a deal breaker for anyone who is running APIs instead of regular, boring, human-accessible websites. We can barely tell humans and robots apart, and have no idea how to tell good robots from bad robots.
- ElectricalUnion 3y agoThe idea is that a IP under heavy DDOS already has effectively 0% availability anyways. More that 0% is a infinite availability increase.
- josevalerio 3y agoAs others have stated, Cloudflare is the best solution here. Other services like AWS WAF will charge you just for validating requests, which means even if the DDOS is mitigated, you're still paying for it to inspect each request and then block it, which now turns it to a denial of wallet attack.
- q3k 3y agoNetwork level or application level DoS? Those are two vastly different beasts.
- remram 3y agoI think about this sometimes. Shouldn't there be a way for your app server to report to your DoS protection service which requests are abusive and which are legit? Like a fail2ban but distributed: if someone connects and sends invalid credentials, block them at the ISP or proxy level, not on your host's firewall. It's possible that this doesn't exist for good reasons though, I am not sure how the numbers work out in terms of relative capacity.
- bashy 3y agoIf your provider has an API for the firewall/protection, just use fail2ban with cURL to block certain traffic. This can be done with Cloudflare using this action: https://github.com/fail2ban/fail2ban/blob/master/config/action.d/cloudflare.conf https://github.com/fail2ban/fail2ban/blob/master/config/acti...
- supriyo-biswas 3y agoThere are lots of posts here about using Cloudflare. There's also X4B[1] and Cohmbaton[2] in case you're looking for a non-Cloudflare alternative. You've mentioned that your hosting provider has DDOS protection (which is usually Voxility[3]/Path[4]) will usually send traffic to your VPS through their scrubbing centers, taking care of the L4 attacks. Ratelimiting and aggressive caching can take care of the rest. If you're completely into this on your own, consider setting up a number of PoPs which relay traffic to your backend servers, and these PoPs should drop UDP traffic dropped the prerouting level to drop packets without high CPU consumption. There's also eBPF/XDP, but those require much more engineering work and a compatible network card that can run a eBPF program, otherwise you're still using the CPU to drop packets. [1] https://www.x4b.net https://www.x4b.net [2] https://www.combahton.net https://www.combahton.net [3] https://www.voxility.com https://www.voxility.com [4] https://path.net https://path.net
- gwnywg 3y agoI'm on Linode since ~2012 and some time ago they added free ddos prottection to their offering, it was advertised on HN too [1]. Hetzner is also offering something to mitigate ddos. There was HN thread asking the same question [2] Having said this, none of my services were targetted by ddos, it made me thinking that maybe maintaining a list of 'healthy' clients IP addresses during good time and then banning anything from outside of that list in bad times? Sounds lame I know, I wait for ideas emerging in this thread :) [1] https://news.ycombinator.com/item?id=22144369 https://news.ycombinator.com/item?id=22144369 [2] https://news.ycombinator.com/item?id=12376596 https://news.ycombinator.com/item?id=12376596
- sredevops01 3y agoYeah essentially this. Then have something crawl your database and find IP's that are crawling your dummy pages, and block those. Most of it is EC2/GCP instances and Azure VMs that people spin up with stolen cards, so you have to block a lot of 3rd party vendors. OVH and some others came up often. Lot's of crawling companies were using end user VPNs, so those are harder to block. The best thing I found was dummy pages to block IPs of bad actors. Also, serving different urls with JS enabled versus disabled, but showing your page as something that works without JS. Unfortunately, as good as CloudFlare is, their layer-7 isn't going to help you if someone is targeting you.
- kijin 3y agoCloudflare's layer-7 protection is crap, but it's still orders of magnitude more effective than anything Linode or Hetzner can pull off. Any major cloud or datacenter can block an old-fashioned UDP flood these days, but botnets have evolved too. Now they speak TLS and HTTP/2, and can send (relatively) small amounts of traffic to select endpoints to generate a large load. In addition to blocking layer-3 and layer-4 floods, the DDoS mitigation service needs to MITM all your layer-7 traffic in order to determine which requests are legit. Cloudflare can do this (to some extent). AWS WAF can do this. Regular hosting companies can't, unless you use their load balancer and let them manage your TLS keys for you.
- 3y ago
- deleted 3y ago[deleted]
- andrewmcwatters 3y agoMy question is, what did people do before Cloudflare? There are different types of DDoS attacks. I vaguely remember what we did over a decade ago, which was blackhole IPs. I'm not sure if people still do this directly with iptables, but at the time, this was the bread-and-butter approach for transport DDoS. If your issue was web application related, then you'd move up to fail2ban. DDoS protection for web applications is a different beast than raw transport. If you're a vicim of an amplified DDoS attack, your filtering may need to occur off the server.
- LinuxBender 3y agowhat did people do before Cloudflare? DDoS scrubbing centers. They are still used today but add some latency and are very expensive. There are also some expensive appliances that learn legit traffic and can be put into a defense mode that drops anything it did not learn at a certain volume. CDN's have incentives to be more cost effective as they want to have your traffic 100% of the time. DDoS scrubbing centers are on-demand via BGP changes.
- snorkel 3y agoLookup edge service providers like Cloudflare or Akamia. If you’re hosted by a large cloud provider then look into their free and premium edge services offerings. Traffic can be filtered to allow legit users, and consider scaling up servers too if cost of downtime outweighs cost of edge services and scaling up.
- tamimio 3y agoWhat kind of DDOS first, then you need to try to isolate the issue, is it from specific region/country? Can you block an IP range? Sometimes you might have a vulnerable service that allow such attacks, and then look into some providers that might help.
- avipars 3y agoCloudflare is a great and relatively easy to implement solution. You can enable bot-mitigation and reCaptchas which will help a lot
- johnklos 3y agoInformation would be nice. You can't properly stop a DDoS if you don't know anything about it.
- dearroy 3y agoGiven the frequent mentions of Cloudflare in this post, I'd like to offer an alternative suggestion. Consider migrating your website to web hosting providers renowned for their DDoS protection, such as OVH or Path. By doing so, you can ensure that your server performance remains unaffected even during ongoing attacks.
- cosmicglobal 3y ago[dead]