4 ms·

0. Place an HA pfSense CARP or OpenBSD pf CARP setup as a pair of transparent proxies in front of everything (eg at the edge on the other side of HA network gea

0. Place an HA pfSense CARP or OpenBSD pf CARP setup as a pair of transparent proxies in front of everything (eg at the edge on the other side of HA network gear with either 2 (or 3, if deploying a private, admin network too) NIC teams for isolating traffic). This will let you do raw L3 traffic measurements on each side with graphite/collectd, cacti, rrdtool, etc. and L2/L3 IP/network banning (if you don't own/admin the network gear or don't want to touch it in production). These are super cheap and only need ~128 MiB RAM each and very little CPU and disk (except for logging, you want a dedicated PCIe SSD or SSD partition if possible). (Your public IP(s) should point to these boxen.) 1. Definitely get stuff behind reverse SMTP/IMAP/POP3 proxy like nginx or haproxy. nginx: Compile it from source if that's all you need, and reduce your attack surface. http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html haproxy: http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infrastructure-with-postfix-and-load-balancers/ http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infr... 2. Setup something like fail2ban: https://rtcamp.com/tutorials/nginx/fail2ban/ https://rtcamp.com/tutorials/nginx/fail2ban/ 3. There are many other tweaks and there are some appliancized VMs for anti-spam and DDoS that can be dropped behind the trusted network-side. (I would advise against Cloudflare-like services for most mature and non-web apps because they are add'l points of failure and increase latency, and they duplicate what good sys/netadmins implement routinely, especially if you're already deployed to multiple DCs servicing multiple continents and/or geodns.) Pedigree: I'm a founder and once-upon-a-time security researcher & sysadmin whom sold out and became SRE manager and then a consultant. I used to maintain multiple deployments of commercial Zimbra (from m&a activities) for clients including hi-ed, non-profits, VIP individuals, and enterprises.

pyvpx 11y ago
this is the first I've heard of being able to block traffic in excess of 10Gbps with "128MiB" and "very little CPU" in addition to Cloudflare-like services adding latency.
- brongondwana 11y ago
  Yeah, it's small-time advice. Good advice for protection against complexity attacks, not so much for protection against tens of Gbps of random junk that fills your entire pipe. (we do run nginx on out frontend machines for both web and mail protocols, protecting the Cyrus servers behind it from complexity attacks and providing fan-out connection routing) We dropped all the DDoS packets at our edge firewall quite comfortably - users wouldn't have even noticed except that it filled up our incoming links, so packets started dropping. I'm really quite impressed at the tech which the big DDoS protection providers have for packet inspection and cleaning the feed before it reaches the end host. It does lower the overall egalitarianism of the internet to have to deploy defenses - we lower our overall routability to put these mega filters in front of incoming packets - but that's the reality of a world where fiends can control tens of thousands of boxes and have them spew traffic at any random network address. You need to filter out at the boundary. Nothing short of filtering beforehand can stop a channel from being filled if it gets more than its capacity per second of incoming packets.
mjevans 11y ago
That doesn't fix the actual source of the problem; if traffic ingress costs you money then they're literally burning your money via that attack of attrition. The way that most of the 'protection' sites work is that they host so much aggregate traffic that it still totals out to more than the incoming attacks/normal traffic and they can literally just eat it at almost no cost (their service is doing that, and some form of filtering to keep it from reaching the actual target; the 'extra' cost for the attack is almost nothing since the processing hardware is nearly fixed in cost). The only way to handle this from a peer to peer perspective is to be able to send the electronic equivalent of 'gag orders' at hosts/ranges that are misbehaving (and have them stick, either by the other edge or by upstream providers there of). Said orders wouldn't be enough, alone, to warrant quarantine from the Internet, however a number of different sources indicating infected behavior would be.
feld 11y ago
This doesn't stop traffic from getting to you. If the attacker has more bandwidth than you, you lose. Complete outage. All your customers think you're down because they can't get packets through. You need the DDoS traffic filtered upstream, not stopped at your firewall. This is not a trivial problem to solve. If you're running BGP you can stop small-ish DoS attacks by setting up a blackhole BGP community that is propagated up to your providers. Any IPs you put in will no longer have their traffic forwarded. This doesn't work for DDOS which has countless IPs attacking you. You will have to blackhole yourself and take the target IP off the internet to stop wasting all your bandwidth. So the solution is to have it filtered upstream by someone who can clean and absorb the attack. It adds complexity to your network architecture (filtering provider has to announce your routes for you) and it's not cheap.

pyvpx 11y ago

this is the first I've heard of being able to block traffic in excess of 10Gbps with "128MiB" and "very little CPU" in addition to Cloudflare-like services adding latency.

brongondwana 11y ago
Yeah, it's small-time advice. Good advice for protection against complexity attacks, not so much for protection against tens of Gbps of random junk that fills your entire pipe. (we do run nginx on out frontend machines for both web and mail protocols, protecting the Cyrus servers behind it from complexity attacks and providing fan-out connection routing) We dropped all the DDoS packets at our edge firewall quite comfortably - users wouldn't have even noticed except that it filled up our incoming links, so packets started dropping. I'm really quite impressed at the tech which the big DDoS protection providers have for packet inspection and cleaning the feed before it reaches the end host. It does lower the overall egalitarianism of the internet to have to deploy defenses - we lower our overall routability to put these mega filters in front of incoming packets - but that's the reality of a world where fiends can control tens of thousands of boxes and have them spew traffic at any random network address. You need to filter out at the boundary. Nothing short of filtering beforehand can stop a channel from being filled if it gets more than its capacity per second of incoming packets.

mjevans 11y ago

That doesn't fix the actual source of the problem; if traffic ingress costs you money then they're literally burning your money via that attack of attrition. The way that most of the 'protection' sites work is that they host so much aggregate traffic that it still totals out to more than the incoming attacks/normal traffic and they can literally just eat it at almost no cost (their service is doing that, and some form of filtering to keep it from reaching the actual target; the 'extra' cost for the attack is almost nothing since the processing hardware is nearly fixed in cost). The only way to handle this from a peer to peer perspective is to be able to send the electronic equivalent of 'gag orders' at hosts/ranges that are misbehaving (and have them stick, either by the other edge or by upstream providers there of). Said orders wouldn't be enough, alone, to warrant quarantine from the Internet, however a number of different sources indicating infected behavior would be.

feld 11y ago

This doesn't stop traffic from getting to you. If the attacker has more bandwidth than you, you lose. Complete outage. All your customers think you're down because they can't get packets through. You need the DDoS traffic filtered upstream, not stopped at your firewall. This is not a trivial problem to solve. If you're running BGP you can stop small-ish DoS attacks by setting up a blackhole BGP community that is propagated up to your providers. Any IPs you put in will no longer have their traffic forwarded. This doesn't work for DDOS which has countless IPs attacking you. You will have to blackhole yourself and take the target IP off the internet to stop wasting all your bandwidth. So the solution is to have it filtered upstream by someone who can clean and absorb the attack. It adds complexity to your network architecture (filtering provider has to announce your routes for you) and it's not cheap.