5 ms·

If we were pure web we would have ducked behind Cloudflare immediately. Since we do SMTP/IMAP/POP3 as well, we've had to go with a more complex (and costly) so

If we were pure web we would have ducked behind Cloudflare immediately. Since we do SMTP/IMAP/POP3 as well, we've had to go with a more complex (and costly) solution. This is our theory for why they're currently attacking email providers. We're not "just a web site", and attackers realise that the situation is more complex for email sites, we can't just hide behind Cloudflare. We're not sure who's actually attacking us, the ransom note comes from a freemail provider and a connection from a tor exit node. We can only guess at their total capabilities.

ramidarigaz 11y ago
Bummer to hear that :( Keep up the good work! I'm a very satisfied customer, and I wish you all the best.
andrem 11y ago
Sorry to hear about that... Do get in touch with one of the providers listed above, they may be able to help you out in the short term for free in exchange for publicity.
- brongondwana 11y ago
  Yep, thanks - we've already spoken to a few of them and got some things in place. Obviously don't want to spell out all our measures in public.
cat-dev-null 11y ago
0. Place an HA pfSense CARP or OpenBSD pf CARP setup as a pair of transparent proxies in front of everything (eg at the edge on the other side of HA network gear with either 2 (or 3, if deploying a private, admin network too) NIC teams for isolating traffic). This will let you do raw L3 traffic measurements on each side with graphite/collectd, cacti, rrdtool, etc. and L2/L3 IP/network banning (if you don't own/admin the network gear or don't want to touch it in production). These are super cheap and only need ~128 MiB RAM each and very little CPU and disk (except for logging, you want a dedicated PCIe SSD or SSD partition if possible). (Your public IP(s) should point to these boxen.) 1. Definitely get stuff behind reverse SMTP/IMAP/POP3 proxy like nginx or haproxy. nginx: Compile it from source if that's all you need, and reduce your attack surface. http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html haproxy: http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infrastructure-with-postfix-and-load-balancers/ http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infr... 2. Setup something like fail2ban: https://rtcamp.com/tutorials/nginx/fail2ban/ https://rtcamp.com/tutorials/nginx/fail2ban/ 3. There are many other tweaks and there are some appliancized VMs for anti-spam and DDoS that can be dropped behind the trusted network-side. (I would advise against Cloudflare-like services for most mature and non-web apps because they are add'l points of failure and increase latency, and they duplicate what good sys/netadmins implement routinely, especially if you're already deployed to multiple DCs servicing multiple continents and/or geodns.) Pedigree: I'm a founder and once-upon-a-time security researcher & sysadmin whom sold out and became SRE manager and then a consultant. I used to maintain multiple deployments of commercial Zimbra (from m&a activities) for clients including hi-ed, non-profits, VIP individuals, and enterprises.
- pyvpx 11y ago
  this is the first I've heard of being able to block traffic in excess of 10Gbps with "128MiB" and "very little CPU" in addition to Cloudflare-like services adding latency.
  - brongondwana 11y ago
    Yeah, it's small-time advice. Good advice for protection against complexity attacks, not so much for protection against tens of Gbps of random junk that fills your entire pipe. (we do run nginx on out frontend machines for both web and mail protocols, protecting the Cyrus servers behind it from complexity attacks and providing fan-out connection routing) We dropped all the DDoS packets at our edge firewall quite comfortably - users wouldn't have even noticed except that it filled up our incoming links, so packets started dropping. I'm really quite impressed at the tech which the big DDoS protection providers have for packet inspection and cleaning the feed before it reaches the end host. It does lower the overall egalitarianism of the internet to have to deploy defenses - we lower our overall routability to put these mega filters in front of incoming packets - but that's the reality of a world where fiends can control tens of thousands of boxes and have them spew traffic at any random network address. You need to filter out at the boundary. Nothing short of filtering beforehand can stop a channel from being filled if it gets more than its capacity per second of incoming packets.
- mjevans 11y ago
  That doesn't fix the actual source of the problem; if traffic ingress costs you money then they're literally burning your money via that attack of attrition. The way that most of the 'protection' sites work is that they host so much aggregate traffic that it still totals out to more than the incoming attacks/normal traffic and they can literally just eat it at almost no cost (their service is doing that, and some form of filtering to keep it from reaching the actual target; the 'extra' cost for the attack is almost nothing since the processing hardware is nearly fixed in cost). The only way to handle this from a peer to peer perspective is to be able to send the electronic equivalent of 'gag orders' at hosts/ranges that are misbehaving (and have them stick, either by the other edge or by upstream providers there of). Said orders wouldn't be enough, alone, to warrant quarantine from the Internet, however a number of different sources indicating infected behavior would be.

ramidarigaz 11y ago

Bummer to hear that :( Keep up the good work! I'm a very satisfied customer, and I wish you all the best.

andrem 11y ago

Sorry to hear about that... Do get in touch with one of the providers listed above, they may be able to help you out in the short term for free in exchange for publicity.

brongondwana 11y ago
Yep, thanks - we've already spoken to a few of them and got some things in place. Obviously don't want to spell out all our measures in public.

cat-dev-null 11y ago

0. Place an HA pfSense CARP or OpenBSD pf CARP setup as a pair of transparent proxies in front of everything (eg at the edge on the other side of HA network gear with either 2 (or 3, if deploying a private, admin network too) NIC teams for isolating traffic). This will let you do raw L3 traffic measurements on each side with graphite/collectd, cacti, rrdtool, etc. and L2/L3 IP/network banning (if you don't own/admin the network gear or don't want to touch it in production). These are super cheap and only need ~128 MiB RAM each and very little CPU and disk (except for logging, you want a dedicated PCIe SSD or SSD partition if possible). (Your public IP(s) should point to these boxen.) 1. Definitely get stuff behind reverse SMTP/IMAP/POP3 proxy like nginx or haproxy. nginx: Compile it from source if that's all you need, and reduce your attack surface. http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html http://nginx.org/en/docs/mail/ngx_mail_proxy_module.html haproxy: http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infrastructure-with-postfix-and-load-balancers/ http://blog.haproxy.com/2012/06/30/efficient-smtp-relay-infr... 2. Setup something like fail2ban: https://rtcamp.com/tutorials/nginx/fail2ban/ https://rtcamp.com/tutorials/nginx/fail2ban/ 3. There are many other tweaks and there are some appliancized VMs for anti-spam and DDoS that can be dropped behind the trusted network-side. (I would advise against Cloudflare-like services for most mature and non-web apps because they are add'l points of failure and increase latency, and they duplicate what good sys/netadmins implement routinely, especially if you're already deployed to multiple DCs servicing multiple continents and/or geodns.) Pedigree: I'm a founder and once-upon-a-time security researcher & sysadmin whom sold out and became SRE manager and then a consultant. I used to maintain multiple deployments of commercial Zimbra (from m&a activities) for clients including hi-ed, non-profits, VIP individuals, and enterprises.

pyvpx 11y ago
this is the first I've heard of being able to block traffic in excess of 10Gbps with "128MiB" and "very little CPU" in addition to Cloudflare-like services adding latency.
- brongondwana 11y ago
  Yeah, it's small-time advice. Good advice for protection against complexity attacks, not so much for protection against tens of Gbps of random junk that fills your entire pipe. (we do run nginx on out frontend machines for both web and mail protocols, protecting the Cyrus servers behind it from complexity attacks and providing fan-out connection routing) We dropped all the DDoS packets at our edge firewall quite comfortably - users wouldn't have even noticed except that it filled up our incoming links, so packets started dropping. I'm really quite impressed at the tech which the big DDoS protection providers have for packet inspection and cleaning the feed before it reaches the end host. It does lower the overall egalitarianism of the internet to have to deploy defenses - we lower our overall routability to put these mega filters in front of incoming packets - but that's the reality of a world where fiends can control tens of thousands of boxes and have them spew traffic at any random network address. You need to filter out at the boundary. Nothing short of filtering beforehand can stop a channel from being filled if it gets more than its capacity per second of incoming packets.
mjevans 11y ago
That doesn't fix the actual source of the problem; if traffic ingress costs you money then they're literally burning your money via that attack of attrition. The way that most of the 'protection' sites work is that they host so much aggregate traffic that it still totals out to more than the incoming attacks/normal traffic and they can literally just eat it at almost no cost (their service is doing that, and some form of filtering to keep it from reaching the actual target; the 'extra' cost for the attack is almost nothing since the processing hardware is nearly fixed in cost). The only way to handle this from a peer to peer perspective is to be able to send the electronic equivalent of 'gag orders' at hosts/ranges that are misbehaving (and have them stick, either by the other edge or by upstream providers there of). Said orders wouldn't be enough, alone, to warrant quarantine from the Internet, however a number of different sources indicating infected behavior would be.