7 ms·

Is the attack on its webmail/website or on their smtp servers? I’ve been wondering about this, but how does one protect smtp servers from distributed attacks? L

Is the attack on its webmail/website or on their smtp servers? I’ve been wondering about this, but how does one protect smtp servers from distributed attacks? Let’s assume smaller attackers, do you just need good firewalls in front of your servers to prevent congestion to the smtp servers? Are there off the shelf tools that can be configured to help here (pf maybe)? Do tools like fail2ban help? For context, I’ve picked up self-hosting again after many years and though I don’t anticipate being a target for large attacks I have been curious what tools individuals have at their disposal or if it’s a fools errand to even try.

LinuxBender 5y ago
> Is the attack on its webmail/website or on their smtp servers? It is an attack on the httpd and smtpd daemons. imap has been unaffected as far as I can tell. > how does one protect smtp servers from distributed attacks? By design, MX can be as distributed and in large number as you can afford or as willing to spend. This can be a combination of load balancer virtual IP's distributing load to many MX servers behind it and many MX DNS records with the same or different priorities. This of course won't help much if the people attacking are paying ddos-as-a-service farms to bring on massive volume and packet rates that overload all your servers. There are DDoS scrubbing services you can pay for that will advertise your AS number or use GRE tunnels or VPN's to clean the attack data for you. These scrubbing solutions are no guarantee of mitigation. > Do tools like fail2ban help? No. That would be pointless whack-a-mole. If an individual person is mad at you and launching a tool from their own PC or a handful of VM's, then yes fail2ban will help. Blocking individual IP's on your MX servers under a real distributed DDoS attack would be futile. Scrubbing centers are about the only solution once the attack is big enough. Or if you had unlimited funds you could deploy many datacenters or point-of-pressence destinations and build your own scrubbing networks but that is very expensive.
- rgrmrts 5y ago
  Thanks for your response! So using specific examples here for smtp, I get a 1gbps guaranteed network from Hetzner so in theory I’d need to distribute over 50 servers to withstand this attack? It’s not clear to me why fail2ban wouldn’t at least help, if the botnet is a thousand machines wouldn’t I (eventually) have them all blocked? And therefore reduce the overall duration of the attack? Or is the problem that it’s hard to differentiate between good clients and bad clients because no single client is sending enough traffic to be suspicious? Also, do you have any specific examples of ddos scrubbing services? Would like to take a look specifically at affordability for individuals.
  - deadbunny 5y ago
    You're thinking too small. DDoSaaS use hundreds of thousands to millions of compromised machines* to attack. You might only see a handful of connections from each IP. Whch makes it fruitless to use something like f2b. * Machines here could be anything from a lightbulb to a server.
  - mirashii 5y ago
    The problem is that fail2ban or any sort of dropping of packets at the host is too late, the volume has overloaded the link.
  - LinuxBender 5y ago
    In regards to fail2ban, assuming the attack was purely SMTP specific which it won't be even then blocking IP's would be futile. The DDoS-as-a-service farms have hundreds of thousands to millions of IP's under their control. Block one IP and ten more show up. One IP is not one attacker. Those farms have probes that can tell how effective their attack is. Some of them even have "proven work" that is reported back to the buyer to validate the effectiveness of their attack. You may want to research this one as it is a very big topic. In reality however, a large portion of the attack won't even show up in logs. The attack will also contain depending on how much the attacker is willing to spend tens of millions or hundreds of millions of packets per second of TCP, UDP packets on random ports, no ports, random protocols, random sizes, random TTL's, random headers. That is a volumetric attack. Fail2ban and most network IDS/IDP's would not even see this attack. It would saturate the uplinks to the ISP before you even see anything. Your ISP will most likely null route you and encourage you to stop advertising your AS number. In reference to scrubbing centers for individuals, that is not a thing unless you have unlimited funds. There are some VPS providers that can scrub tiny attacks. Linode, Vultr, OVH to name a few but they can only deal with tiny attacks. If you want to research this for your business, my suggestion would be to get on the NANOG mailing lists and discuss it with all the network engineers to find out which scrubbing services are currently most effective. This is a moving target and a very big investment. I am not a fan of any of the companies that provide these services, but that is only based on my limited experiences with them.
    - 5y ago
- tgsovlerkhgsel 5y ago
  I agree that you do need some form of first stage that can take the traffic volume (usually a third party service, but could also be 100+ Gbps of bandwidth you get somewhere and an appliance that can do IP filtering), no good way around that. But once you have that, you should be able to tell them "forward me all traffic except <list of IPs>" and then list the IPs that are sending you the most (remaining) traffic, or even cost (e.g. if the IP is sending little traffic, but performing many TLS handshakes). That's where a fail2ban like tool would come in, no? The benefit of this approach is that it works completely independently of the protocol you're running. TCP, UDP, doesn't matter, as long as the attacker cannot spoof IP addresses at scale.
  - LinuxBender 5y ago
    Agreed adding bandwidth can help against some attacks. as long as the attacker cannot spoof IP addresses at scale That is kindof the rub. Until a majority of tier-1 backbone providers implement bcp38 [1] or some derivative of it, spoofing from the ddos farms is trivial. There has been talk of implementing this for many years but very little action. Perhaps when DDoS attacks cost enough tax revenue or impact investors, perhaps there may be push for legislation in some countries to implement but in an ideal world most of the providers would work through this as one big team. I just make some network engineers laugh, or smirk, or other [1] - https://en.wikipedia.org/wiki/Ingress_filtering#Networks https://en.wikipedia.org/wiki/Ingress_filtering#Networks
    - elaus 5y ago
      Curious as you speak of "very little action", yet the Wikipedia article you linked says: > […] contrary to general opinion about the lack of BCP 38 deployment, some 80% of the Internet (by various measures) were already applying anti-spoofing packet filtering I know very little about global routing, so I'm wondering if it's an ineffective way of filtering against these attacks we see today or is 80% not enough to be effective?
      - LinuxBender 5y ago
        It really isn't enough. Everyone needs to do their part [1] not just ISP's but also large businesses, cloud providers, others. And this has to occur in all regions. I admittedly over-simplified the problem statement. [1] - https://archive.nanog.org/sites/default/files/1_Kaeo_Ddos_Trends.pdf https://archive.nanog.org/sites/default/files/1_Kaeo_Ddos_Tr... [PDF]
cotillion 5y ago
At this scale you pretty much need to apply some sort of DDoS scrubbing service. Your ISP might already have one they can route traffic through or if you have your own AS you can let a DDoS service announce the target prefixes. A game i run was recently hit with a 102 Ggbps CLDAP reflection attack. We were down for a while until our ISPs DDoS protection detected it after that we were mostly unaffected. If the attack is difficult to separate from legitimate traffic you'll still suffer though.
noaheverett 5y ago
If they're hitting the outbound SMTP servers, there's no way (at least that I know of) to protect the IP/servers via obfuscation with a service like Cloudflare etc. Email deliverability relies heavily on the source IP of the sending SMTP for reputation and is going to be viewable in the headers of an email. Also changing IPs isn't a small task unless you're sitting on a load of good reputation IPs that are pre-warmed up. I think a scrubbing service would be the only way to help or of course, having enough resources to deal with it directly (bandwidth, cpu, etc). Disclaimer: I run Pingly [1] an email hosting service, but ironically our signups are turned off at the moment due to a botnet that hits us with fake accounts to send spam that I'm working to mitigate completely. [1] https://pingly.com https://pingly.com
- bob1029 5y ago
  Cloudflare has a product where you don't actually need any public IPs to host your apps. You install a daemon on each server and firewall off the box. It makes it virtually impossible for someone to get around the DDOS protection. https://developers.cloudflare.com/cloudflare-one/connections/connect-apps https://developers.cloudflare.com/cloudflare-one/connections...
lxchase 5y ago
I’ve been a server operator and was subject to DDoS attacks. I’ve also tried self hosting since I am familiar with the challenges. Short answer is 80% of the time you will be able to block a <10gbps attack (assuming that’s your uplink speed) at the node level. Most boosters all use similar attack methods, and it just requires you to drop packets as upstream as possible (preferably on the NIC as a hardware upload). In most cases you can block common SOURCE ports of packets. Note, there may be legitimate uses for some protocols, but I doubt most customers use them. If the attack is >10gbps you must likely need a BGP scrubber, that can divert your traffic and “tank” any bandwidth before it comes to your line lease. If your uplink is saturated it doesn’t matter how much your hardware can filter, good traffic won’t be able to get through. There are theoretical cheap ways where you can assist your customer subject to attacks to setup a GRE tunnel to act as that scrubbing however it’s a bit annoying. Get a VM on a host that has that capability already, and route your packets through them first. Con is you have one more failure mode and increased latency.
- lxchase 5y ago
  To add to what I said, the general options and availability on the stack is something like: 1. Network level: - BGP Filtering (meaning in order to reach your IP range, packets must first route through another company that has sufficient bandwidth to receive AND filter out bad packets), then clean traffic will go through to your data center, and to your server. - GRE Tunnels, Similar to above, but it will not be transparent in that you will likely use their IPs. 2. Datacenter level: - Colocate or find a dedicated server that sits behind a dedicated appliance that solely exists to act as a filter. You will also need to ask what their upstream link speeds are (i.e. 40gbps to that appliance). You still might encounter leaks, and rely on the fact that they have configured correctly or are willing to apply custom filtering if you have an advanced attack. 3. Node/Server level: - By now, you can only filter whatever your line rate is (i.e. 1gbps or 10gbps most likely). There are various methods, but all of them require to create custom filters and be active in patching up leaks. You'll want to do it as far up in the server stack as possible. Best option is SmartNICs or NICs that support hardware offloads. Second best option is in front of IPtables. Most tutorials online talk about IPtables. That assumes you have Linux first, and second not the optimal way. Use tc (traffic control) instead, it's further up on the network stack.

LinuxBender 5y ago

> Is the attack on its webmail/website or on their smtp servers? It is an attack on the httpd and smtpd daemons. imap has been unaffected as far as I can tell. > how does one protect smtp servers from distributed attacks? By design, MX can be as distributed and in large number as you can afford or as willing to spend. This can be a combination of load balancer virtual IP's distributing load to many MX servers behind it and many MX DNS records with the same or different priorities. This of course won't help much if the people attacking are paying ddos-as-a-service farms to bring on massive volume and packet rates that overload all your servers. There are DDoS scrubbing services you can pay for that will advertise your AS number or use GRE tunnels or VPN's to clean the attack data for you. These scrubbing solutions are no guarantee of mitigation. > Do tools like fail2ban help? No. That would be pointless whack-a-mole. If an individual person is mad at you and launching a tool from their own PC or a handful of VM's, then yes fail2ban will help. Blocking individual IP's on your MX servers under a real distributed DDoS attack would be futile. Scrubbing centers are about the only solution once the attack is big enough. Or if you had unlimited funds you could deploy many datacenters or point-of-pressence destinations and build your own scrubbing networks but that is very expensive.

rgrmrts 5y ago
Thanks for your response! So using specific examples here for smtp, I get a 1gbps guaranteed network from Hetzner so in theory I’d need to distribute over 50 servers to withstand this attack? It’s not clear to me why fail2ban wouldn’t at least help, if the botnet is a thousand machines wouldn’t I (eventually) have them all blocked? And therefore reduce the overall duration of the attack? Or is the problem that it’s hard to differentiate between good clients and bad clients because no single client is sending enough traffic to be suspicious? Also, do you have any specific examples of ddos scrubbing services? Would like to take a look specifically at affordability for individuals.
- deadbunny 5y ago
  You're thinking too small. DDoSaaS use hundreds of thousands to millions of compromised machines* to attack. You might only see a handful of connections from each IP. Whch makes it fruitless to use something like f2b. * Machines here could be anything from a lightbulb to a server.
- mirashii 5y ago
  The problem is that fail2ban or any sort of dropping of packets at the host is too late, the volume has overloaded the link.
- LinuxBender 5y ago
  In regards to fail2ban, assuming the attack was purely SMTP specific which it won't be even then blocking IP's would be futile. The DDoS-as-a-service farms have hundreds of thousands to millions of IP's under their control. Block one IP and ten more show up. One IP is not one attacker. Those farms have probes that can tell how effective their attack is. Some of them even have "proven work" that is reported back to the buyer to validate the effectiveness of their attack. You may want to research this one as it is a very big topic. In reality however, a large portion of the attack won't even show up in logs. The attack will also contain depending on how much the attacker is willing to spend tens of millions or hundreds of millions of packets per second of TCP, UDP packets on random ports, no ports, random protocols, random sizes, random TTL's, random headers. That is a volumetric attack. Fail2ban and most network IDS/IDP's would not even see this attack. It would saturate the uplinks to the ISP before you even see anything. Your ISP will most likely null route you and encourage you to stop advertising your AS number. In reference to scrubbing centers for individuals, that is not a thing unless you have unlimited funds. There are some VPS providers that can scrub tiny attacks. Linode, Vultr, OVH to name a few but they can only deal with tiny attacks. If you want to research this for your business, my suggestion would be to get on the NANOG mailing lists and discuss it with all the network engineers to find out which scrubbing services are currently most effective. This is a moving target and a very big investment. I am not a fan of any of the companies that provide these services, but that is only based on my limited experiences with them.
  - 5y ago
tgsovlerkhgsel 5y ago
I agree that you do need some form of first stage that can take the traffic volume (usually a third party service, but could also be 100+ Gbps of bandwidth you get somewhere and an appliance that can do IP filtering), no good way around that. But once you have that, you should be able to tell them "forward me all traffic except <list of IPs>" and then list the IPs that are sending you the most (remaining) traffic, or even cost (e.g. if the IP is sending little traffic, but performing many TLS handshakes). That's where a fail2ban like tool would come in, no? The benefit of this approach is that it works completely independently of the protocol you're running. TCP, UDP, doesn't matter, as long as the attacker cannot spoof IP addresses at scale.
- LinuxBender 5y ago
  Agreed adding bandwidth can help against some attacks. as long as the attacker cannot spoof IP addresses at scale That is kindof the rub. Until a majority of tier-1 backbone providers implement bcp38 [1] or some derivative of it, spoofing from the ddos farms is trivial. There has been talk of implementing this for many years but very little action. Perhaps when DDoS attacks cost enough tax revenue or impact investors, perhaps there may be push for legislation in some countries to implement but in an ideal world most of the providers would work through this as one big team. I just make some network engineers laugh, or smirk, or other [1] - https://en.wikipedia.org/wiki/Ingress_filtering#Networks https://en.wikipedia.org/wiki/Ingress_filtering#Networks
  - elaus 5y ago
    Curious as you speak of "very little action", yet the Wikipedia article you linked says: > […] contrary to general opinion about the lack of BCP 38 deployment, some 80% of the Internet (by various measures) were already applying anti-spoofing packet filtering I know very little about global routing, so I'm wondering if it's an ineffective way of filtering against these attacks we see today or is 80% not enough to be effective?
    - LinuxBender 5y ago
      It really isn't enough. Everyone needs to do their part [1] not just ISP's but also large businesses, cloud providers, others. And this has to occur in all regions. I admittedly over-simplified the problem statement. [1] - https://archive.nanog.org/sites/default/files/1_Kaeo_Ddos_Trends.pdf https://archive.nanog.org/sites/default/files/1_Kaeo_Ddos_Tr... [PDF]

cotillion 5y ago

At this scale you pretty much need to apply some sort of DDoS scrubbing service. Your ISP might already have one they can route traffic through or if you have your own AS you can let a DDoS service announce the target prefixes. A game i run was recently hit with a 102 Ggbps CLDAP reflection attack. We were down for a while until our ISPs DDoS protection detected it after that we were mostly unaffected. If the attack is difficult to separate from legitimate traffic you'll still suffer though.

noaheverett 5y ago

If they're hitting the outbound SMTP servers, there's no way (at least that I know of) to protect the IP/servers via obfuscation with a service like Cloudflare etc. Email deliverability relies heavily on the source IP of the sending SMTP for reputation and is going to be viewable in the headers of an email. Also changing IPs isn't a small task unless you're sitting on a load of good reputation IPs that are pre-warmed up. I think a scrubbing service would be the only way to help or of course, having enough resources to deal with it directly (bandwidth, cpu, etc). Disclaimer: I run Pingly [1] an email hosting service, but ironically our signups are turned off at the moment due to a botnet that hits us with fake accounts to send spam that I'm working to mitigate completely. [1] https://pingly.com https://pingly.com

bob1029 5y ago
Cloudflare has a product where you don't actually need any public IPs to host your apps. You install a daemon on each server and firewall off the box. It makes it virtually impossible for someone to get around the DDOS protection. https://developers.cloudflare.com/cloudflare-one/connections/connect-apps https://developers.cloudflare.com/cloudflare-one/connections...

lxchase 5y ago

I’ve been a server operator and was subject to DDoS attacks. I’ve also tried self hosting since I am familiar with the challenges. Short answer is 80% of the time you will be able to block a <10gbps attack (assuming that’s your uplink speed) at the node level. Most boosters all use similar attack methods, and it just requires you to drop packets as upstream as possible (preferably on the NIC as a hardware upload). In most cases you can block common SOURCE ports of packets. Note, there may be legitimate uses for some protocols, but I doubt most customers use them. If the attack is >10gbps you must likely need a BGP scrubber, that can divert your traffic and “tank” any bandwidth before it comes to your line lease. If your uplink is saturated it doesn’t matter how much your hardware can filter, good traffic won’t be able to get through. There are theoretical cheap ways where you can assist your customer subject to attacks to setup a GRE tunnel to act as that scrubbing however it’s a bit annoying. Get a VM on a host that has that capability already, and route your packets through them first. Con is you have one more failure mode and increased latency.

lxchase 5y ago
To add to what I said, the general options and availability on the stack is something like: 1. Network level: - BGP Filtering (meaning in order to reach your IP range, packets must first route through another company that has sufficient bandwidth to receive AND filter out bad packets), then clean traffic will go through to your data center, and to your server. - GRE Tunnels, Similar to above, but it will not be transparent in that you will likely use their IPs. 2. Datacenter level: - Colocate or find a dedicated server that sits behind a dedicated appliance that solely exists to act as a filter. You will also need to ask what their upstream link speeds are (i.e. 40gbps to that appliance). You still might encounter leaks, and rely on the fact that they have configured correctly or are willing to apply custom filtering if you have an advanced attack. 3. Node/Server level: - By now, you can only filter whatever your line rate is (i.e. 1gbps or 10gbps most likely). There are various methods, but all of them require to create custom filters and be active in patching up leaks. You'll want to do it as far up in the server stack as possible. Best option is SmartNICs or NICs that support hardware offloads. Second best option is in front of IPtables. Most tutorials online talk about IPtables. That assumes you have Linux first, and second not the optimal way. Use tc (traffic control) instead, it's further up on the network stack.