13 ms·

Fastmail, Runbox, and Posteo under DDoS extortion attack

291 points by conjuredbytes 5y ago

deleted 5y ago
[deleted]
ryantgtg 5y ago
Wonder if it’s the same folks who hit voip.ms last week. That one lasted a while.
- 29083011397778 5y ago
  Reading through r/VOIP, I could have sworn Voip.ms was 2 weeks ago - then other providers began to be hit with DDoS attacks. Frankly, it doesn't seem like such a bad strategy. With the amount of panic the attack on Voip.ms caused, a number of people were discussing switching voip providers. And switching targets to other providers just as they get a massive influx of new customers seems like a force multiplier IMO.
sysadm1n 5y ago
> Since these DDoS attacks started we have worked with our system administrators and Internet Service Provider to mitigate the attacks How do you mitigate such an attack though? I know Cloudflare can stop this, but how do you create your own bespoke 'DDOS mitigation' tool, and what does that look like?
- capableweb 5y ago
  One way is to build your software on top of distributed/content-addressed P2P software (not Blockchain, but pure P2P). The angle of attack disappears completely then. Otherwise, with centralized infrastructure, you identify the bad traffic and send it elsewhere, an instance that keeps connections open but delivers nothing and have a small amount of resource usage. Problem is that you still need to be able to handle the traffic, but at least you avoid hitting your main infrastructure.
  - Mister_Snuggles 5y ago
    > One way is to build your software on top of distributed/content-addressed P2P software (not Blockchain, but pure P2P). The angle of attack disappears completely then. I can see how this works for some applications, but how would this work for SMTP and IMAP/JMAP?
    - capableweb 5y ago
      You can't force centralized/federated protocols into distributed protocols so you wouldn't.
    - a1369209993 5y ago
      > how would this work for SMTP You can actually get this semi-transparently with a TOR-like system. (I'm not sure TOR itself actually implements all of this, and also TOR is perennially underprovisioned for political reasons, so this mostly won't work in practice.) First, you need a distributed DNS mechanism to publish "example.com. 9999 IN MX 1 abcdefghijklmnop.onion". This is mostly static, so DDOS doesn't really work. You then come up with a sequence of rendezvous servers, which we'll number starting from 1. (I think actual TOR just assumes a single (or few) rendezvous server is sufficient, but I'd need to go dig through the code to be sure.) You then try (in sequence) servers 1,2+rand(2),4+rand(4),...,2^k+rand(2^k),... where rand(x) picks a random number in [0,x). (0: I'm not sure whether distinct rands should share lower bits; see below.) If a server is overloaded it just drops traffic on the floor. The destination server then checks the rendezvous servers in sequence until it's gotten enough successes that any client would have tried one of the successful rendezvous servers in its random sequence ([0] above affects the distribution here). Under heavy load, the destination server also sets a proof of work requirement (clientHello statisfies the standard hash-has-x-leading-zeros), which allows the rendezvous servers to drop most of the incoming traffic. Legitimate clients by definition are not spamming connections as fast as they can, so they can burn CPU to meet this requirement. DDOS clients can also burn CPU on this, but that reduces the rate at which they generate traffic. The end result is volumetric attacks are spread over 2^k rendezvous servers, where k is dynamicly chosen such that they can handle the load[2], while for faux-legitimate attacks[1], DDOS will just push up the computational costs for legitimate clients without ever actually shutting down the target[3]. This works for anything TCP-like. 1: You can DDOS anything by just behaving like (absurdly many) legitmate clients, eg `while true;do wget http\://example.com/;done`. 2: If the rendezvous and destination servers are all similar, then the work per server scales as the square root of the attack volume. Or put the other way around, the amount of attack traffic this setup can absorb scales as the square of the traffic each server can handle. 3: So for faux-legitimate attacks, the attacker's goal is not to overwhelm the server, but to maximize the costs to legitimate clients trying to connect; the attacker will generate only (roughly) as much traffic as the destination can handle, with as large a proof-of-work as possible. Assuming the destination server normally runs around 50% load, the total work imposed on legitimate clients (distributed over all of them) will be about the same as the attacker's available CPU. If the destination server normally runs significantly below 50% load, the imposed work will be proportionately lower.
- ahmedalsudani 5y ago
  Adding capacity, dropping suspect traffic.
- 0des 5y ago
  You simply need bigger pipes to ingest more traffic than the attack can provide. It is presumed these days that packet analysis in some cases can require too much power costwise rather than scaling up the connection to swallow it.
  - bombcar 5y ago
    And the "further up the chain" you can move the mitigation, the easier that is. Mitigating on your box requires a huge pipe to your box, but if your provider can mitigate at their border router, well those are bigger and already have huge traffic to and through them. And it's in their interest to talk to their providers and mitigate even further back if possible. And some mitigations are relatively easy (block all DNS traffic to this subnet, etc).
- Koenvh 5y ago
  This is one such an approach: https://www.concordia-h2020.eu/blog-post/setting-up-a-national-ddos-clearing-house/ https://www.concordia-h2020.eu/blog-post/setting-up-a-nation...
dubcanada 5y ago
On another note, the amount they are asking for seems really reasonable, like how are they making money? A DDoS attack must cost more then like 0.06BTC (like $3500 USD) to run all weekend?
- rgrmrts 5y ago
  I’d assume they’re not actually paying for these attacks and have access to a large botnet, which would imply no cost (outside of hours spent constructing the attack) to the adversary, right?
  - jacobobryant 5y ago
    They might be renting the botnet.
- stilley2 5y ago
  If they're using a botnet of compromised servers then I suspect the marginal cost is negligible.
- harikb 5y ago
  It could also be seed investment to see who is likely to pay up. ROI comes from second attack.
- bombcar 5y ago
  A DDoS can be rented for less than $100/mo depending on how much bandwidth you want to flood. Remember, they're compromised machines; you're not paying for egress bandwidth.
  - jorgesborges 5y ago
    Are most machines involved in attacks like this compromised? Are we talking generic consumer desktop machines, like my grandma’s old XP desktop running in her basement?
    - 5y ago
rgrmrts 5y ago
Is the attack on its webmail/website or on their smtp servers? I’ve been wondering about this, but how does one protect smtp servers from distributed attacks? Let’s assume smaller attackers, do you just need good firewalls in front of your servers to prevent congestion to the smtp servers? Are there off the shelf tools that can be configured to help here (pf maybe)? Do tools like fail2ban help? For context, I’ve picked up self-hosting again after many years and though I don’t anticipate being a target for large attacks I have been curious what tools individuals have at their disposal or if it’s a fools errand to even try.
- LinuxBender 5y ago
  > Is the attack on its webmail/website or on their smtp servers? It is an attack on the httpd and smtpd daemons. imap has been unaffected as far as I can tell. > how does one protect smtp servers from distributed attacks? By design, MX can be as distributed and in large number as you can afford or as willing to spend. This can be a combination of load balancer virtual IP's distributing load to many MX servers behind it and many MX DNS records with the same or different priorities. This of course won't help much if the people attacking are paying ddos-as-a-service farms to bring on massive volume and packet rates that overload all your servers. There are DDoS scrubbing services you can pay for that will advertise your AS number or use GRE tunnels or VPN's to clean the attack data for you. These scrubbing solutions are no guarantee of mitigation. > Do tools like fail2ban help? No. That would be pointless whack-a-mole. If an individual person is mad at you and launching a tool from their own PC or a handful of VM's, then yes fail2ban will help. Blocking individual IP's on your MX servers under a real distributed DDoS attack would be futile. Scrubbing centers are about the only solution once the attack is big enough. Or if you had unlimited funds you could deploy many datacenters or point-of-pressence destinations and build your own scrubbing networks but that is very expensive.
  - rgrmrts 5y ago
    Thanks for your response! So using specific examples here for smtp, I get a 1gbps guaranteed network from Hetzner so in theory I’d need to distribute over 50 servers to withstand this attack? It’s not clear to me why fail2ban wouldn’t at least help, if the botnet is a thousand machines wouldn’t I (eventually) have them all blocked? And therefore reduce the overall duration of the attack? Or is the problem that it’s hard to differentiate between good clients and bad clients because no single client is sending enough traffic to be suspicious? Also, do you have any specific examples of ddos scrubbing services? Would like to take a look specifically at affordability for individuals.
stilley2 5y ago
Just to add a data point, I had trouble accessing fastmail early this morning (around 7 EDT). I haven't had any issues since then.
eloeffler 5y ago
The corresponding posteo blog post is archived here: https://archive.md/ZmKAA https://archive.md/ZmKAA
NabiDev 5y ago
Own the whole email provider, use protonmail? Ironic.
- spinax 5y ago
  The name on that email is not one of the Runbox folks listed on their About page, so one can only guess who that actually is or how the email was sent; it could have been a BCC for all we know. https://runbox.com/about/runbox-team/ https://runbox.com/about/runbox-team/
  - lrae 5y ago
    I don't think they'd publish some random e-mail address of a random person. Are we sure that isn't the address the extortion mail came from? Would make more sense?
- jlund-molfese 5y ago
  It doesn't look like that email actually belongs to Fastmail staff, but if it did, having a backup on another email service is a good idea in case there's a DNS issue with Fastmail or something. It's the same idea behind a VPS provider serving their status page on another company's infrastructure.
TacticalCoder 5y ago
We still hear about DDoS attacks like this once in a while but it seems it's not anywhere near as common as it used to be. What happened? It looks like the bad guys are really having more and more trouble mounting succesful DDoS: how comes? It also looks like, in despair, they're targetting smaller fishes. Why? Smaller botnets? Cloudflare and OVH and the likes just being too good at absorbing everything and anything you can throw at them? Simple firewall rules getting rid of 99% of the crap? What's the reason it's not as prevalent as it used to be?
- jacques_chester 5y ago
  One factor: the main target was internet gambling sites. They banded together and collectively agreed not to pay any DDoSers. Source: Security Engineering by Ross Anderson.
- bikingbismuth 5y ago
  The general quality of DDoS scrubbing services has dramatically improved in the last 10 years. I work for a large tech company and Silverline has protected us from 100G+ attacks.
  - short12 5y ago
    What about 250, 500, 1tbps. Hell 2.5 is possible now 100gbps is basically trivial test for a new botnet
    - zinekeller 5y ago
      Cloudflare is the most well-known, but there are lots of providers now that provides these level of service, from the old guys like Akamai's Prolexic to new ones like Imperva to tier-1 ISPs like Telia's. Additionally, depending on the exact service, you can certainly firewall traffic - close to the source. The specific problem here is that mail servers, since that is not the target of DDoS until now, which means that there are few companies who do provide mail exchange-specific DDoS protection, which means larger companies (Verizon/Yahoo, Microsoft/Outlook, Google/Gmail) just operate servers well beyond what they really need, and I don't think that they can just run to Cloudflare and violate their privacy promise in the process.
psdmm 5y ago
تتحرك جسيمات المادة الصلبة حركة اهتزازية باستمرار - https://mesael.com/25/%D8%AA%D8%AA%D8%AD%D8%B1%D9%83-%D8%AC%D8%B3%D9%8A%D9%85%D8%A7%D8%AA-%D8%A7%D9%84%D9%85%D8%A7%D8%AF%D8%A9-%D8%A7%D9%84%D8%B5%D9%84%D8%A8%D8%A9-%D8%AD%D8%B1%D9%83%D8%A9-%D8%A7%D9%87%D8%AA%D8%B2%D8%A7%D8%B2%D9%8A%D8%A9-%D8%A8%D8%A7%D8%B3%D8%AA%D9%85%D8%B1%D8%A7%D8%B1 https://mesael.com/25/%D8%AA%D8%AA%D8%AD%D8%B1%D9%83-%D8%AC%...
SavantIdiot 5y ago
I've been with RunBox since 2012 because of Norway's internet privacy policies. I support them not giving into to extortion. If I don't have email for a few days, or a month, big deal. I remember how to use a phone to pay bills.
- LeoPanthera 5y ago
  This is a surprisingly insensitive comment. Email is as much a a part of modern life as phone calls and postal mail. Asking people to "just go without" for a few days may be possible for you, but is certainly not possible for a huge fraction of people.
- Aachen 5y ago
  My employer uses runbox. I'm happy you're so unattached that you can go a month without any email (can't sign up anywhere, for example) but we'd miss incoming invoices, can't send invoices to our customers, can't deliver reports to customers (we do security audits so that's kinda important to be able to deliver using e.g. pgp), would miss requests for new assignments... I'm fully in favor of not paying into the extortion and not going for a global tap system like cloudflare, but we do need some solution in at most a few days. In the Netherlands there is for example the Nawas (it's also a pun, referring to a laundry cleaning thing) that scrubs malicious traffic for ISPs. I don't know the details of how they're being attacked, but with typical reflector services it's easy to remove that traffic based on a source port. Any large Norwegian hosting company (they just need an uplink bigger than the DDoS, or work with whoever the traffic comes from at their peering points) could provide that service as well. Not saying this is trivial but there are options other than waiting.
Wronnay 5y ago
I don't wonder that Runbox and Posteo are vulnerable, but I would have thought that Fastmail is too big for such an attack...
- meribold 5y ago
  Fastmail is working just fine for me. At least the web interface was briefly offline yesterday, though.
trulyrandom 5y ago
Coincidentally, the app password that I've used for Fastmail's CalDAV service for years suddenly started causing 403's today. I wonder if that's related (but can't think of how it could be)
- mmcclimon 5y ago
  (I work for Fastmail.) One of our attempts at doing some mitigation of the attack caused this; we fixed it about 16:00 US Eastern this afternoon. Sorry about that! Without getting into too much detail: we were limiting some HTTP methods, but unintentionally blocked REPORT, which DAV clients use to see what’s changed.
  - rswail 5y ago
    This is going to sound weird, but my father can't send email with attachments currently. I'm a long time fastmail user/customer and I'm also a long time IT guy so can't see what's going on. Basically SMTP login and everything works fine, client sends the message and then it just times out.
  - trulyrandom 5y ago
    All good now. Thanks!
muttantt 5y ago
Yet another reason to host your email at your own domain which you can move at will.
- Aachen 5y ago
  Or less hassle than completely moving over to a competitor (terrorists win in that case, to use counter strike terminology): add a backup MX record, perhaps to a small vps that just forwards mail to the real server with no retry timeout.
  - throwaway984393 5y ago
    It sounds like it could work. Are there any mail admins in the comments who could propose an smtp daemon & configs that would store all mail for a domain and wait indefinitely to forward it on? I think there's a lot of users/businesses who would like to implement this.
- qwertox 5y ago
  Is it really worth the hassle to move to another provider/self-hosted server for such a temporary problem? Moving to another provider would mean to set up a new account, and usually the free tiers won't allow to use your own domain. So you would be locked to a new provider where you could face the same problems. Temporarily moving to a self-hosted server would be an option, but probably just to receive emails during this time. You'd have to set up your certificate, and optionally DKIM, DMARC, SPF or whatever is required to ensure that your sent emails arrive properly. I can't imagine self-hosting email being something which won't give you a hard time every now and then. Generally yes, I am in favor of owning your email-domain, but then using it with a professional provider like mailbox.org unless you really know enough about the topic. But in this case, where the issue is a DDoS attack, I wouldn't do anything, since all undelivered email will be re-sent at a later time.
- kiwijamo 5y ago
  Our Bank "A" suffered a major DDoS attack recently. My non-techy partner, upset, declared were moving to Bank "B". I pointed out that Bank B had also been target of DDoS attacks so we would be moving banks only to face the exact same issues. Point being you may move to a different service provider (email, banking, whatever) to find the new one has the same problems anyway.
julianlam 5y ago
The sooner these kids realize that DDoS extortion attacks do nothing except waste resources (and monetary resource to CloudFlare, basically), the sooner they'll give up. They hit a number of VoIP providers recently, too.
- pan69 5y ago
  Anyone knows what the cost of performing a DDoS is these days? As a target, if you can't do anything else, is it best to hang in there and wait it out due to mounting costs for the attacker?
- muttantt 5y ago
  The latest attacks on VoIP providers resulted in a SIP/VoIP specific solution from Cloudflare. I expect we will see Cloudflare mail protection soon enough... Makes you wonder.
  - yarcob 5y ago
    I first learned that someone cracked my app from a company that offered to help me fight cracked downloads of my app. I was immediately suspicious.
    - a1369209993 5y ago
      This isn't new; it's called a protection racket and it works especially well if you actually do have a mechanism to protect against the thing (though of course that's not a requirement if you're generating most of the problem yourself).
  - piperswe 5y ago
    Cloudflare Spectrum can already protect arbitrary TCP services, such as SMTP or IMAP
  - prdonahue 5y ago
    Stay tuned .. Today we can help quite a bit by proxying TCP traffic using Cloudflare Spectrum, e.g., spoofed traffic will never reach origin as it can't complete a handshake, we can use things like SYN cookies to challenge source, etc. In the future, there's a lot more we plan to do here.
throwawaysea 5y ago
If you didn’t want to depend on a big provider like AWS or Cloudflare, what is the approach to fending off a DDoS attack? What type of hardware would you need to acquire? What type of software? Are there guides on this type of thing?
- neom 5y ago
  You could try to do it yourself with firewall rules, reverse proxies, things like that, but to fight large scale ddos you really need to be moving the traffic around using BGP, and you'd want to dump the traffic somewhere, so you'd need bandwidth to dump the traffic into, that's why companies like cloudflare exist, they're able to work with bgp and they have a lot of bandwidth to absorb the traffic on behalf of the customer.
- bluGill 5y ago
  You need a massive amount of bandwidth and a few redundant servers. There are counties with less bandwidth than you need to handle. It isn't impossible, but cloudflare isn't evil (that I know of?) and so it is best to support them as your backup
exegete 5y ago
mailbox.org was also attacked https://twitter.com/mailbox_org/status/1451237052855595012?s=21 https://twitter.com/mailbox_org/status/1451237052855595012?s... I kept having errors in sending emails and logging into the web interface and the mailbox.org status page indicated nothing was wrong. Why have the page if that’s not where I’m supposed to find out something like this?
- omnimus 5y ago
  The server was probably running it was just flooded. Probably requires different type of checks to catch this.
dang 5y ago
Url changed from https://blog.runbox.com/2021/10/runbox-is-under-attack-by-extortionists/ https://blog.runbox.com/2021/10/runbox-is-under-attack-by-ex..., which points to this.
deleted 5y ago
[deleted]
Aachen 5y ago
DDoS is said to be very cheap. What if we used that to boot the culprits? It's obviously illegal, so just as a thought experiment: if those amplifiers (dns resolvers or whatever is popular at the moment) started experiencing issues due to their servers being a nuisance to others...? It still seems much easier to just blackhole IPs that are causing problems, like collectively (at the edge of your AS) block IPs that long-term host a service that is actively involved in facilitating DDoS attacks, but for some reason nobody is doing that. This could be a more direct way: see where DDoS traffic is coming from and... poof
- prdonahue 5y ago
  Not really practical or effective to implement. The attacks more often than not come from botnets comprised of compromised consumer devices. You can’t just outright drop traffic from residential ISPs. We didn’t disclose it at the time but this 17.2M rps attack came from (home) Mikrotik devices that were running proxy services: https://blog.cloudflare.com/cloudflare-thwarts-17-2m-rps-ddos-attack-the-largest-ever-reported/ https://blog.cloudflare.com/cloudflare-thwarts-17-2m-rps-ddo....
  - Aachen 5y ago
    Ah right, I was figuring most of those misconfigured udp services (dns, ntp, ...) were running on servers rather than regular home IPs. That does make it a little different. Still, if an ISP has had multiple abuse reports for the same subscriber and they're not doing anything, after some time it starts to become reasonable to block this IP, and in a further escalation, this ISP's ranges altogether until they clean their act up. I remember getting the Internet connection blocked as a teenager on an XS4ALL connection for being an ass on the Internet (I tried to DoS a domain squatter that tried to sell a domain I wanted for a thousand times the price with no added value). The abuse desk which I had to contact to unblock it took my promise to not do it again seriously (as did I), not sure how other ISPs handle this.
deleted 5y ago
[deleted]
swozey 5y ago
Well that explains it, I literally switched from gmail to fastmail 2 days ago and it was going amazingly until last night when I couldn't load it. "Oh great, an unreliable service I just paid a year for" -- this makes me want to support them even more. Their customer support was fantastic. I didn't inquire about the outage, instead a dns issue I'd created. If you're a heavy gmail user you should read this story of my last year (out of 10+) with them and IMO migrate off as soon as you can. I wish I hadn't used an @gmail as long as I did. For some unknown reason, they won't tell me, G marked my google pay account as "possible fraud, unable to verify identity" at some point this year. No idea when, I realized it when CC expired and my 3 month old Google Fi account wouldn't pay its bill and it wouldn't let me enter a credit card to pay for any G service. My G services started to drop like flies once my card expired. They locked me out of Google Pay and won't let me enter a new CC to pay for subs/gdrive. I had to clear out the entire 50gb of my gdrive so that I could get EMAIL again to my now "free" gmail account which is hovering around its 15gb limit, that's how old it is. The real kicker; I can't contact support anymore because I'm not a "paying customer" who gets their "world class support." The fun part of this was I was completely without a cell phone for a week until I got onto ATT. Support (live chat) was completely useless and just sent me to a page to send them a copy of my license and a utility bill. I did that about 4 times now and I still can't change CCs on my Google account to pay for things. And they still won't contact me, won't tell me what the deal is or anything. It's been about 6 months. I was joking to a friend before I switched to Fi "watch this be a terrible idea, it breaks and I'm without a phone for a month and have nobody to fix it." Welp. Anyway, I tried out Protonmail months ago and was completely unimpressed with its search, interface, etc. I had low expectations of fastmail (due to proton exp) but it's great, I'm really impressed. It threads conversations like you'd expect and I could actually find things I searched for, I sold a house in the middle of my Proton use and it was miserable keeping track of all of the emails/docs/etc I was getting/sending for some reason. I remember searching for important attachments and it finding 10s or hundreds of attachments that were my realtors signature picture of herself that showed up in every email. Sorry for the tangent, be very careful of how much you rely on G services because this cliche horror story you randomly see on twitter/reddit completely happened to me. I'll never be able to stop using this @gmail account because I have so, so many things tied to it but I'm going to try my best to undo most of that. My biggest fear now is that my G account gets completely locked and all of the things I use G to authenticate to will be lost. Undoing all that is a nightmare. Use a custom domain for your email..
deleted 5y ago
[deleted]
oars 5y ago
Fantastic discussion about DDoS in these comments, thank you.

291 points by conjuredbytes 5y ago

deleted 5y ago

[deleted]

ryantgtg 5y ago

Wonder if it’s the same folks who hit voip.ms last week. That one lasted a while.

29083011397778 5y ago
Reading through r/VOIP, I could have sworn Voip.ms was 2 weeks ago - then other providers began to be hit with DDoS attacks. Frankly, it doesn't seem like such a bad strategy. With the amount of panic the attack on Voip.ms caused, a number of people were discussing switching voip providers. And switching targets to other providers just as they get a massive influx of new customers seems like a force multiplier IMO.

sysadm1n 5y ago

> Since these DDoS attacks started we have worked with our system administrators and Internet Service Provider to mitigate the attacks How do you mitigate such an attack though? I know Cloudflare can stop this, but how do you create your own bespoke 'DDOS mitigation' tool, and what does that look like?

capableweb 5y ago
One way is to build your software on top of distributed/content-addressed P2P software (not Blockchain, but pure P2P). The angle of attack disappears completely then. Otherwise, with centralized infrastructure, you identify the bad traffic and send it elsewhere, an instance that keeps connections open but delivers nothing and have a small amount of resource usage. Problem is that you still need to be able to handle the traffic, but at least you avoid hitting your main infrastructure.
- Mister_Snuggles 5y ago
  > One way is to build your software on top of distributed/content-addressed P2P software (not Blockchain, but pure P2P). The angle of attack disappears completely then. I can see how this works for some applications, but how would this work for SMTP and IMAP/JMAP?
  - capableweb 5y ago
    You can't force centralized/federated protocols into distributed protocols so you wouldn't.
  - a1369209993 5y ago
    > how would this work for SMTP You can actually get this semi-transparently with a TOR-like system. (I'm not sure TOR itself actually implements all of this, and also TOR is perennially underprovisioned for political reasons, so this mostly won't work in practice.) First, you need a distributed DNS mechanism to publish "example.com. 9999 IN MX 1 abcdefghijklmnop.onion". This is mostly static, so DDOS doesn't really work. You then come up with a sequence of rendezvous servers, which we'll number starting from 1. (I think actual TOR just assumes a single (or few) rendezvous server is sufficient, but I'd need to go dig through the code to be sure.) You then try (in sequence) servers 1,2+rand(2),4+rand(4),...,2^k+rand(2^k),... where rand(x) picks a random number in [0,x). (0: I'm not sure whether distinct rands should share lower bits; see below.) If a server is overloaded it just drops traffic on the floor. The destination server then checks the rendezvous servers in sequence until it's gotten enough successes that any client would have tried one of the successful rendezvous servers in its random sequence ([0] above affects the distribution here). Under heavy load, the destination server also sets a proof of work requirement (clientHello statisfies the standard hash-has-x-leading-zeros), which allows the rendezvous servers to drop most of the incoming traffic. Legitimate clients by definition are not spamming connections as fast as they can, so they can burn CPU to meet this requirement. DDOS clients can also burn CPU on this, but that reduces the rate at which they generate traffic. The end result is volumetric attacks are spread over 2^k rendezvous servers, where k is dynamicly chosen such that they can handle the load[2], while for faux-legitimate attacks[1], DDOS will just push up the computational costs for legitimate clients without ever actually shutting down the target[3]. This works for anything TCP-like. 1: You can DDOS anything by just behaving like (absurdly many) legitmate clients, eg `while true;do wget http\://example.com/;done`. 2: If the rendezvous and destination servers are all similar, then the work per server scales as the square root of the attack volume. Or put the other way around, the amount of attack traffic this setup can absorb scales as the square of the traffic each server can handle. 3: So for faux-legitimate attacks, the attacker's goal is not to overwhelm the server, but to maximize the costs to legitimate clients trying to connect; the attacker will generate only (roughly) as much traffic as the destination can handle, with as large a proof-of-work as possible. Assuming the destination server normally runs around 50% load, the total work imposed on legitimate clients (distributed over all of them) will be about the same as the attacker's available CPU. If the destination server normally runs significantly below 50% load, the imposed work will be proportionately lower.
ahmedalsudani 5y ago
Adding capacity, dropping suspect traffic.
0des 5y ago
You simply need bigger pipes to ingest more traffic than the attack can provide. It is presumed these days that packet analysis in some cases can require too much power costwise rather than scaling up the connection to swallow it.
- bombcar 5y ago
  And the "further up the chain" you can move the mitigation, the easier that is. Mitigating on your box requires a huge pipe to your box, but if your provider can mitigate at their border router, well those are bigger and already have huge traffic to and through them. And it's in their interest to talk to their providers and mitigate even further back if possible. And some mitigations are relatively easy (block all DNS traffic to this subnet, etc).
Koenvh 5y ago
This is one such an approach: https://www.concordia-h2020.eu/blog-post/setting-up-a-national-ddos-clearing-house/ https://www.concordia-h2020.eu/blog-post/setting-up-a-nation...

dubcanada 5y ago

On another note, the amount they are asking for seems really reasonable, like how are they making money? A DDoS attack must cost more then like 0.06BTC (like $3500 USD) to run all weekend?

rgrmrts 5y ago
I’d assume they’re not actually paying for these attacks and have access to a large botnet, which would imply no cost (outside of hours spent constructing the attack) to the adversary, right?
- jacobobryant 5y ago
  They might be renting the botnet.
stilley2 5y ago
If they're using a botnet of compromised servers then I suspect the marginal cost is negligible.
harikb 5y ago
It could also be seed investment to see who is likely to pay up. ROI comes from second attack.
bombcar 5y ago
A DDoS can be rented for less than $100/mo depending on how much bandwidth you want to flood. Remember, they're compromised machines; you're not paying for egress bandwidth.
- jorgesborges 5y ago
  Are most machines involved in attacks like this compromised? Are we talking generic consumer desktop machines, like my grandma’s old XP desktop running in her basement?
  - 5y ago

rgrmrts 5y ago

Is the attack on its webmail/website or on their smtp servers? I’ve been wondering about this, but how does one protect smtp servers from distributed attacks? Let’s assume smaller attackers, do you just need good firewalls in front of your servers to prevent congestion to the smtp servers? Are there off the shelf tools that can be configured to help here (pf maybe)? Do tools like fail2ban help? For context, I’ve picked up self-hosting again after many years and though I don’t anticipate being a target for large attacks I have been curious what tools individuals have at their disposal or if it’s a fools errand to even try.

LinuxBender 5y ago
> Is the attack on its webmail/website or on their smtp servers? It is an attack on the httpd and smtpd daemons. imap has been unaffected as far as I can tell. > how does one protect smtp servers from distributed attacks? By design, MX can be as distributed and in large number as you can afford or as willing to spend. This can be a combination of load balancer virtual IP's distributing load to many MX servers behind it and many MX DNS records with the same or different priorities. This of course won't help much if the people attacking are paying ddos-as-a-service farms to bring on massive volume and packet rates that overload all your servers. There are DDoS scrubbing services you can pay for that will advertise your AS number or use GRE tunnels or VPN's to clean the attack data for you. These scrubbing solutions are no guarantee of mitigation. > Do tools like fail2ban help? No. That would be pointless whack-a-mole. If an individual person is mad at you and launching a tool from their own PC or a handful of VM's, then yes fail2ban will help. Blocking individual IP's on your MX servers under a real distributed DDoS attack would be futile. Scrubbing centers are about the only solution once the attack is big enough. Or if you had unlimited funds you could deploy many datacenters or point-of-pressence destinations and build your own scrubbing networks but that is very expensive.
- rgrmrts 5y ago
  Thanks for your response! So using specific examples here for smtp, I get a 1gbps guaranteed network from Hetzner so in theory I’d need to distribute over 50 servers to withstand this attack? It’s not clear to me why fail2ban wouldn’t at least help, if the botnet is a thousand machines wouldn’t I (eventually) have them all blocked? And therefore reduce the overall duration of the attack? Or is the problem that it’s hard to differentiate between good clients and bad clients because no single client is sending enough traffic to be suspicious? Also, do you have any specific examples of ddos scrubbing services? Would like to take a look specifically at affordability for individuals.

stilley2 5y ago

Just to add a data point, I had trouble accessing fastmail early this morning (around 7 EDT). I haven't had any issues since then.

eloeffler 5y ago

The corresponding posteo blog post is archived here: https://archive.md/ZmKAA https://archive.md/ZmKAA

NabiDev 5y ago

Own the whole email provider, use protonmail? Ironic.

spinax 5y ago
The name on that email is not one of the Runbox folks listed on their About page, so one can only guess who that actually is or how the email was sent; it could have been a BCC for all we know. https://runbox.com/about/runbox-team/ https://runbox.com/about/runbox-team/
- lrae 5y ago
  I don't think they'd publish some random e-mail address of a random person. Are we sure that isn't the address the extortion mail came from? Would make more sense?
jlund-molfese 5y ago
It doesn't look like that email actually belongs to Fastmail staff, but if it did, having a backup on another email service is a good idea in case there's a DNS issue with Fastmail or something. It's the same idea behind a VPS provider serving their status page on another company's infrastructure.

TacticalCoder 5y ago

We still hear about DDoS attacks like this once in a while but it seems it's not anywhere near as common as it used to be. What happened? It looks like the bad guys are really having more and more trouble mounting succesful DDoS: how comes? It also looks like, in despair, they're targetting smaller fishes. Why? Smaller botnets? Cloudflare and OVH and the likes just being too good at absorbing everything and anything you can throw at them? Simple firewall rules getting rid of 99% of the crap? What's the reason it's not as prevalent as it used to be?

jacques_chester 5y ago
One factor: the main target was internet gambling sites. They banded together and collectively agreed not to pay any DDoSers. Source: Security Engineering by Ross Anderson.
bikingbismuth 5y ago
The general quality of DDoS scrubbing services has dramatically improved in the last 10 years. I work for a large tech company and Silverline has protected us from 100G+ attacks.
- short12 5y ago
  What about 250, 500, 1tbps. Hell 2.5 is possible now 100gbps is basically trivial test for a new botnet
  - zinekeller 5y ago
    Cloudflare is the most well-known, but there are lots of providers now that provides these level of service, from the old guys like Akamai's Prolexic to new ones like Imperva to tier-1 ISPs like Telia's. Additionally, depending on the exact service, you can certainly firewall traffic - close to the source. The specific problem here is that mail servers, since that is not the target of DDoS until now, which means that there are few companies who do provide mail exchange-specific DDoS protection, which means larger companies (Verizon/Yahoo, Microsoft/Outlook, Google/Gmail) just operate servers well beyond what they really need, and I don't think that they can just run to Cloudflare and violate their privacy promise in the process.

psdmm 5y ago

تتحرك جسيمات المادة الصلبة حركة اهتزازية باستمرار - https://mesael.com/25/%D8%AA%D8%AA%D8%AD%D8%B1%D9%83-%D8%AC%D8%B3%D9%8A%D9%85%D8%A7%D8%AA-%D8%A7%D9%84%D9%85%D8%A7%D8%AF%D8%A9-%D8%A7%D9%84%D8%B5%D9%84%D8%A8%D8%A9-%D8%AD%D8%B1%D9%83%D8%A9-%D8%A7%D9%87%D8%AA%D8%B2%D8%A7%D8%B2%D9%8A%D8%A9-%D8%A8%D8%A7%D8%B3%D8%AA%D9%85%D8%B1%D8%A7%D8%B1 https://mesael.com/25/%D8%AA%D8%AA%D8%AD%D8%B1%D9%83-%D8%AC%...

SavantIdiot 5y ago

I've been with RunBox since 2012 because of Norway's internet privacy policies. I support them not giving into to extortion. If I don't have email for a few days, or a month, big deal. I remember how to use a phone to pay bills.

LeoPanthera 5y ago
This is a surprisingly insensitive comment. Email is as much a a part of modern life as phone calls and postal mail. Asking people to "just go without" for a few days may be possible for you, but is certainly not possible for a huge fraction of people.
Aachen 5y ago
My employer uses runbox. I'm happy you're so unattached that you can go a month without any email (can't sign up anywhere, for example) but we'd miss incoming invoices, can't send invoices to our customers, can't deliver reports to customers (we do security audits so that's kinda important to be able to deliver using e.g. pgp), would miss requests for new assignments... I'm fully in favor of not paying into the extortion and not going for a global tap system like cloudflare, but we do need some solution in at most a few days. In the Netherlands there is for example the Nawas (it's also a pun, referring to a laundry cleaning thing) that scrubs malicious traffic for ISPs. I don't know the details of how they're being attacked, but with typical reflector services it's easy to remove that traffic based on a source port. Any large Norwegian hosting company (they just need an uplink bigger than the DDoS, or work with whoever the traffic comes from at their peering points) could provide that service as well. Not saying this is trivial but there are options other than waiting.

Wronnay 5y ago

I don't wonder that Runbox and Posteo are vulnerable, but I would have thought that Fastmail is too big for such an attack...

meribold 5y ago
Fastmail is working just fine for me. At least the web interface was briefly offline yesterday, though.

trulyrandom 5y ago

Coincidentally, the app password that I've used for Fastmail's CalDAV service for years suddenly started causing 403's today. I wonder if that's related (but can't think of how it could be)

mmcclimon 5y ago
(I work for Fastmail.) One of our attempts at doing some mitigation of the attack caused this; we fixed it about 16:00 US Eastern this afternoon. Sorry about that! Without getting into too much detail: we were limiting some HTTP methods, but unintentionally blocked REPORT, which DAV clients use to see what’s changed.
- rswail 5y ago
  This is going to sound weird, but my father can't send email with attachments currently. I'm a long time fastmail user/customer and I'm also a long time IT guy so can't see what's going on. Basically SMTP login and everything works fine, client sends the message and then it just times out.
- trulyrandom 5y ago
  All good now. Thanks!

muttantt 5y ago

Yet another reason to host your email at your own domain which you can move at will.

Aachen 5y ago
Or less hassle than completely moving over to a competitor (terrorists win in that case, to use counter strike terminology): add a backup MX record, perhaps to a small vps that just forwards mail to the real server with no retry timeout.
- throwaway984393 5y ago
  It sounds like it could work. Are there any mail admins in the comments who could propose an smtp daemon & configs that would store all mail for a domain and wait indefinitely to forward it on? I think there's a lot of users/businesses who would like to implement this.
qwertox 5y ago
Is it really worth the hassle to move to another provider/self-hosted server for such a temporary problem? Moving to another provider would mean to set up a new account, and usually the free tiers won't allow to use your own domain. So you would be locked to a new provider where you could face the same problems. Temporarily moving to a self-hosted server would be an option, but probably just to receive emails during this time. You'd have to set up your certificate, and optionally DKIM, DMARC, SPF or whatever is required to ensure that your sent emails arrive properly. I can't imagine self-hosting email being something which won't give you a hard time every now and then. Generally yes, I am in favor of owning your email-domain, but then using it with a professional provider like mailbox.org unless you really know enough about the topic. But in this case, where the issue is a DDoS attack, I wouldn't do anything, since all undelivered email will be re-sent at a later time.
kiwijamo 5y ago
Our Bank "A" suffered a major DDoS attack recently. My non-techy partner, upset, declared were moving to Bank "B". I pointed out that Bank B had also been target of DDoS attacks so we would be moving banks only to face the exact same issues. Point being you may move to a different service provider (email, banking, whatever) to find the new one has the same problems anyway.

julianlam 5y ago

The sooner these kids realize that DDoS extortion attacks do nothing except waste resources (and monetary resource to CloudFlare, basically), the sooner they'll give up. They hit a number of VoIP providers recently, too.

pan69 5y ago
Anyone knows what the cost of performing a DDoS is these days? As a target, if you can't do anything else, is it best to hang in there and wait it out due to mounting costs for the attacker?
muttantt 5y ago
The latest attacks on VoIP providers resulted in a SIP/VoIP specific solution from Cloudflare. I expect we will see Cloudflare mail protection soon enough... Makes you wonder.
- yarcob 5y ago
  I first learned that someone cracked my app from a company that offered to help me fight cracked downloads of my app. I was immediately suspicious.
  - a1369209993 5y ago
    This isn't new; it's called a protection racket and it works especially well if you actually do have a mechanism to protect against the thing (though of course that's not a requirement if you're generating most of the problem yourself).
- piperswe 5y ago
  Cloudflare Spectrum can already protect arbitrary TCP services, such as SMTP or IMAP
- prdonahue 5y ago
  Stay tuned .. Today we can help quite a bit by proxying TCP traffic using Cloudflare Spectrum, e.g., spoofed traffic will never reach origin as it can't complete a handshake, we can use things like SYN cookies to challenge source, etc. In the future, there's a lot more we plan to do here.

throwawaysea 5y ago

If you didn’t want to depend on a big provider like AWS or Cloudflare, what is the approach to fending off a DDoS attack? What type of hardware would you need to acquire? What type of software? Are there guides on this type of thing?

neom 5y ago
You could try to do it yourself with firewall rules, reverse proxies, things like that, but to fight large scale ddos you really need to be moving the traffic around using BGP, and you'd want to dump the traffic somewhere, so you'd need bandwidth to dump the traffic into, that's why companies like cloudflare exist, they're able to work with bgp and they have a lot of bandwidth to absorb the traffic on behalf of the customer.
bluGill 5y ago
You need a massive amount of bandwidth and a few redundant servers. There are counties with less bandwidth than you need to handle. It isn't impossible, but cloudflare isn't evil (that I know of?) and so it is best to support them as your backup

exegete 5y ago

mailbox.org was also attacked https://twitter.com/mailbox_org/status/1451237052855595012?s=21 https://twitter.com/mailbox_org/status/1451237052855595012?s... I kept having errors in sending emails and logging into the web interface and the mailbox.org status page indicated nothing was wrong. Why have the page if that’s not where I’m supposed to find out something like this?

omnimus 5y ago
The server was probably running it was just flooded. Probably requires different type of checks to catch this.

dang 5y ago

Url changed from https://blog.runbox.com/2021/10/runbox-is-under-attack-by-extortionists/ https://blog.runbox.com/2021/10/runbox-is-under-attack-by-ex..., which points to this.

deleted 5y ago

[deleted]

Aachen 5y ago

DDoS is said to be very cheap. What if we used that to boot the culprits? It's obviously illegal, so just as a thought experiment: if those amplifiers (dns resolvers or whatever is popular at the moment) started experiencing issues due to their servers being a nuisance to others...? It still seems much easier to just blackhole IPs that are causing problems, like collectively (at the edge of your AS) block IPs that long-term host a service that is actively involved in facilitating DDoS attacks, but for some reason nobody is doing that. This could be a more direct way: see where DDoS traffic is coming from and... poof

prdonahue 5y ago
Not really practical or effective to implement. The attacks more often than not come from botnets comprised of compromised consumer devices. You can’t just outright drop traffic from residential ISPs. We didn’t disclose it at the time but this 17.2M rps attack came from (home) Mikrotik devices that were running proxy services: https://blog.cloudflare.com/cloudflare-thwarts-17-2m-rps-ddos-attack-the-largest-ever-reported/ https://blog.cloudflare.com/cloudflare-thwarts-17-2m-rps-ddo....
- Aachen 5y ago
  Ah right, I was figuring most of those misconfigured udp services (dns, ntp, ...) were running on servers rather than regular home IPs. That does make it a little different. Still, if an ISP has had multiple abuse reports for the same subscriber and they're not doing anything, after some time it starts to become reasonable to block this IP, and in a further escalation, this ISP's ranges altogether until they clean their act up. I remember getting the Internet connection blocked as a teenager on an XS4ALL connection for being an ass on the Internet (I tried to DoS a domain squatter that tried to sell a domain I wanted for a thousand times the price with no added value). The abuse desk which I had to contact to unblock it took my promise to not do it again seriously (as did I), not sure how other ISPs handle this.

deleted 5y ago

[deleted]

swozey 5y ago

Well that explains it, I literally switched from gmail to fastmail 2 days ago and it was going amazingly until last night when I couldn't load it. "Oh great, an unreliable service I just paid a year for" -- this makes me want to support them even more. Their customer support was fantastic. I didn't inquire about the outage, instead a dns issue I'd created. If you're a heavy gmail user you should read this story of my last year (out of 10+) with them and IMO migrate off as soon as you can. I wish I hadn't used an @gmail as long as I did. For some unknown reason, they won't tell me, G marked my google pay account as "possible fraud, unable to verify identity" at some point this year. No idea when, I realized it when CC expired and my 3 month old Google Fi account wouldn't pay its bill and it wouldn't let me enter a credit card to pay for any G service. My G services started to drop like flies once my card expired. They locked me out of Google Pay and won't let me enter a new CC to pay for subs/gdrive. I had to clear out the entire 50gb of my gdrive so that I could get EMAIL again to my now "free" gmail account which is hovering around its 15gb limit, that's how old it is. The real kicker; I can't contact support anymore because I'm not a "paying customer" who gets their "world class support." The fun part of this was I was completely without a cell phone for a week until I got onto ATT. Support (live chat) was completely useless and just sent me to a page to send them a copy of my license and a utility bill. I did that about 4 times now and I still can't change CCs on my Google account to pay for things. And they still won't contact me, won't tell me what the deal is or anything. It's been about 6 months. I was joking to a friend before I switched to Fi "watch this be a terrible idea, it breaks and I'm without a phone for a month and have nobody to fix it." Welp. Anyway, I tried out Protonmail months ago and was completely unimpressed with its search, interface, etc. I had low expectations of fastmail (due to proton exp) but it's great, I'm really impressed. It threads conversations like you'd expect and I could actually find things I searched for, I sold a house in the middle of my Proton use and it was miserable keeping track of all of the emails/docs/etc I was getting/sending for some reason. I remember searching for important attachments and it finding 10s or hundreds of attachments that were my realtors signature picture of herself that showed up in every email. Sorry for the tangent, be very careful of how much you rely on G services because this cliche horror story you randomly see on twitter/reddit completely happened to me. I'll never be able to stop using this @gmail account because I have so, so many things tied to it but I'm going to try my best to undo most of that. My biggest fear now is that my G account gets completely locked and all of the things I use G to authenticate to will be lost. Undoing all that is a nightmare. Use a custom domain for your email..

deleted 5y ago

[deleted]

oars 5y ago

Fantastic discussion about DDoS in these comments, thank you.