12 ms·
FastMail under DDoS Attack
- athenot 11y agoIt's nice to see this published proactively. At the very least, transparency like this helps users understand what the circumstances are ahead of time. Best of luck to the Fastmail team, I hope they are able to weather the storm out.
- alastair 11y agoZoho Mail also attacked recently: https://www.zoho.com/service-updates/blog/zoho-services-under-criminal-attack.html https://www.zoho.com/service-updates/blog/zoho-services-unde...
- deleted 11y ago[deleted]
- awqrre 11y agoDDoS is a type of attack that is so old that I wonder why it is still possible to exploit it. It is at least 15 years old? (edit: I am not blaming FastMail, this is just a general assumption)
- AYBABTME 11y agoThere's too many ways to do it by abusing legit protocols, tricking web browsers, etc, what fool proof countermeasure would you think there is? Other than out-doing the attacker by over provisioning your networks?
- sowbug 11y agoIt's a generic term, not a type of attack. The question is similar to asking why we haven't developed a defense against being blown up when explosions are old. As long as there are services, and as long as those services have finite capacity, there will be DDoSes -- both accidental and intentional.
- toast0 11y agoDdos traffic generation capacity and server capacity scale in similar ways. As more and more of my servers get 10g Ethernet, more and more idiots are running chargen on 10g Ethernet.
- Analemma_ 11y agoIt's still possible to exploit because "DDoS" isn't one attack, but a category: people keep finding new ways to slam a target with tons of traffic. For a while, the most common way was with botnets of compromised PCs. They still exist, but big attacks with them are less common since Microsoft has gotten better at securing people's computers. The big thing now is "amplification attacks": basically, finding a way to send a small amount of data and get some other host to flood your target with a huge amount of data in response. Search "NTP amplification attack" for details. More recently, China has weaponized the Great Firewall to be yet another DDoS vector: they inject JavaScript into pages that people visit, and that JS floods a target with requests. As long as there is some way to point a lot of requests somewhere you want, DDoS attacks will be a thing.
- awqrre 11y agonot necessarily... you could use AI to detect attacks or some other methods that I am not aware of.
- alextgordon 11y agoCommander Data, please report to engineering
- deleted 11y ago[deleted]
- ori_b 11y agoBecause it's an attack on fundamental physical resources, rather than on any technological feature. There are exactly two workarounds: Minimize processing of packets that you think are coming from the attackers, and have more servers. Asking why we haven't solved DDOS is kind of like saying "Why can't we cure decapitation? We've got all of these new antibiotics!"
- brongondwana 11y agoNot even more servers - fatter pipe. We are using less than 1% of the inbound capacity of our connection in normal operation, yet the attackers easily filled it. The firewalls that we are running were easily able to absorb the additional traffic. Neil's metaphor of the post office was quite accurate - even in the heaviest attack, when we didn't have upstream DDoS protection turned on, about 10% of user traffic was getting through just fine - it's just that a random 10% of traffic makes for a very poor TCP experience.
- gozo 11y agoSeveral people seem to be saying that DDoS isn't one attack, which is correct, but not particularly relevant. Many times the attacks are simpler today, because there's little need for sophistication. Previously many attacks used to use some form of multiplier, either for the traffic itself or the resources targeted. The reason the attacks are still viable is because little has happened to the Internet itself. We still have the same challenges we had 15 years ago. Some things are slowly getting better, but it's still fundamentally the same. Increase in overall bandwidth and vulnerabilities doesn't help either.
- rasz_pl 11y agoWe need technical capability for all the tier 1/2 boys to be able to tell if packets incoming from network X are legitimate, and drop them if they are part of currently ongoing attack. Afaik fixed silicon edge routers peering tier 1/2 networks were the biggest obstacle in filtering good traffic from spoofed/botnet one. Just a year ago we had huge problems when BGP rose to >512K entries, which is an order of magnitude easier. Did anything change since ~10 years ago? Last time I dabbled in this it was so bad even Tier3 (ISPs) werent filtering spoofed packets.
- feld 11y agoTechnical capability exists. Everyone running BGP should be implementing BCP38. http://www.bcp38.info http://www.bcp38.info
- touchofevil 11y agoRunbox was also recently under ddos attack https://blog.runbox.com/2015/11/ddos-attacks-on-runbox/ https://blog.runbox.com/2015/11/ddos-attacks-on-runbox/
- andrem 11y agoDDoS attacks today are such a commodity. It takes next to nothing to launch them. You can get upwards of 200Gbps for 1/6 of a bitcoin. It is very easy to setup and you can DDoS your favourite site in a matter of minutes. These are not very smart attacks and can be mitigated even using the free tier of cloudflare. I don't have the background on the mail provider attacks but 6.5k ransom seems to come from attackers who use easily available booters. Hushmail switched to Cloudflare throughout their attacks and that seemed to have helped, not sure what fastmail will do. But any public web service should not be in a position where they are vulnerable to off the shelf DDoS attacks. DDoS attack as suffered by for example Githuh with heavy coordination and nation states behind them require more specialised defenses. There are commercial alternatives out there that go from anywhere between 9k-40k per month depending on bandwidth and technology - see Imperva, Prolexic, Neustar, Nexusguard, Blacklotus, Incapsula, etc.. Apart from the initial setup which is more involved than Cloudflare's there is not much to do apart from throwing money at it. Quite the money making business really :)
- jonahx 11y agoWhat's a good setup for a medium sized site to use that would protect it from off the shelf attacks?
- joshmn 11y agoIf you're only web, you should be perfectly fine behind CloudFlare.
- toast0 11y agoThe only real defense is to serve your site from somewhere with more incoming bandwidth than incoming abuse. Afaik, the only type of filtering you're likely to get from an upstream provider is null routing of attacked ips, which helps protect their network, but doesn't help you serve users (you can switch ips, but abusers will likely switch too)
- giancarlostoro 11y agoWe used to use TCP proxies when I ran a game server where skiddies would try to attack it. I know buyvm.net is where we got one small $15 a year VPS with $3 a month "DDOS protection", not sure if we ever had issues after that with DDoS or not, we kept getting more and more proxies. We would give users a different one upon refreshing the page (it was a browser based game, but they would connect to a game server via TCP). I know the Minecraft community offers proxies as well. If you're on HTTP though, cloudflare is one of the cheaper options.
- deleted 11y ago[deleted]
- copperx 11y agoWhy do we always hear of DDoS attacks to mail services, but never to Gmail? are they more DDoS resistant? are the attacks not public?
- livejamie 11y agoI imagine it's a lot easier to DDoS a smaller company than it would to DDoS Google?
- brongondwana 11y agoIt's mostly a matter of scale. Google are big enough and their internet connections are big enough that they can absorb a regular-sized DDoS without breaking a sweat.
- mathrawka 11y agoI bet the amount of traffic GMail handles would be considered a DoS if it was redirected to one of the smaller mail sites.
- brongondwana 11y agoBack in the day (/shakes fist at cloud) it was the slashdot effect. Rapid spikes in popularity overloading systems. We certainly couldn't handle scaling up to gmail size all at once. Maybe over a couple of years. Of course our growth strategy is quality (people willing to pay for a good service quid-pro-quo) over quantity (free service and monetise later via ads/analytics), so we've had a slow steady growth for the entire 15 years we've been operating rather than the viral growth and sell-out/pivot that unicorns are known for.
- jorangreef 11y agoQuality and quantity are not necessarily opposed. People like Ford and Edison used to believe that you could get more quality as volume increased. And in fact, if you wanted to increase quality, then you had to increase volume.
- kevinSuttle 11y agoAnd Protonmail. https://protonmail.com/blog/protonmail-ddos-attacks/ https://protonmail.com/blog/protonmail-ddos-attacks/ Fastmail, pro tip: don't pay any ransom.
- robn_fastmail 11y agoNot a chance :)
- mahmud 11y agoHang in there, mate. Love your work :-)
- PuffinBlue 11y agoThanks for taking this stance. I'm a customer (I think we actually talked together recently about Apple/Google vCard implementation) and I appreciate you guys making a stand against this criminal behaviour. It could seem to some people that such a stance is easy, but no matter the strength of principle, when you see your business go offline and customers start banging at the doors for you to sort it out then the situation becomes more complex. So from this customer - keep up this stance, keep being transparent and I for one will stick by you guys without hesitation!
- Nyr 11y agoIndeed, I don't want them to fund criminals with my money like those Swiss guys did. DDoS mitigation isn't that expensive antway considering the size of FastMail.
- jacquesm 11y agoWill recommend fastmail.
- SuddsMcDuff 11y agoI'm a Fastmail customer and I support this stance.
- 11y ago
- duncan_bayne 11y agoAnd they've very clearly refused to pay the Danegeld demanded by their attackers. Smart, principled and yet another reason I'm a happy customer of theirs.
- bsder 11y agoIt looks like it's a general attack on email providers because they can't duck behind Cloudflare (or similar).
- jayess 11y agoI ditched gmail for fastmail two years ago and don't regret it for a second. I haven't noticed any disruption at all. Keep up the good work, guys.
- herbig 11y agoI've just started checking out Fastmail as a result of this thread. I hadn't heard of it before but I'm really into it. I've been trying out different email services lately in an effort to untether myself from Google, and this one seems like a really good choice. I'm not into the iOS themed non native Android app though, but being an Android developer I'm more averse to that nonsense. It seems to work great and is really fast though. Does anyone know if there is a native app planned? I'd be willing to pay a subscription for email if it's worth it. I'd love to hear others' experience with them.
- Karunamon 11y agoAnother +1 for Fastmail. I've been putting off migration from Google for months, but finally got around to moving things over. The migration was easy, their UI is shockingly quick (about halfway through processing ~20k archived messages), and it's got most of the features you could possibly want.
- gfosco 11y agoI've been using it for my main domain for over a year... It's a great service and I'm happy with it. I wanted to move away from GMail, without losing a good spam filtering solution, and it works very well. I recommend Fastmail.
- piker 11y agoGreat experience with our business account. We pay some trivial amount and our emails are out the door... fast. There's also the basic support that you would expect built in. Our experience is that they just "do one thing well". It's probably the second-to-last external dependency we'd [edit: internalize], with Stripe being the last and Analytics and Github being the first two, if that gives any perspective.
- Raphael 11y agoCouldn't you just use a generic email client?
- sbierwagen 11y ago
- 1rae 11y agoHushmail recently had it as well. https://help.hushmail.com/entries/107539976 https://help.hushmail.com/entries/107539976
- rdl 11y agoWow, seriously, fuck these people. There has to be a technical solution to this, since it's infeasible to find/fix/finish the Armada Collective. SMTP/IMAP/etc. are pretty crappy protocols in a lot of ways, but they're what everyone has deployed. They can be proxied like HTTP/HTTPS. There are spam/reputation issues with outgoing traffic, too, which makes this even more annoying.
- ppoint 11y agoThere is a SilverLine by F5 Networks: https://f5.com/products/platforms/silverline/f5-silverline-ddos-protection https://f5.com/products/platforms/silverline/f5-silverline-d....
- deleted 11y ago[deleted]
- nullrouted 11y agoThere are quite a few solutions to this: www.staminus.net www.arbor.net www.radware.com www.voxility.com
- mmaunder 11y agoWe recently were hit by multiple DDoS attacks over a weekend. We have our own servers in a data center with 5 redundant 1 Gbps links. The DDoS was 20Gbps according to the upstream providers. Our upstream implemented layer 7 mitigation which did an unbelievably effective job at stopping the attack in it's tracks. I don't know the tech that they used, but it performs deep packet inspection up to the application layer and they charge a modest additional fee for passing our traffic through that system. The effect was that our traffic dropped to very slightly below normal levels during the attack, which would indicate that there were probably a few false positives, but we didn't have a single customer complaint.
- pluma 11y agoAs a happy FastMail customer: thank you for standing up to them and announcing this in advance. More power to you!
- benmac 11y agoGiven that so many of us are now hosting on AWS, I'd like to ask the question - who has been hit with a DDOS attack / extortion letter while is hosting on AWS? It would seem that there's many old-tech companies hosting in data centers that would seem to be far more vulnerable to non-TCP attack vectors than AWS-hosted systems. Is that who is generally targeted here? Are there any stories, anecdotal or otherwise, about people getting hit with DDOS attacks while using AWS. Here's a talk by AWS on their measures against attacks - https://www.youtube.com/watch?v=Ys0gG1koqJA https://www.youtube.com/watch?v=Ys0gG1koqJA. The only thing short of Silverline etc defense that they seem to be lacking is the reporting dashboard indicating when they've defended against DDOS attacks. So has anyone received a letter from DD4BC and other miscreants whilst hosting their domains on AWS?
- alessioalex 11y agoWhat I love about FastMail is that I can neatly organize my emails into folders based by categories (via rules for subdomains), such as: newsletters folder (service_name@newsletters.mydomain.com), social (service_name@social.mydomain.com) etc. Everything is so clean and I don't even see any spam, they're top notch.
- Swinx43 11y agoI have not noticed any service interruption at all. Well done in handling this attack! This is just one more reason why I will remain a loyal FastMail customer.
- blowski 11y agoWe're a FastMail reseller, and so far we haven't experienced any problems - either on delivery, or the front end interface. So good job defending against it.
- DyslexicAtheist 11y agolast week protonmail, then zoho (their services were offline for 6 days) ... a new one every week it seems
- deleted 11y ago[deleted]