7 ms·

Fastmail.com suffering DDOS attack

69 points by moonlighter 10y ago

tshtf 10y ago
Not many alternatives: 1.) Run your own SMTP infrastructure. Setup SPF/DKIM/DMARC. Realize your outbound emails still don't always reach their destination. Also you have to fight inbound SPAM. 2.) Use gmail or Google Apps. Things just work. Cede control to Google.
- _quick_q 10y ago
  >> Use gmail or Google Apps. Things just work. Cede control to Google. Google is not without its problems. Just now on HN, in fact: https://news.ycombinator.com/item?id=12008365 https://news.ycombinator.com/item?id=12008365
- greyone 10y ago
  (2) Hmmm..well Google Calendar just had an outage. I'll stick with Fastmail is very reliable and independent. They can't really be blamed for a DOS attack.
  - delroth 10y ago
    > They can't really be blamed for a DOS attack. That doesn't really hold when their competitors have teams of engineers working on DDoS mitigations and successfully handle most of these attacks. Falling to a DoS is a technical failure just like other causes of outages.
    - corford 10y ago
      A ~1 hour response time from identifying to mitigating the DDoS isn't exactly bad. I'd say their engineers did a pretty good job.
      - leblancfg 10y ago
        Security really isn't my strong suit, but isn't it standard practice to automate DDoS mitigation strategies?
        corford 10y ago
        Depends on the type and size of the DDoS and where on the network path it's hitting.
        abpavel 10y ago
        DoS is a threat from identifiable source, like Russia. DDoS looks just like normal users, like terrorist sleeping cells.
- toast0 10y ago
  > 2.) Use gmail or Google Apps. Things just work. Cede control to Google. Until they don't, and then good luck with getting ahold of someone who can actually do anything to fix your problem(s).
  - cbg0 10y ago
    If you're paying for Google Apps, you get e-mail and phone support.
    - toast0 10y ago
      Yes -- but the people you can contact may not have the ability to fix problems if they're complex. For example, we wanted to disable clicking on links in email for our users because of phishing -- support wasn't able to help with that.
  - nmjohn 10y ago
    I pay $5 / month for google apps for my personal domain. I've had to contact support twice, the first time I got a phone call < 10 minutes after my email, and the second was < 20 minutes via email. In my experience, Google support is _excellent_ for their paid products.
    - eropple 10y ago
      That's where I'm at. I've had similar good luck with Office 365, too--similar frequency of problems and responsiveness when I've had them.
- troydavis 10y ago
  #2 is generally true of FastMail as well. In 2 years as a customer, it's been close to bulletproof.
  - abpavel 10y ago
    Used Microsoft/Google/Fastmail in parallel for my company for 6 months. Fastmail won.
- peterwwillis 10y ago
  Don't do #1 if you want to reliably accept or send mail. Fun for experimenting/learning, though. Also, you can buy your own domain, and have Google merely act as the SMTP relay and temporary storage. They can also forward your mail to another server for you.
  - CaptSpify 10y ago
    I'm gonna disagree with you. I've run my own mail server for my primary email for years and very rarely have any issues.
  - snassar 10y ago
    While there are pain points when it comes to hosting your own mail, it isn't nearly as difficult as you make it. It gets more difficult if you are providing a service that has to have uptime guarantees or are providing mail to many users but if you take the time to learn and educate yourself on current standards, hosting your own mail for fun and for profit is a doable thing that more developers and admins should do. We are at a time when we actually have relatively easy to use software to managing mail servers, so us it. For what it is worth, I have not had problems with Google, Microsoft, Yahoo, or domains that use their services whether it comes to sending or receiving. Sometimes a server is stuck in an SPAM prevention queue or I might have to whitelist a particularly silly server, but that doesn't happen very often.
- JoshTriplett 10y ago
  > 1.) Run your own SMTP infrastructure. Setup SPF/DKIM/DMARC. Realize your outbound emails still don't always reach their destination. Also you have to fight inbound SPAM. And if someone wants to DDoS you, you're a lot more vulnerable than a major provider like Fastmail. Personally, I use a hybrid solution: I use Gandi's SMTP servers for outbound and inbound mail, but I run my own IMAP server for unlimited storage under my control.
  - b101010 10y ago
    If the attacker has ever seen the headers of a message you sent through fastmails SMTP service they have your public IP (Received from header) and can Dos you directly anyway. They do something similar with their webmail service, but the data is encrypted so it can't be read by a third party. https://www.fastmail.com/about/reportabuse.html https://www.fastmail.com/about/reportabuse.html (last paragraph) EDIT: Fastmail is fairly priced (for me) and i like the features they offer but i wish they wouldn't do this (or rather, i wish they would do the same for the SMTP service as they do for the webmail service)
    - unhammer 10y ago
      What happens if you send through port 565 instead of 587/465? :)
  - dingaling 10y ago
    I've never been subject to DOS in fifteen-ish years of running SMTP. Personal users aren't of interest to DOS ransomers since a call to the ISP will drop their traffic at the border. Without SLAs costing me money, as would be the case with a big provider, I coulf outlast the DOS. Just inconvenient and annoying.
- dangrossman 10y ago
  Rackspace Mail hosts 3 million paid mailboxes, and comes with a 100% availability SLA and 24/7/365 support. Google's not the only game in town.
  - feld 10y ago
    Sure, but I know nothing about Rackspace and refuse to trust them as a result. Plus I doubt they have anything that can compete with Fastmail's web interface, powered by the JMAP protocol they authored. So here's what I know about Fastmail that I want to know about any provider I host with: - I know Fastmail will refuse any requests from the US govt to access data because they're Australian and legally the request has to come from an Australian court - I know Fastmail's servers use encrypted storage - I know the specs of Fastmail's servers (they post them on their Help/FAQ) - I know Fastmail's actually trying to make email better, obvious by their invention of the JMAP protocol - I trust Fastmail to be able to recover from any serious issues because they actually have on staff developers of Cyrus, the open source IMAP software they use. This means their admins have actual recourse when Bad Things^TM happen, vs the usual when an admin runs out of options, eg, "let's just post on mailing lists and hope we can find an answer" - I know the fine details of how their spam filtering works, because it's publicly documented. (and it's quite well integrated with some tricks I couldn't employ at my last ISP job as I didn't have developers to assist) - I know their infrastructure is primarily hosted in NYI with the backup in Iceland. - I know they are serious about security, as they've been proponents of full SSL/TLS vs STARTTLS which could be MITM and downgraded (yes, many MTAs will let you require STARTTLS, but there are always possibilities of client bugs that could be exploited when you let an attacker intercept plaintext and inject data before the upgrade to a validated TLS session) - I know how their backups work, because it's documented and I also have the ability to undelete emails which almost no provider gives the end user. - I know their support is responsive and competent, as they've actually fixed Webmail bugs and put them into production for me within 48 hours - Fastmail does PUSH email on iOS, while GMail, Rackspace, and most other providers don't offer this because it requires custom integration with Apple's Push Notifications service. tl;dr yeah, the average provider might promise the moon but can they actually deliver when the shit hits the fan? will they actually strive to please their users and make the internet a better place? probably not.
peterwwillis 10y ago
Something i'm realizing more and more... What the hell do I really need remotely hosted mail for? We all know mail is insecure. Unless you look really really hard, you aren't sure if the mail you received was spoofed or modified, a child can spoof mail and any MitM can modify it. So in general you can't trust your mail anyway, even if it's received by a reputable company. Sending mail is almost just as subjective... a random ISP's mail smarthost is just as good for getting your mail delivered as a hosted mail provider. All I really need is a way to get my mails, once. Once you have the mail, you can back it up to an infinite number of places (Git repository, anyone?) if in the future you need to search it. So really, the only thing I need is 1) to receive mail, 2) to filter the spam, and 3) to keep a backup of my mail somewhere. Considering this, why do we even need domain-specific mail? Like, myusername at Gmail dotcom, for example. I don't need it sent to GMail... I need it sent to me. I don't care what server receives it. I don't even need to store my mail there once i've read it - I can keep it offline, and back it up to remote repositories to search. With a format + protocol like Git, this would be fast, efficient, reliable, secure, and compatible. So really, if we just had a distributed decentralized peer-to-peer mail network, a unique address system, and a retrofitted mail storage protocol (IMAP5?), we could send mail anywhere, receive it anywhere, store it anywhere, and spam could be filtered by whatever product or company was hosting your Git backup. With the new address system we could even build in personal crypto keys and teach people how to send real, honest-to-god, secure mails, potentially even anonymously. Now somebody tell me how someone already thought of this and how it won't work :-)
- cjcole 10y ago
  > format + protocol like Git > distributed decentralized peer-to-peer network > a unique address system > personal crypto keys Funny. You just described a perfect fit for Urbit.
  - peterwwillis 10y ago
    Urbit seems to try to provide a centralized, personalized model for application services. I don't think centralizing is the way to go, mainly because my entire digital life is dependent on de-centralizing all my services. They try to make it out to be like some kind of container you can put anywhere, but that's like saying we should all use one kind of bag for everything we ever need to carry in our daily lives. There's good reasons I have 10 different kinds of bags at home.
reptation 10y ago
Is this part of a more general attack on Internet infrastructure today in the U.S.? http://downdetector.com/ http://downdetector.com/ has been showing many sites with issues (Google, Outlook, etc.)
- misframer 10y ago
  Does Google ever have DDOS problems?
  - Tiksi 10y ago
    Just going off the public numbers for one of google's ASNs[0], with the amount of public peers they have (and likely far more private peers, and I'm just assuming google doesn't buy transit and ignoring that), it's unlikely that a DDoS would ever cause an outage for more than a small subset of people. You may be able to saturate a link or two, but that's not difficult to route around generally. That is of course only for network saturation DDoS, I'm sure there are ways google could be ddosed at the application or server level, but they likely have enough infra in place to be able to eat the attack without anyone outside of google noticing. [0] https://www.peeringdb.com/net/433 https://www.peeringdb.com/net/433
616c 10y ago
I guess this had to be recent. I had not noticed all day, but it seems to be the last few hours?
gnopgnip 10y ago
There is a significant outage for Office 365 mail today also.
PaulHoule 10y ago
That's why it wasn't so fast today.
abpavel 10y ago
As an avid fastmail user I did notice slight loading delays, but I attributed them to my wifi/iphone. I mean... It's mail, not live SCADA telemetry...
tracker1 10y ago
Given the timeframe, I commend them for keeping the notices open and public. It's nice to see. When I went through the A(zure)pocolypse a few years back, didn't see anything for about 15-20 minutes... though admittedly if I weren't in the middle of testing something may have not noticed for a while either. All said, you can't mitigate all DDoS easily, and it's nice to see that they were pretty responsive and open... Also, while email can be very important, it shouldn't be eminently critical.

69 points by moonlighter 10y ago

tshtf 10y ago

Not many alternatives: 1.) Run your own SMTP infrastructure. Setup SPF/DKIM/DMARC. Realize your outbound emails still don't always reach their destination. Also you have to fight inbound SPAM. 2.) Use gmail or Google Apps. Things just work. Cede control to Google.

_quick_q 10y ago
>> Use gmail or Google Apps. Things just work. Cede control to Google. Google is not without its problems. Just now on HN, in fact: https://news.ycombinator.com/item?id=12008365 https://news.ycombinator.com/item?id=12008365
greyone 10y ago
(2) Hmmm..well Google Calendar just had an outage. I'll stick with Fastmail is very reliable and independent. They can't really be blamed for a DOS attack.
- delroth 10y ago
  > They can't really be blamed for a DOS attack. That doesn't really hold when their competitors have teams of engineers working on DDoS mitigations and successfully handle most of these attacks. Falling to a DoS is a technical failure just like other causes of outages.
  - corford 10y ago
    A ~1 hour response time from identifying to mitigating the DDoS isn't exactly bad. I'd say their engineers did a pretty good job.
    - leblancfg 10y ago
      Security really isn't my strong suit, but isn't it standard practice to automate DDoS mitigation strategies?
      - corford 10y ago
        Depends on the type and size of the DDoS and where on the network path it's hitting.
      - abpavel 10y ago
        DoS is a threat from identifiable source, like Russia. DDoS looks just like normal users, like terrorist sleeping cells.
toast0 10y ago
> 2.) Use gmail or Google Apps. Things just work. Cede control to Google. Until they don't, and then good luck with getting ahold of someone who can actually do anything to fix your problem(s).
- cbg0 10y ago
  If you're paying for Google Apps, you get e-mail and phone support.
  - toast0 10y ago
    Yes -- but the people you can contact may not have the ability to fix problems if they're complex. For example, we wanted to disable clicking on links in email for our users because of phishing -- support wasn't able to help with that.
- nmjohn 10y ago
  I pay $5 / month for google apps for my personal domain. I've had to contact support twice, the first time I got a phone call < 10 minutes after my email, and the second was < 20 minutes via email. In my experience, Google support is _excellent_ for their paid products.
  - eropple 10y ago
    That's where I'm at. I've had similar good luck with Office 365, too--similar frequency of problems and responsiveness when I've had them.
troydavis 10y ago
#2 is generally true of FastMail as well. In 2 years as a customer, it's been close to bulletproof.
- abpavel 10y ago
  Used Microsoft/Google/Fastmail in parallel for my company for 6 months. Fastmail won.
peterwwillis 10y ago
Don't do #1 if you want to reliably accept or send mail. Fun for experimenting/learning, though. Also, you can buy your own domain, and have Google merely act as the SMTP relay and temporary storage. They can also forward your mail to another server for you.
- CaptSpify 10y ago
  I'm gonna disagree with you. I've run my own mail server for my primary email for years and very rarely have any issues.
- snassar 10y ago
  While there are pain points when it comes to hosting your own mail, it isn't nearly as difficult as you make it. It gets more difficult if you are providing a service that has to have uptime guarantees or are providing mail to many users but if you take the time to learn and educate yourself on current standards, hosting your own mail for fun and for profit is a doable thing that more developers and admins should do. We are at a time when we actually have relatively easy to use software to managing mail servers, so us it. For what it is worth, I have not had problems with Google, Microsoft, Yahoo, or domains that use their services whether it comes to sending or receiving. Sometimes a server is stuck in an SPAM prevention queue or I might have to whitelist a particularly silly server, but that doesn't happen very often.
JoshTriplett 10y ago
> 1.) Run your own SMTP infrastructure. Setup SPF/DKIM/DMARC. Realize your outbound emails still don't always reach their destination. Also you have to fight inbound SPAM. And if someone wants to DDoS you, you're a lot more vulnerable than a major provider like Fastmail. Personally, I use a hybrid solution: I use Gandi's SMTP servers for outbound and inbound mail, but I run my own IMAP server for unlimited storage under my control.
- b101010 10y ago
  If the attacker has ever seen the headers of a message you sent through fastmails SMTP service they have your public IP (Received from header) and can Dos you directly anyway. They do something similar with their webmail service, but the data is encrypted so it can't be read by a third party. https://www.fastmail.com/about/reportabuse.html https://www.fastmail.com/about/reportabuse.html (last paragraph) EDIT: Fastmail is fairly priced (for me) and i like the features they offer but i wish they wouldn't do this (or rather, i wish they would do the same for the SMTP service as they do for the webmail service)
  - unhammer 10y ago
    What happens if you send through port 565 instead of 587/465? :)
- dingaling 10y ago
  I've never been subject to DOS in fifteen-ish years of running SMTP. Personal users aren't of interest to DOS ransomers since a call to the ISP will drop their traffic at the border. Without SLAs costing me money, as would be the case with a big provider, I coulf outlast the DOS. Just inconvenient and annoying.
dangrossman 10y ago
Rackspace Mail hosts 3 million paid mailboxes, and comes with a 100% availability SLA and 24/7/365 support. Google's not the only game in town.
- feld 10y ago
  Sure, but I know nothing about Rackspace and refuse to trust them as a result. Plus I doubt they have anything that can compete with Fastmail's web interface, powered by the JMAP protocol they authored. So here's what I know about Fastmail that I want to know about any provider I host with: - I know Fastmail will refuse any requests from the US govt to access data because they're Australian and legally the request has to come from an Australian court - I know Fastmail's servers use encrypted storage - I know the specs of Fastmail's servers (they post them on their Help/FAQ) - I know Fastmail's actually trying to make email better, obvious by their invention of the JMAP protocol - I trust Fastmail to be able to recover from any serious issues because they actually have on staff developers of Cyrus, the open source IMAP software they use. This means their admins have actual recourse when Bad Things^TM happen, vs the usual when an admin runs out of options, eg, "let's just post on mailing lists and hope we can find an answer" - I know the fine details of how their spam filtering works, because it's publicly documented. (and it's quite well integrated with some tricks I couldn't employ at my last ISP job as I didn't have developers to assist) - I know their infrastructure is primarily hosted in NYI with the backup in Iceland. - I know they are serious about security, as they've been proponents of full SSL/TLS vs STARTTLS which could be MITM and downgraded (yes, many MTAs will let you require STARTTLS, but there are always possibilities of client bugs that could be exploited when you let an attacker intercept plaintext and inject data before the upgrade to a validated TLS session) - I know how their backups work, because it's documented and I also have the ability to undelete emails which almost no provider gives the end user. - I know their support is responsive and competent, as they've actually fixed Webmail bugs and put them into production for me within 48 hours - Fastmail does PUSH email on iOS, while GMail, Rackspace, and most other providers don't offer this because it requires custom integration with Apple's Push Notifications service. tl;dr yeah, the average provider might promise the moon but can they actually deliver when the shit hits the fan? will they actually strive to please their users and make the internet a better place? probably not.

peterwwillis 10y ago

Something i'm realizing more and more... What the hell do I really need remotely hosted mail for? We all know mail is insecure. Unless you look really really hard, you aren't sure if the mail you received was spoofed or modified, a child can spoof mail and any MitM can modify it. So in general you can't trust your mail anyway, even if it's received by a reputable company. Sending mail is almost just as subjective... a random ISP's mail smarthost is just as good for getting your mail delivered as a hosted mail provider. All I really need is a way to get my mails, once. Once you have the mail, you can back it up to an infinite number of places (Git repository, anyone?) if in the future you need to search it. So really, the only thing I need is 1) to receive mail, 2) to filter the spam, and 3) to keep a backup of my mail somewhere. Considering this, why do we even need domain-specific mail? Like, myusername at Gmail dotcom, for example. I don't need it sent to GMail... I need it sent to me. I don't care what server receives it. I don't even need to store my mail there once i've read it - I can keep it offline, and back it up to remote repositories to search. With a format + protocol like Git, this would be fast, efficient, reliable, secure, and compatible. So really, if we just had a distributed decentralized peer-to-peer mail network, a unique address system, and a retrofitted mail storage protocol (IMAP5?), we could send mail anywhere, receive it anywhere, store it anywhere, and spam could be filtered by whatever product or company was hosting your Git backup. With the new address system we could even build in personal crypto keys and teach people how to send real, honest-to-god, secure mails, potentially even anonymously. Now somebody tell me how someone already thought of this and how it won't work :-)

cjcole 10y ago
> format + protocol like Git > distributed decentralized peer-to-peer network > a unique address system > personal crypto keys Funny. You just described a perfect fit for Urbit.
- peterwwillis 10y ago
  Urbit seems to try to provide a centralized, personalized model for application services. I don't think centralizing is the way to go, mainly because my entire digital life is dependent on de-centralizing all my services. They try to make it out to be like some kind of container you can put anywhere, but that's like saying we should all use one kind of bag for everything we ever need to carry in our daily lives. There's good reasons I have 10 different kinds of bags at home.

reptation 10y ago

Is this part of a more general attack on Internet infrastructure today in the U.S.? http://downdetector.com/ http://downdetector.com/ has been showing many sites with issues (Google, Outlook, etc.)

misframer 10y ago
Does Google ever have DDOS problems?
- Tiksi 10y ago
  Just going off the public numbers for one of google's ASNs[0], with the amount of public peers they have (and likely far more private peers, and I'm just assuming google doesn't buy transit and ignoring that), it's unlikely that a DDoS would ever cause an outage for more than a small subset of people. You may be able to saturate a link or two, but that's not difficult to route around generally. That is of course only for network saturation DDoS, I'm sure there are ways google could be ddosed at the application or server level, but they likely have enough infra in place to be able to eat the attack without anyone outside of google noticing. [0] https://www.peeringdb.com/net/433 https://www.peeringdb.com/net/433

616c 10y ago

I guess this had to be recent. I had not noticed all day, but it seems to be the last few hours?

gnopgnip 10y ago

There is a significant outage for Office 365 mail today also.

PaulHoule 10y ago

That's why it wasn't so fast today.

abpavel 10y ago

As an avid fastmail user I did notice slight loading delays, but I attributed them to my wifi/iphone. I mean... It's mail, not live SCADA telemetry...

tracker1 10y ago

Given the timeframe, I commend them for keeping the notices open and public. It's nice to see. When I went through the A(zure)pocolypse a few years back, didn't see anything for about 15-20 minutes... though admittedly if I weren't in the middle of testing something may have not noticed for a while either. All said, you can't mitigate all DDoS easily, and it's nice to see that they were pretty responsive and open... Also, while email can be very important, it shouldn't be eminently critical.