8 ms·

Good morning. I'm going to be here to answer specific questions, and I owe you a personal response to this as well, which I'm about to start working on! There

Good morning. I'm going to be here to answer specific questions, and I owe you a personal response to this as well, which I'm about to start working on! There is no doubt that in this specific case our human factor screwed up, and I'm really sorry about that. First I'm going to post the standard response that our team has written for any new support tickets that come in about this today, then write my own personal apology and response here. --- Thanks for getting in touch with us about the report on Hacker News about our security procedures. As we say in our recent post about security at https://blog.fastmail.com/2017/12/05/the-fastmail-security-mindset/ https://blog.fastmail.com/2017/12/05/the-fastmail-security-m..., security is a process, not a checkbox. We do our best to be continually improving and upgrading our security procedures, and offering our security-minded customers the most robust, industry-standard options possible. However, we have been less diligent about forcing older accounts to upgrade their security settings. With a range of possible security option states, customer support is occasionally placed in a position to make a judgement call. As the post indicates, the incident in question happened immediately after a major round of security changes. There’s no way around it; someone made an exception they shouldn't have. Social engineering is always one of our biggest concerns. As any number of well-known break-ins have demonstrated, the "best" security hack is often to sidestep it. Since that incident, we have taken substantially more aggressive steps to close off avenues of attack and human review. We are constantly trying to narrow the number of accounts that even can go to a human for review, and for those that must go to a human to provide as much notice as possible to the account owner before possibly allowing the attacker to have access. For instance, some cases take 24 hours before the reset password goes into effect. If you are a legitimate account owner, this has the often frustrating side effect of locking you out of your account for 24 hours. But, if you have been attacked, this gives you the opportunity to keep the attacker out. Thank you for sharing your concern with us, and I hope we’ve addressed yours.

dannysu 9y ago
This can be enough for me to consider leaving depending on how it's fixed. This response says absolutely nothing about how the vulnerability is prevented in the future. It's just a bunch of vague promises and mumbo jumbo. What specific procedures are in place to prevent it? At a minimum, I expect to see something specific like when you guys almost lost your domain because of Gandi [1]. And even then, can I have an option to select absolutely no human intervention possible? Having any human intervention is simply not acceptable. I already have multiple ways of recovering my account, and I never, ever want human assistance on this. I use a password manager, and I will never, ever need FastMail assistance on login. [1]: https://blog.fastmail.com/2014/04/10/when-two-factor-authentication-is-not-enough/ https://blog.fastmail.com/2014/04/10/when-two-factor-authent...
- ilikepi 9y ago
  > And even then, can I have an option to select absolutely no human intervention possible? > I already have multiple ways of recovering my account, and I never, ever want human assistance on this. Yep, I'll second this feature request. Put as many disclaimers and confirmation mechanisms on it as you need to in order to keep people from accidentally enabling it. I will happily assume responsibility for it.
- dannysu 9y ago
  Also, could you please add ability to get a phone call (instead of text message) to receive recovery options? That way I can setup my grandparents' phone number or something obscure as yet another recovery option. And then, please let me lock down any possibility of your support staff screwing up.
  - brongondwana 9y ago
    That's a possibility. We've talked about using something like twillio to read the message out. We haven't had the feedback that this is in high demand. It also brings issues of its own. Home phones are hard to block number on, and it could be used to troll people in the middle of their night. We need to consider those risks too - it's not a simple and obvious win.
- bad_user 9y ago
  At a minimum, normally when speaking of the other big mail providers, you wouldn’t get an explanation on a public forum at all. Especially because technical folks like us don’t know to communicate well, words can be misinterpreted, etc. It’s actually not a good strategy to respond to such concerns in public. Also in my opinion, people that make threats of leaving in public unless certain demands are met usually have their mind set already and can’t be swayed. As for never needing human assistance, never say never — if relying solely on your password manager, I hope you have a digital last will for your spouse or children.
  - dannysu 9y ago
    I understand what you're saying, but I think perhaps we can agree that the current response is insufficient? Have you taken a look at the link I supplied above where FastMail wrote about how 2fa protection could be bypassed at Gandi? They were very specific and clear about the recommendations being implemented. Now, compare that to their current response. I think definitely the difference can be seen. This is serious stuff. And, I'm absolutely serious about never needing human assistance. I already have mechanism setup for my family to retrieve my digital assets should I disappear tomorrow. I worked on this together with my wife. I know most people have not thought about this and you're right in your skepticism, but I'm serious.
tptacek 9y ago
Exactly which employees in your organization have the ability to alter recovery email settings? How many of those employees are there? In what fashion do you audit and track the activities of those employees? What training are these employees given to avoid social engineering? What firm provides the courseware? What's the escalation process for complicated, non-no-brainer reset situations? If a support person isn't absolutely sure whether they should reset something, how do they get a second opinion? Are the support people who are entitled and able to make these changes incentivized to close tickets as quickly as possible? Do you monitor "out-of-process" changes to recovery email and password settings, so that you can see trends over time and by particular staff members? Has any third party security firm assessed your service recently specifically for this attack vector, for instance by conducting social engineering testing against your support staff? What's the firm? How are you MINIMIZING, rather than just improving, the "human factors" involved in assessing whether accounts can be altered based on anonymous incoming callers and requesters? This is a HUGE, TERRIFYING vulnerability. Email providers are the single most important security service people use; if your email is compromised, many (most!) of your other services are compromised as well.
- brongondwana 9y ago
  Wow, that's a lot of questions, and I can't answer all of them without creating security risks! Our absolute focus is on minimizing the human factors. In the past year and a bit since that incident, we have improved our escalation policies and support training, as well as let some support staff go. But more importantly, we now have an automated account recovery system which can be used to verify ownership of the account using a number of different factors (not all of which I'd like to talk about in public - again, if an attacker knows the full algorithm it helps them game it).
  - tptacek 9y ago
    Which of these questions can you not answer without creating security risks? I didn't ask you anything about your automated system. Is it possible under any set of circumstances for your human employees to alter accounts? If the automated system fails, are accountholders out of luck?
analogist 9y ago
Hi Bron. I'm a customer of both Fastmail and GSuite, and I have enjoyed your service for a few years now. I still use Fastmail for some things, like sieve, and very much will continue paying just for the ongoing development of open-standard email like JMAP. But there are definitely a few things that I can't shake when I learned about them that very much pertains to the security mindset that prevents me from moving my primary emails onto Fastmail. Security paradigms have been steadily moving beyond a hard-boundary-soft-center, to a defense-in-depth, distrust-your-own-services model. I was alarmed to learn last year, for example, that you use OpenVPN with fixed symmetric keys (--secret) rather than TLS with any forward secrecy (--tls-auth) for VPN between your NYI and AMS datacenters. https://blog.fastmail.com/2016/12/19/secure-datacentre-interlinks/ https://blog.fastmail.com/2016/12/19/secure-datacentre-inter... Presumably, running datalinks like this means you would have to have perfect trust in your long term key management and rotation. Is that something you plan on improving in the future? Similarly -- I stumbled on this entirely by accident after your blog post about moving datacenters -- that your head of security ops & infrastructure tweeted "I will probably root my phone soon because Samsung's emoji set is worse than not having convenient OTA updates" https://twitter.com/robn/status/919194089920311296 https://twitter.com/robn/status/919194089920311296 I don't want to conflate anything -- a tweet on an engineer's own time about their personal devices isn't by itself a security problem. But it does reflect on the security mindset. If you had a BYOD policy, and this phone did end up being flashed to Lineage and be 3 patch levels behind (esp with Android's track record of RCE-via-media CVEs), this could definitely become a weakness on your entire infrastructure, and thereby on all of us as customers. This is the type of thing I couldn't shake after learning about it. Of course, trust has to be placed somewhere. You have to be able to place trust on your ops and your infrastructure, but that's also a process, not a checkbox. People and devices can be trusted a little less in the overall security system, to provide redundant security. Could you clarify your position on how your staff is trained about the human weak points, security as a lifestyle if you're security and ops, and how your security mindset incorporates defense in depth?
- bad_user 9y ago
  If an Android phone connecting to the company’s WiFi or the user’s email and whatnot is enough to compromise the infrastructure, then the company has bigger problems. I’ve worked in companies with liberal BYOD policies for portable devices, but also tasted really restricted environments and such environments are basically highly regulated security theaters. Users do stupid things of course and in corporations it’s worth it to restrict their devices, but restricting developers on what they can install and do on their own devices has a negative ROI and doesn’t go well. If you can’t trust a dev to manage his own phone, you can’t trust him to build your infrastructure either. And yes, we make mistakes as we are only human, which is why a phone should not be enough to compromise that infrastructure anyway. PS: your mention of that Twitter account is creepy.

dannysu 9y ago

This can be enough for me to consider leaving depending on how it's fixed. This response says absolutely nothing about how the vulnerability is prevented in the future. It's just a bunch of vague promises and mumbo jumbo. What specific procedures are in place to prevent it? At a minimum, I expect to see something specific like when you guys almost lost your domain because of Gandi [1]. And even then, can I have an option to select absolutely no human intervention possible? Having any human intervention is simply not acceptable. I already have multiple ways of recovering my account, and I never, ever want human assistance on this. I use a password manager, and I will never, ever need FastMail assistance on login. [1]: https://blog.fastmail.com/2014/04/10/when-two-factor-authentication-is-not-enough/ https://blog.fastmail.com/2014/04/10/when-two-factor-authent...

ilikepi 9y ago
> And even then, can I have an option to select absolutely no human intervention possible? > I already have multiple ways of recovering my account, and I never, ever want human assistance on this. Yep, I'll second this feature request. Put as many disclaimers and confirmation mechanisms on it as you need to in order to keep people from accidentally enabling it. I will happily assume responsibility for it.
dannysu 9y ago
Also, could you please add ability to get a phone call (instead of text message) to receive recovery options? That way I can setup my grandparents' phone number or something obscure as yet another recovery option. And then, please let me lock down any possibility of your support staff screwing up.
- brongondwana 9y ago
  That's a possibility. We've talked about using something like twillio to read the message out. We haven't had the feedback that this is in high demand. It also brings issues of its own. Home phones are hard to block number on, and it could be used to troll people in the middle of their night. We need to consider those risks too - it's not a simple and obvious win.
bad_user 9y ago
At a minimum, normally when speaking of the other big mail providers, you wouldn’t get an explanation on a public forum at all. Especially because technical folks like us don’t know to communicate well, words can be misinterpreted, etc. It’s actually not a good strategy to respond to such concerns in public. Also in my opinion, people that make threats of leaving in public unless certain demands are met usually have their mind set already and can’t be swayed. As for never needing human assistance, never say never — if relying solely on your password manager, I hope you have a digital last will for your spouse or children.
- dannysu 9y ago
  I understand what you're saying, but I think perhaps we can agree that the current response is insufficient? Have you taken a look at the link I supplied above where FastMail wrote about how 2fa protection could be bypassed at Gandi? They were very specific and clear about the recommendations being implemented. Now, compare that to their current response. I think definitely the difference can be seen. This is serious stuff. And, I'm absolutely serious about never needing human assistance. I already have mechanism setup for my family to retrieve my digital assets should I disappear tomorrow. I worked on this together with my wife. I know most people have not thought about this and you're right in your skepticism, but I'm serious.

tptacek 9y ago

Exactly which employees in your organization have the ability to alter recovery email settings? How many of those employees are there? In what fashion do you audit and track the activities of those employees? What training are these employees given to avoid social engineering? What firm provides the courseware? What's the escalation process for complicated, non-no-brainer reset situations? If a support person isn't absolutely sure whether they should reset something, how do they get a second opinion? Are the support people who are entitled and able to make these changes incentivized to close tickets as quickly as possible? Do you monitor "out-of-process" changes to recovery email and password settings, so that you can see trends over time and by particular staff members? Has any third party security firm assessed your service recently specifically for this attack vector, for instance by conducting social engineering testing against your support staff? What's the firm? How are you MINIMIZING, rather than just improving, the "human factors" involved in assessing whether accounts can be altered based on anonymous incoming callers and requesters? This is a HUGE, TERRIFYING vulnerability. Email providers are the single most important security service people use; if your email is compromised, many (most!) of your other services are compromised as well.

brongondwana 9y ago
Wow, that's a lot of questions, and I can't answer all of them without creating security risks! Our absolute focus is on minimizing the human factors. In the past year and a bit since that incident, we have improved our escalation policies and support training, as well as let some support staff go. But more importantly, we now have an automated account recovery system which can be used to verify ownership of the account using a number of different factors (not all of which I'd like to talk about in public - again, if an attacker knows the full algorithm it helps them game it).
- tptacek 9y ago
  Which of these questions can you not answer without creating security risks? I didn't ask you anything about your automated system. Is it possible under any set of circumstances for your human employees to alter accounts? If the automated system fails, are accountholders out of luck?

analogist 9y ago

Hi Bron. I'm a customer of both Fastmail and GSuite, and I have enjoyed your service for a few years now. I still use Fastmail for some things, like sieve, and very much will continue paying just for the ongoing development of open-standard email like JMAP. But there are definitely a few things that I can't shake when I learned about them that very much pertains to the security mindset that prevents me from moving my primary emails onto Fastmail. Security paradigms have been steadily moving beyond a hard-boundary-soft-center, to a defense-in-depth, distrust-your-own-services model. I was alarmed to learn last year, for example, that you use OpenVPN with fixed symmetric keys (--secret) rather than TLS with any forward secrecy (--tls-auth) for VPN between your NYI and AMS datacenters. https://blog.fastmail.com/2016/12/19/secure-datacentre-interlinks/ https://blog.fastmail.com/2016/12/19/secure-datacentre-inter... Presumably, running datalinks like this means you would have to have perfect trust in your long term key management and rotation. Is that something you plan on improving in the future? Similarly -- I stumbled on this entirely by accident after your blog post about moving datacenters -- that your head of security ops & infrastructure tweeted "I will probably root my phone soon because Samsung's emoji set is worse than not having convenient OTA updates" https://twitter.com/robn/status/919194089920311296 https://twitter.com/robn/status/919194089920311296 I don't want to conflate anything -- a tweet on an engineer's own time about their personal devices isn't by itself a security problem. But it does reflect on the security mindset. If you had a BYOD policy, and this phone did end up being flashed to Lineage and be 3 patch levels behind (esp with Android's track record of RCE-via-media CVEs), this could definitely become a weakness on your entire infrastructure, and thereby on all of us as customers. This is the type of thing I couldn't shake after learning about it. Of course, trust has to be placed somewhere. You have to be able to place trust on your ops and your infrastructure, but that's also a process, not a checkbox. People and devices can be trusted a little less in the overall security system, to provide redundant security. Could you clarify your position on how your staff is trained about the human weak points, security as a lifestyle if you're security and ops, and how your security mindset incorporates defense in depth?

bad_user 9y ago
If an Android phone connecting to the company’s WiFi or the user’s email and whatnot is enough to compromise the infrastructure, then the company has bigger problems. I’ve worked in companies with liberal BYOD policies for portable devices, but also tasted really restricted environments and such environments are basically highly regulated security theaters. Users do stupid things of course and in corporations it’s worth it to restrict their devices, but restricting developers on what they can install and do on their own devices has a negative ROI and doesn’t go well. If you can’t trust a dev to manage his own phone, you can’t trust him to build your infrastructure either. And yes, we make mistakes as we are only human, which is why a phone should not be enough to compromise that infrastructure anyway. PS: your mention of that Twitter account is creepy.