5 ms·

Wow, that's a lot of questions, and I can't answer all of them without creating security risks! Our absolute focus is on minimizing the human factors. In the

Wow, that's a lot of questions, and I can't answer all of them without creating security risks! Our absolute focus is on minimizing the human factors. In the past year and a bit since that incident, we have improved our escalation policies and support training, as well as let some support staff go. But more importantly, we now have an automated account recovery system which can be used to verify ownership of the account using a number of different factors (not all of which I'd like to talk about in public - again, if an attacker knows the full algorithm it helps them game it).

tptacek 9y ago
Which of these questions can you not answer without creating security risks? I didn't ask you anything about your automated system. Is it possible under any set of circumstances for your human employees to alter accounts? If the automated system fails, are accountholders out of luck?
- brongondwana 9y ago
  If the automated system fails and you have 2fa, then it gets escalated to the two most senior members of the security team. In some cases we haven't had sufficient information on the account to ever verify that account's owner, and they never got their account back. Some users refuse to give us enough information to allow us to later positively identify them - so yes, those people will be out of luck if they lose their credentials.
  - hitekker 9y ago
    At this point, you should write a blog post to address the myriad of concerns popping up in this thread.
    - brongondwana 9y ago
      We're working on that! It may not be finished today.
      - hitekker 9y ago
        That's fair. I respect that, as CEO, you're genuinely responding to your customers in this thread instead of fobbing it off to someone else. Hopefully, this can all be explained, resolved, and /or remedied in good time.
  - jf 9y ago
    I agree with hitekker, I'm feeling pretty nervous about being a FastMail customer right now and will start looking for a more secure alternative now. The main reason I moved to FastMail is because I stopped trusting Google to keep my mail secure.
    - tptacek 9y ago
      Google is the gold standard for email account service. Nobody in the industry does a better job at that one thing than Google does.
      - mythz 9y ago
        Could you share some info/links on what makes it the Gold Standard?
        pvg 9y ago
        Can you think of some info/links that would suggest the opposite?
        mythz 9y ago
        I'm not looking to discredit the claim, I'm genuinely curious to learn about what they've done to earn the Gold Standard from @tptacek Google were previously reading our emails for Ad purposes and some of their employees are still able to read our Emails, their privacy policy also indictates they will hand over our emails if requested by law enforcement which suggests it's weaker than protonmail.com end-to-end encryption: > All emails are secured automatically with end-to-end encryption. This means even we cannot decrypt and read your emails. As a result, your encrypted emails cannot be shared with third parties. If this is the case, how is Google being held as the Gold Standard?
        pvg 9y ago
        I'm genuinely curious to learn I get and am not questioning that. It's just that your curiosity doesn't seem to have motivated you to do a first pass of, I don't want to call it 'research', but just basic poking around on the topic. You want links and info from some dude on the internet because what he says contradicts stuff you know from... something a vendor said about their product. It's a totally sensible question but it's not some particularly arcane mystery to dig into. In tptacek's case, in a jiffy, you can bring up the 60-odd comments of his that mention 'Gmail' and get a reasonable idea of what he thinks of it and why. And if you think he's got it wrong, you can say, hey, tptacek, I think you're full of poop when you said [...]. And then maybe you can hash it out and one or both of you will learn something. But 'Citation, please', especially on trivially searchable topics mostly says 'I'm kind of curious, but I don't really care'. The person you're asking probably isn't going to care either.
js2 9y ago
I'm a FastMail customer. Your response is troubling to me in that it didn't answer most of tptacek's questions. It's troubling enough for me to start looking at other email providers. :-( I would like FM to provide something akin to Google's advanced protection program. Those of us who are careful not to lose our login credentials should not have to suffer a weak recovery process for the convenience of those who do. I personally would rather opt my account into a stronger recovery process even if I can't access my account for several days or a week or more.
- brongondwana 9y ago
  I have now responded in more detail - at the time I was busy trying to spread the love around, and also support my team as they dealt with the support requests and digesting the response on here. Check out the longer response here: https://news.ycombinator.com/item?id=15859024 https://news.ycombinator.com/item?id=15859024
  - js2 9y ago
    Thank you for the additional clarity. I apologize for not being more patient in allowing you to reply.
wglb 9y ago
Wow, that's a lot of questions, and I can't answer all of them without creating security risks Questions like these are not unreasonable for a customer to ask a service provider with respect to identity management and protection of that customer’s proprietary and confidential information. With respect to the first question “Exactly which employees have the ability to alter recovery email settings.” Not being able to have a prepared answer for this question suggests that you don’t have a formal policy or standard procedure around role based capabilities in your operation. The second question is an extension of the first. “In what fashion do you audit and track the activities of these employees?” Not being able to answer that question suggests that you don’t have an auditing process around employee actions with respect to account changes. “What training are these employees given to avoid social engineering?” Not being able to answer this question suggests that you don’t have such training in place. “What’s the escalation process for non-no-brainer reset situations?” If your processes are written down and staff are trained in them, a very simple description here would not create a security’s risk of any kind. Not doing so suggests that the process is not formally specified or is quite ad hoc. “Are the support people who are enabled and entitled to lose the tickets incentivized to close the tickets as soon as possible?” It seems that your internal security posture would make that clear, and it is unclear how stating that correctness is more important than speed in user account modification poses a security risk. I’ll pause here and summarize. Answering any of these questions is not going to pose a security risk unless such answers expose to your users reasonable measures that you are not taking or haven’t thought of.
- brongondwana 9y ago
  Which employees? At the time of this compromise, that list was all support staff as well as the technical staff in Melbourne. It is a specific role that's granted to specific people, to answer your question about having a procedure or policy. Today, that role is granted to a much more limited set of senior security staff (currently 3 people). Regular support staff can not alter security-sensitive details about accounts. If your account is owned by someone else (e.g. family or business, or part of a resold package) then they can still alter recovery options, as they own the account. In 2016 before we had automated account recovery, lost password was in the top 3 categories of ticket every single week! Every member of the support team dealt with multiple account-loss tickets per day, both forgotten password or stolen account. Stolen account losses are way down now we have app passwords, we often only have to block a single app password and notify the user rather than locking the entire account. Forgotten passwords have not reduced, but most people are able to recover using the automated tooling. --- In what fashion do we audit and track? A few ways - we log every API call at the lowest level. We log each override when the support person accesses user accounts against the ticket that they come in through, so we can see why they were accessing that user. We could always do with better tooling to introspect logs, but the data is all captured and can be followed through after the fact. Support staff have no way to wipe their audit trail. --- Training - in 2016 we didn't have much formal training for our support staff - they learned on the job from each other. We are very aware that this was a failing at that time. We have more training now. We did a lot of work at unifying our support teams across the FastMail and Pobox/Listbox family throughout 2017, and that led to better training and induction materials, as well as better internal reference material for support staff to use. Early in the induction process for all new support staff is a description of how social engineering works and a warning that urgency is often used in social engineering attempts, so when in doubt, slow down and get a second opinion (which leads to complaints about slow support, but that's the tradeoff here.) --- Escalation process - as mentioned earlier, if you had 2fa enabled then it has always gone straight to the senior security team, which is based in Melbourne and consists of our most experienced and trusted people. Neil (author of the blog post this HN refers to) is of course one of these people. With lost passwords no longer a highly common support request, all support tickets requesting manual account recovery are escalated to our senior team for review. --- Support people have no incentive to close tickets quickly. Absolutely. That is a bad metric, and it's not a metric we have ever used. Time to first response and time to followup responses are tracked, but there's no incentive to close tickets. This answer is a no brainer and I should have answered it in the first response - sorry. I was still rushing through initial responses at that time, and there were too many points in that post to think about them all at once and still respond quickly. The real-time nature of this hacker-news medium encourages fast answers above complete answers. I hope this longer response helps clear up remaining questions, at least to those who see it! --- There's another blog post coming soon about the account recovery system in particular, which addresses exactly how we've minimising human involvement in recovery decisions while not excessively punishing real human frailty amongst our customers.
deleted 9y ago
[deleted]

tptacek 9y ago

Which of these questions can you not answer without creating security risks? I didn't ask you anything about your automated system. Is it possible under any set of circumstances for your human employees to alter accounts? If the automated system fails, are accountholders out of luck?

brongondwana 9y ago
If the automated system fails and you have 2fa, then it gets escalated to the two most senior members of the security team. In some cases we haven't had sufficient information on the account to ever verify that account's owner, and they never got their account back. Some users refuse to give us enough information to allow us to later positively identify them - so yes, those people will be out of luck if they lose their credentials.
- hitekker 9y ago
  At this point, you should write a blog post to address the myriad of concerns popping up in this thread.
  - brongondwana 9y ago
    We're working on that! It may not be finished today.
    - hitekker 9y ago
      That's fair. I respect that, as CEO, you're genuinely responding to your customers in this thread instead of fobbing it off to someone else. Hopefully, this can all be explained, resolved, and /or remedied in good time.
- jf 9y ago
  I agree with hitekker, I'm feeling pretty nervous about being a FastMail customer right now and will start looking for a more secure alternative now. The main reason I moved to FastMail is because I stopped trusting Google to keep my mail secure.
  - tptacek 9y ago
    Google is the gold standard for email account service. Nobody in the industry does a better job at that one thing than Google does.
    - mythz 9y ago
      Could you share some info/links on what makes it the Gold Standard?
      - pvg 9y ago
        Can you think of some info/links that would suggest the opposite?
        mythz 9y ago
        I'm not looking to discredit the claim, I'm genuinely curious to learn about what they've done to earn the Gold Standard from @tptacek Google were previously reading our emails for Ad purposes and some of their employees are still able to read our Emails, their privacy policy also indictates they will hand over our emails if requested by law enforcement which suggests it's weaker than protonmail.com end-to-end encryption: > All emails are secured automatically with end-to-end encryption. This means even we cannot decrypt and read your emails. As a result, your encrypted emails cannot be shared with third parties. If this is the case, how is Google being held as the Gold Standard?
        pvg 9y ago
        I'm genuinely curious to learn I get and am not questioning that. It's just that your curiosity doesn't seem to have motivated you to do a first pass of, I don't want to call it 'research', but just basic poking around on the topic. You want links and info from some dude on the internet because what he says contradicts stuff you know from... something a vendor said about their product. It's a totally sensible question but it's not some particularly arcane mystery to dig into. In tptacek's case, in a jiffy, you can bring up the 60-odd comments of his that mention 'Gmail' and get a reasonable idea of what he thinks of it and why. And if you think he's got it wrong, you can say, hey, tptacek, I think you're full of poop when you said [...]. And then maybe you can hash it out and one or both of you will learn something. But 'Citation, please', especially on trivially searchable topics mostly says 'I'm kind of curious, but I don't really care'. The person you're asking probably isn't going to care either.

js2 9y ago

I'm a FastMail customer. Your response is troubling to me in that it didn't answer most of tptacek's questions. It's troubling enough for me to start looking at other email providers. :-( I would like FM to provide something akin to Google's advanced protection program. Those of us who are careful not to lose our login credentials should not have to suffer a weak recovery process for the convenience of those who do. I personally would rather opt my account into a stronger recovery process even if I can't access my account for several days or a week or more.

brongondwana 9y ago
I have now responded in more detail - at the time I was busy trying to spread the love around, and also support my team as they dealt with the support requests and digesting the response on here. Check out the longer response here: https://news.ycombinator.com/item?id=15859024 https://news.ycombinator.com/item?id=15859024
- js2 9y ago
  Thank you for the additional clarity. I apologize for not being more patient in allowing you to reply.

wglb 9y ago

Wow, that's a lot of questions, and I can't answer all of them without creating security risks Questions like these are not unreasonable for a customer to ask a service provider with respect to identity management and protection of that customer’s proprietary and confidential information. With respect to the first question “Exactly which employees have the ability to alter recovery email settings.” Not being able to have a prepared answer for this question suggests that you don’t have a formal policy or standard procedure around role based capabilities in your operation. The second question is an extension of the first. “In what fashion do you audit and track the activities of these employees?” Not being able to answer that question suggests that you don’t have an auditing process around employee actions with respect to account changes. “What training are these employees given to avoid social engineering?” Not being able to answer this question suggests that you don’t have such training in place. “What’s the escalation process for non-no-brainer reset situations?” If your processes are written down and staff are trained in them, a very simple description here would not create a security’s risk of any kind. Not doing so suggests that the process is not formally specified or is quite ad hoc. “Are the support people who are enabled and entitled to lose the tickets incentivized to close the tickets as soon as possible?” It seems that your internal security posture would make that clear, and it is unclear how stating that correctness is more important than speed in user account modification poses a security risk. I’ll pause here and summarize. Answering any of these questions is not going to pose a security risk unless such answers expose to your users reasonable measures that you are not taking or haven’t thought of.

brongondwana 9y ago
Which employees? At the time of this compromise, that list was all support staff as well as the technical staff in Melbourne. It is a specific role that's granted to specific people, to answer your question about having a procedure or policy. Today, that role is granted to a much more limited set of senior security staff (currently 3 people). Regular support staff can not alter security-sensitive details about accounts. If your account is owned by someone else (e.g. family or business, or part of a resold package) then they can still alter recovery options, as they own the account. In 2016 before we had automated account recovery, lost password was in the top 3 categories of ticket every single week! Every member of the support team dealt with multiple account-loss tickets per day, both forgotten password or stolen account. Stolen account losses are way down now we have app passwords, we often only have to block a single app password and notify the user rather than locking the entire account. Forgotten passwords have not reduced, but most people are able to recover using the automated tooling. --- In what fashion do we audit and track? A few ways - we log every API call at the lowest level. We log each override when the support person accesses user accounts against the ticket that they come in through, so we can see why they were accessing that user. We could always do with better tooling to introspect logs, but the data is all captured and can be followed through after the fact. Support staff have no way to wipe their audit trail. --- Training - in 2016 we didn't have much formal training for our support staff - they learned on the job from each other. We are very aware that this was a failing at that time. We have more training now. We did a lot of work at unifying our support teams across the FastMail and Pobox/Listbox family throughout 2017, and that led to better training and induction materials, as well as better internal reference material for support staff to use. Early in the induction process for all new support staff is a description of how social engineering works and a warning that urgency is often used in social engineering attempts, so when in doubt, slow down and get a second opinion (which leads to complaints about slow support, but that's the tradeoff here.) --- Escalation process - as mentioned earlier, if you had 2fa enabled then it has always gone straight to the senior security team, which is based in Melbourne and consists of our most experienced and trusted people. Neil (author of the blog post this HN refers to) is of course one of these people. With lost passwords no longer a highly common support request, all support tickets requesting manual account recovery are escalated to our senior team for review. --- Support people have no incentive to close tickets quickly. Absolutely. That is a bad metric, and it's not a metric we have ever used. Time to first response and time to followup responses are tracked, but there's no incentive to close tickets. This answer is a no brainer and I should have answered it in the first response - sorry. I was still rushing through initial responses at that time, and there were too many points in that post to think about them all at once and still respond quickly. The real-time nature of this hacker-news medium encourages fast answers above complete answers. I hope this longer response helps clear up remaining questions, at least to those who see it! --- There's another blog post coming soon about the account recovery system in particular, which addresses exactly how we've minimising human involvement in recovery decisions while not excessively punishing real human frailty amongst our customers.

deleted 9y ago

[deleted]