5 ms·

Now to write a more detailed response. If this winds up out of order later, I first posted: https://news.ycombinator.com/item?id=15856609 https://news.ycombina

Now to write a more detailed response. If this winds up out of order later, I first posted: https://news.ycombinator.com/item?id=15856609 https://news.ycombinator.com/item?id=15856609 Again, ghouse, I'm really really sorry about what happened to your account. It was wrong and we screwed up. As other comments have already noted, it was during the transition to a new security system which was designed precisely to remove the human factor from decision making. I'm an Australian, and I'm a great fan of our "100 points of ID" system, which is designed to remove the human factor from identifying people. https://en.wikipedia.org/wiki/100_point_check https://en.wikipedia.org/wiki/100_point_check While I wasn't aware of your account's issue at the time (and we'll be having some discussions internally about why not!), FastMail management were all aware that we needed to get the human factor out of decision making about account access, particularly since somebody tried to pull a similar swindle on our domains! https://blog.fastmail.com/2014/04/10/when-two-factor-authentication-is-not-enough/ https://blog.fastmail.com/2014/04/10/when-two-factor-authent... We spent a lot of 2016 and 2017 working on an automated account recovery system which allows recovery of locked accounts via a carefully audited set of automated steps, which includes a 24 hour lockout to allow the owner to notice an attempt on their account. If this had existed in 2016 then we would have sent you there rather than having a human make a (poor in this case) judgement call! https://www.fastmail.com/help/account/icantlogin.html https://www.fastmail.com/help/account/icantlogin.html

tptacek 9y ago
I don't understand this response. I'm glad you're working to minimize human factors. Can you explain how exactly you're doing that? I asked some specific questions here: https://news.ycombinator.com/item?id=15856755 https://news.ycombinator.com/item?id=15856755
ghouse 9y ago
Thank you. I appreciate your response and understand that you've made improvements since this incident in 2016.
xmodem 9y ago
> a 24 hour lockout to allow the owner to notice an attempt on their account. I've been pretty careful to ensure that I don't lock myself out of my account (multiple U2F keys, strong password saved in password manager with backups) But if a determined attacker kicks this off just as I'm stepping on a flight from Sydney to London, 24 hours isn't going to be enough. (I should add also - I'm a mostly happy Fastmail customer)
- brongondwana 9y ago
  You can't even get to the 24 hour lockout unless you've successfully passed the security checks. We add the 24 hour lockout as an additional level of protection for 2fa accounts (even though they've given two factors of recovery by then) or if we can't confirm that you are resetting from a computer which has successfully logged in to that account before.
  - tptacek 9y ago
    It sounds like if I use Fastmail, and I go on vacation (and thus go a day without checking my email), someone can max out the automated system and then get a human being at Fastmail to potentially reset my recovery email. Is this the case?
    - brongondwana 9y ago
      You can't max out the automated system. Our procedures have to balance the concerns of very different groups of people. Some people have explicitly directed us to enforce stringent account security requirements by enabling multi factor authentication. For those people, we assume that they have their own security practices and are diligent in maintaining them. Those people are aware of the risk of losing access to their mail if they lose their credentials. The other, much larger group of our customers, come to use because they want email that has support. Many of these customers forget their passwords and still need to get to their email (which is more common than you might imagine if you are surrounded by a hacker-news demographic!) Our procedures have to balance between those two sets of needs, and they evolve over time. This incident came up in a period of transition. It should never had happened, and it's a great object lesson to us about how to do better in future transitions. Having said that, based on this conversation today we are reviewing all our processes around re-establishing access for regular people who haven't requested additional security by enabling second factors. We absolutely can and will do better than we did in 2016.
dannysu 9y ago
What I'm hearing is that the human aspect remains and there is absolutely no prevention of this happening in the future. The other response contains weasel words like "For instance, _some cases_ take 24 hours before the reset password goes into effect". Why "some cases"? Why isn't it all cases? I think we as customers deserve complete transparency on this and know what prevention will be in place.
- brongondwana 9y ago
  I've had conflicting advice about complete transparency - if we give the entire algorithm, then that helps attackers find the exact surface that will get them in, so we don't publish the full ruleset we use. Here's an example of some inputs that go into it though: we store a cryptographic token in a cookie which tells us the first time your account successfully authenticated from a computer. If we have a history of you using the same computer over multiple years, that's different than a new computer. But cookies can be cloned, so it's only a signal, not proof in itself. If it's from the same IP address as multiple successful logins in the past, that's a signal. We're not the only site that uses methods like this to help identify people when they've lost their password. People make mistakes. Taking a hard line "you lose your password, you lose your entire email account with all its history, and you don't get your money back either" might sound attractive to a certain demographic. They are not the bulk of our userbase. Even locking people out for 24 hours is a pretty big imposition that you want to avoid if you're really confident (algorithmically) that it's the same person. If people in the "security is more important than easy recovery" demographic haven't turned on 2FA yet, then they certainly haven't signaled that they want things locked down in case of doubt. Even of those who HAVE turned on 2FA, you'd be surprised at how many lose one or both of their factors. It's easy to say "I won't mess up", but people do. Which is why our post today says in bold "If that happens, you will lose access to your account permanently."
caconym_ 9y ago
I use Fastmail and I like it but this is extremely disturbing. I appreciate you being relatively candid with us, but it doesn’t change the fact that you allowed a customer’s account to be compromised by the most basic attack out there. Complexity and vulnerability go hand-in-hand. A product providing a critical service like email should be opt-in to any form of recovery not requiring pure secrets provided directly by the user (or provided to an already logged-in user for this specific purpose). Failing that, there should at least be an opt-out for such dangerous recovery methods. I’m going to watch this thread and your blog for a while, and I hope you can provide some real assurances for security-minded technical folks like me for whom email really is the “keys to the kingdom”. Failing that, I may have to look for another email provider.
- brongondwana 9y ago
  https://blog.fastmail.com/2017/12/06/security-account-recovery/ https://blog.fastmail.com/2017/12/06/security-account-recove... is probably what you're looking for. I've also made additional responses in this thread.
rahimnathwani 9y ago
'I'm an Australian, and I'm a great fan of our "100 points of ID" system, which is designed to remove the human factor from identifying people.' According the page you linked, a birth certificate and bank statement (or even a 'Document issued by <SNIP> or registered corporations.') would be enough. So if I get your birth certificate and have an Australian corporation, I can issue a letter saying you're a customer for a year. So I have 100 points and can pretend to be you? That doesn't seem secure at all. The birth certificate has no photo (or, if it does, it won't be useful except to determine ethnicity), and the document from a registered corporation can be trivially faked.
- brongondwana 9y ago
  We don't use the 100 points of ID system of course, because we're an online service. The 100 points of ID is something that's used in person to decide whether you can open a new bank account using that name. The concept behind the 100 points of ID is that there's a fixed standard and it's not a per-time decision made by a human, it's a consistent set of rules applied without fear or favour.
  - dannysu 9y ago
    It's a consistent, fixed standard that you hope you have trained your employees to adhere. Fingers crossed. /s
  - rahimnathwani 9y ago
    "We don't use the 100 points of ID system of course, because we're an online service. Right, but you said you're a 'great fan' of it. My reading of the wikipedia description that you linked suggests that the system is wholly inadequate (mainly as you can satisfy the 100 points without photo ID). So I'd like to know: are you really a fan of that system (the particulars of its rules), or just a fan of the idea behind it (that there exists a consistent set of rules)?
    - brongondwana 9y ago
      Tell me how you bootstrap photo ID in your country, and I'll tell you whether I think photo ID means anything. In our case, we don't care at all what you look like, just that you're the same person we were talking to earlier - and ideally that you're the owner of the method being used to pay, though that's not always true or necessary. So the photo is meaningless to us. Besides: who is looking at the photo and confirming that it's the same person as the one in front of them? Yep, an human. The whole point of this discussion is stopping the human making human-factor judgement calls.
dvcrn 9y ago
Please consider adding a option to never ever allow recovering of the account without password, similar to how gandi does it. My email account / domain is my central hub for all my accounts. All of them can be taken over through fastmail (with the exception of my domain and other extremely crucial services) if an attacker happens to obtain access to it. I want to have the security that this attack can not happen to me.
- brongondwana 9y ago
  I'm assuming you have 2FA turned on already. It sounds from what you're saying like you (and at least a few other hacker news posters) want is an even stricter "no seriously, I promise I won't ever screw up" mode. We try not to have those kinds of modes, because (for example): https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html It turns out, black and white security models lead to massive losses of availability when people screw them up - and people do. Though I have to confess to being amazed to see Tony Finch amongst the recent "oops". NASA is maybe not so much of a surprise. Having said that - if there's enough demand, that would be a worthwhile feature. Accounts that aren't being used are cheap for us to run, and that flag would make the security team's job really easy - just say "no, go find your own way in" without having to review anything!
  - warp 9y ago
    As a paying fastmail customer, I would appreciate such an option. However, you do have to make sure that if I lose access to that account, I should be able to create a new FastMail account and have new traffic to my domain be directed to the new account. i.e. you do need some way to migrate a custom domain to a new account, if the new user can prove ownership and/or control of the domain name.
    - brongondwana 9y ago
      Yes, that has to be an option anyway, if somebody sells a domain and doesn't release it from FastMail. I have a blog post for this advent series (already written and everything) about why we don't allow split billing on a single domain.
dannysu 9y ago
The blog post was in 2014. This security bypass happened in 2016. I think what we're witnessing here is that despite best intentions and past experience, humans are going to be humans. I actually felt good after reading that blog post in 2014 thinking that you guys are going to be better than most companies here. Nope. But I think a lesson can be learned here. The lesson is simply that humans are the weakest link. As much as you might try to add process and try to minimize, the best is having zero human capability at all. So when tptacek asks _who_ has ability to change things about an account, we really do want to know. Because those people are the weakest links. (don't mean naming names, but understanding who in general has those powers) I mentioned elsewhere. I own my domain. I backup my emails. It's way more likely for a FastMail human loophole to screw me over than for me to need human assistance on login (which is never).
- brongondwana 9y ago
  https://blog.fastmail.com/2017/12/06/security-account-recovery/ https://blog.fastmail.com/2017/12/06/security-account-recove... - it's currently 3 people who have that ability. It had to be more before we had the automated tooling, those three people couldn't handle 3 figures per day (I'm not kidding) of regular password losses by regular users.
whamlastxmas 9y ago
It's not really clear to me. As a paying fastmail customer, can someone still contact your support and get the recovery email address changed?

tptacek 9y ago

I don't understand this response. I'm glad you're working to minimize human factors. Can you explain how exactly you're doing that? I asked some specific questions here: https://news.ycombinator.com/item?id=15856755 https://news.ycombinator.com/item?id=15856755

ghouse 9y ago

Thank you. I appreciate your response and understand that you've made improvements since this incident in 2016.

xmodem 9y ago

> a 24 hour lockout to allow the owner to notice an attempt on their account. I've been pretty careful to ensure that I don't lock myself out of my account (multiple U2F keys, strong password saved in password manager with backups) But if a determined attacker kicks this off just as I'm stepping on a flight from Sydney to London, 24 hours isn't going to be enough. (I should add also - I'm a mostly happy Fastmail customer)

brongondwana 9y ago
You can't even get to the 24 hour lockout unless you've successfully passed the security checks. We add the 24 hour lockout as an additional level of protection for 2fa accounts (even though they've given two factors of recovery by then) or if we can't confirm that you are resetting from a computer which has successfully logged in to that account before.
- tptacek 9y ago
  It sounds like if I use Fastmail, and I go on vacation (and thus go a day without checking my email), someone can max out the automated system and then get a human being at Fastmail to potentially reset my recovery email. Is this the case?
  - brongondwana 9y ago
    You can't max out the automated system. Our procedures have to balance the concerns of very different groups of people. Some people have explicitly directed us to enforce stringent account security requirements by enabling multi factor authentication. For those people, we assume that they have their own security practices and are diligent in maintaining them. Those people are aware of the risk of losing access to their mail if they lose their credentials. The other, much larger group of our customers, come to use because they want email that has support. Many of these customers forget their passwords and still need to get to their email (which is more common than you might imagine if you are surrounded by a hacker-news demographic!) Our procedures have to balance between those two sets of needs, and they evolve over time. This incident came up in a period of transition. It should never had happened, and it's a great object lesson to us about how to do better in future transitions. Having said that, based on this conversation today we are reviewing all our processes around re-establishing access for regular people who haven't requested additional security by enabling second factors. We absolutely can and will do better than we did in 2016.

dannysu 9y ago

What I'm hearing is that the human aspect remains and there is absolutely no prevention of this happening in the future. The other response contains weasel words like "For instance, _some cases_ take 24 hours before the reset password goes into effect". Why "some cases"? Why isn't it all cases? I think we as customers deserve complete transparency on this and know what prevention will be in place.

brongondwana 9y ago
I've had conflicting advice about complete transparency - if we give the entire algorithm, then that helps attackers find the exact surface that will get them in, so we don't publish the full ruleset we use. Here's an example of some inputs that go into it though: we store a cryptographic token in a cookie which tells us the first time your account successfully authenticated from a computer. If we have a history of you using the same computer over multiple years, that's different than a new computer. But cookies can be cloned, so it's only a signal, not proof in itself. If it's from the same IP address as multiple successful logins in the past, that's a signal. We're not the only site that uses methods like this to help identify people when they've lost their password. People make mistakes. Taking a hard line "you lose your password, you lose your entire email account with all its history, and you don't get your money back either" might sound attractive to a certain demographic. They are not the bulk of our userbase. Even locking people out for 24 hours is a pretty big imposition that you want to avoid if you're really confident (algorithmically) that it's the same person. If people in the "security is more important than easy recovery" demographic haven't turned on 2FA yet, then they certainly haven't signaled that they want things locked down in case of doubt. Even of those who HAVE turned on 2FA, you'd be surprised at how many lose one or both of their factors. It's easy to say "I won't mess up", but people do. Which is why our post today says in bold "If that happens, you will lose access to your account permanently."

caconym_ 9y ago

I use Fastmail and I like it but this is extremely disturbing. I appreciate you being relatively candid with us, but it doesn’t change the fact that you allowed a customer’s account to be compromised by the most basic attack out there. Complexity and vulnerability go hand-in-hand. A product providing a critical service like email should be opt-in to any form of recovery not requiring pure secrets provided directly by the user (or provided to an already logged-in user for this specific purpose). Failing that, there should at least be an opt-out for such dangerous recovery methods. I’m going to watch this thread and your blog for a while, and I hope you can provide some real assurances for security-minded technical folks like me for whom email really is the “keys to the kingdom”. Failing that, I may have to look for another email provider.

brongondwana 9y ago
https://blog.fastmail.com/2017/12/06/security-account-recovery/ https://blog.fastmail.com/2017/12/06/security-account-recove... is probably what you're looking for. I've also made additional responses in this thread.

rahimnathwani 9y ago

'I'm an Australian, and I'm a great fan of our "100 points of ID" system, which is designed to remove the human factor from identifying people.' According the page you linked, a birth certificate and bank statement (or even a 'Document issued by <SNIP> or registered corporations.') would be enough. So if I get your birth certificate and have an Australian corporation, I can issue a letter saying you're a customer for a year. So I have 100 points and can pretend to be you? That doesn't seem secure at all. The birth certificate has no photo (or, if it does, it won't be useful except to determine ethnicity), and the document from a registered corporation can be trivially faked.

brongondwana 9y ago
We don't use the 100 points of ID system of course, because we're an online service. The 100 points of ID is something that's used in person to decide whether you can open a new bank account using that name. The concept behind the 100 points of ID is that there's a fixed standard and it's not a per-time decision made by a human, it's a consistent set of rules applied without fear or favour.
- dannysu 9y ago
  It's a consistent, fixed standard that you hope you have trained your employees to adhere. Fingers crossed. /s
- rahimnathwani 9y ago
  "We don't use the 100 points of ID system of course, because we're an online service. Right, but you said you're a 'great fan' of it. My reading of the wikipedia description that you linked suggests that the system is wholly inadequate (mainly as you can satisfy the 100 points without photo ID). So I'd like to know: are you really a fan of that system (the particulars of its rules), or just a fan of the idea behind it (that there exists a consistent set of rules)?
  - brongondwana 9y ago
    Tell me how you bootstrap photo ID in your country, and I'll tell you whether I think photo ID means anything. In our case, we don't care at all what you look like, just that you're the same person we were talking to earlier - and ideally that you're the owner of the method being used to pay, though that's not always true or necessary. So the photo is meaningless to us. Besides: who is looking at the photo and confirming that it's the same person as the one in front of them? Yep, an human. The whole point of this discussion is stopping the human making human-factor judgement calls.

dvcrn 9y ago

Please consider adding a option to never ever allow recovering of the account without password, similar to how gandi does it. My email account / domain is my central hub for all my accounts. All of them can be taken over through fastmail (with the exception of my domain and other extremely crucial services) if an attacker happens to obtain access to it. I want to have the security that this attack can not happen to me.

brongondwana 9y ago
I'm assuming you have 2FA turned on already. It sounds from what you're saying like you (and at least a few other hacker news posters) want is an even stricter "no seriously, I promise I won't ever screw up" mode. We try not to have those kinds of modes, because (for example): https://ianix.com/pub/dnssec-outages.html https://ianix.com/pub/dnssec-outages.html It turns out, black and white security models lead to massive losses of availability when people screw them up - and people do. Though I have to confess to being amazed to see Tony Finch amongst the recent "oops". NASA is maybe not so much of a surprise. Having said that - if there's enough demand, that would be a worthwhile feature. Accounts that aren't being used are cheap for us to run, and that flag would make the security team's job really easy - just say "no, go find your own way in" without having to review anything!
- warp 9y ago
  As a paying fastmail customer, I would appreciate such an option. However, you do have to make sure that if I lose access to that account, I should be able to create a new FastMail account and have new traffic to my domain be directed to the new account. i.e. you do need some way to migrate a custom domain to a new account, if the new user can prove ownership and/or control of the domain name.
  - brongondwana 9y ago
    Yes, that has to be an option anyway, if somebody sells a domain and doesn't release it from FastMail. I have a blog post for this advent series (already written and everything) about why we don't allow split billing on a single domain.

dannysu 9y ago

The blog post was in 2014. This security bypass happened in 2016. I think what we're witnessing here is that despite best intentions and past experience, humans are going to be humans. I actually felt good after reading that blog post in 2014 thinking that you guys are going to be better than most companies here. Nope. But I think a lesson can be learned here. The lesson is simply that humans are the weakest link. As much as you might try to add process and try to minimize, the best is having zero human capability at all. So when tptacek asks _who_ has ability to change things about an account, we really do want to know. Because those people are the weakest links. (don't mean naming names, but understanding who in general has those powers) I mentioned elsewhere. I own my domain. I backup my emails. It's way more likely for a FastMail human loophole to screw me over than for me to need human assistance on login (which is never).

brongondwana 9y ago
https://blog.fastmail.com/2017/12/06/security-account-recovery/ https://blog.fastmail.com/2017/12/06/security-account-recove... - it's currently 3 people who have that ability. It had to be more before we had the automated tooling, those three people couldn't handle 3 figures per day (I'm not kidding) of regular password losses by regular users.

whamlastxmas 9y ago

It's not really clear to me. As a paying fastmail customer, can someone still contact your support and get the recovery email address changed?