8 ms·

We experienced our first successful attack at my startup a few weeks ago. What got us wasn't anything on the top ten list. I'm pretty sure it isn't covered any

We experienced our first successful attack at my startup a few weeks ago. What got us wasn't anything on the top ten list. I'm pretty sure it isn't covered anywhere in OWASP to my knowledge. Users reuse passwords across different websites. An attacker tried a database of usernames/passwords sourced from elsewhere; a small percentage (about 1000 out of more than 10M requests) succeeded. 100 of those had something to steal. Attacker used a botnet, so our IP-based fail-to-ban logic was ineffective. We thought about lots of ways to deal with this moving forward. My boss (CEO) didn't want to implement any kind of 2 factor authentication, because it's cumbersome and will lower conversion rates. We took a different strategy which is kind of complicated to explain, but it's not nearly as secure. Anyway. What gets me is like: Password authentication SUCKS. It's a terrible terrible authentication strategy. It's awful. It should not be relied upon. It would be good if humans didn't reuse passwords. But we do. So it sucks.

sixhobbits 9y ago
You can mitigate this almost completely by finding that "database of usernames/passwords sourced from elsewhere" (they're not hard to find) and blacklisting them. Users should not be allowed to use any breached password when they register. A simple message saying "this message was included in a recent password breach and is therefore not secure" should suffice to prevent users getting annoyed that they can't use their favourite password on your site. Enforcing a minimum length of 10 or even 12 is a great way to eliminate nearly all previously leaked passwords from being used on your site, and it further encourages users to use password managers. Passwords are shit, but they're here to stay for a while still.
- wongarsu 9y ago
  HaveIBeenPwned makes this really easy by publishing a list of hashed passwords that have been observed in breaches [1]. The list is by no means complete, but it should cover a lot and is very easy to setup. 1: https://haveibeenpwned.com/Passwords https://haveibeenpwned.com/Passwords
  - methodover 9y ago
    That... is a great idea. I'll do it. Thank you!
- methodover 9y ago
  About the minimum password length: I did change it to 10 the day of the attack. The CEO wants us to change it to 8. We've seen a small dip in conversions (like 1%), and the longer password requirement could be why, he thinks.
  - tirant 9y ago
    That should be easy to prove with some basic A/B testing.
alexk 9y ago
Have you considered more conservative rate-limiting external IPs on login endpoints? E.g. with nginx it will make sense to set a custom rate limiting zone to prevent many requests from the same IP specifically to the login page: https://www.nginx.com/blog/rate-limiting-nginx/ https://www.nginx.com/blog/rate-limiting-nginx/ This will not fix the root cause, but will make it considerably harder to scan for matching passwords. You can also set up Fail2Ban to block IPs that failed to authenticate many times: https://serverfault.com/questions/421046/how-to-limit-nginx-auth-basic-re-tries https://serverfault.com/questions/421046/how-to-limit-nginx-...
- methodover 9y ago
  We do have fail to ban type logic in place. The attacker used a botnet; requests came in from many different IP addresses.
  - greenyouse 9y ago
    Have you tried rate limiting based on the user account? That should block a distributed attack since each login would count against the rate limit independent of the IP address.
    - ScottBurson 9y ago
      How would that help in this situation? The attacker had a database of user/password pairs they were trying; they weren't trying to brute-force a particular account.
      - greenyouse 9y ago
        Oops, you're completely correct. I was thinking of the brute force scenario.
skywhopper 9y ago
Passwords suck, okay. What alternative is there?
- toast0 9y ago
  Oauth helps, because people are more likely to have a decent password already for Gmail/Facebook/whatever than to make a new good password on the spot to try your site. If you require an email address, you could send an email with a link (or code) to login and set a long loved cookie to keep users logged in. Again, it's likely that users will have a good password on their email. Both of these options have negatives in that they tie your user to an external identity provider.
  - methodover 9y ago
    There could be a great opportunity for Google or Facebook to take the lead in managing user logins. The rest of the world could just piggy-back on one of those two. I'd love to never have to deal with password bullshit ever again as an app creator: I'll just make my users log in with Google/FB/etc.
    - majewsky 9y ago
      And lose all those users who deleted their Facebook account, and don't want their Google account to be associated with more data than necessary.
  - abritinthebay 9y ago
    > people are more likely to have a decent password already for Gmail/Facebook/whatever Oh man... do I have news for you.
iraklism 9y ago
We deal with this almost every week, as in, we get into systems by searching through email:password leaks and use them. There are a number of mitigating controls that can be applied here. Most will hamper usability, some will not. There is a “simple” solution. Enforce 2FA. If not at the login, then before “dangerous” actions (transfer funds , change password , buy X/Y/Z )
- methodover 9y ago
  That was one of the ideas that we pitched to the CEO. Only sensitive actions would require 2FA. CEO shot it down, saying it would require too much work on the part of the customer.
  - busterarm 9y ago
    Users will go through the work if you incentivize the behavior.
  - orf 9y ago
    Why not give customers the option to choose to enable it then?
    - marcins 9y ago
      The kind of customer that reuses passwords is probably the kind that won't enable 2FA if it's optional.
      - orf 9y ago
        Sure, but then it's on them rather than you.
- lphartley 9y ago
  That's a simple solution from a security perspective. From a business perspective the most simple solution is guaranteeing that you'll cover all the damage customers might possibly suffer.
busterarm 9y ago
Make 2FA mandatory for users who were breached or are using passwords that are in known password lists. I don't know how much you spent in support, but U2F Zeros are dirt cheap. You could probably just proactively mail them to your clients and encourage them to use 2FA. Or offer discounts or other perks to users with 2FA.
- methodover 9y ago
  It's interesting that you went for TRUE 2FA. I was thinking about going for email/text-based 2FA. I know it's not true second factor authentication, but it seems so much more accessible than requiring a separate device. Our customers are not at all tech savvy, generally speaking; there's no way they'd go for a dongle.
k4ch0w 9y ago
Hey man, Did you add a captcha to your login? You could add one after 2 failed attempts from an ip. I understand not wanting 2FA for lower conversion rates, but I'd recommend implementing it in the future. It's one of the best ways to mitigate an attack against knowing your password. The session cookie is another story. As for the passwords, I'd enforce standards around what a user is allowed to use. You want to have it be at least 10+ chars with special characters. If they reuse the password on another site and that gets compromised there really isn't much you can do. However, if we assume the reused password from the other site is hashed with a salt and you enforce more chars, they are much harder to crack and thus prevent the plaintext password from being recovered and accessing your site. A great way to demo what more chars does to a password here https://www.betterbuys.com/estimating-password-cracking-times/ https://www.betterbuys.com/estimating-password-cracking-time... You should also maintain a record of the common IP address an account uses and require an email/text to allow access that would further mitigate your problem. I know your a startup, so this is more of a luxury feature.
- methodover 9y ago
  I like the idea of asking for a captcha after 2 failed logins. We had set up a pretty restrictive Cloudflare rule which blocked access to the site for some amount of time (60 minutes?) from IP addresses that got more than 5 (IIRC) failed logins within a short, sliding window of time (2 minutes?). But the botnet had a ton of machines on it. The rule didn't block nearly as many requests as I hoped it would. A captcha, though, could be set to be pretty restrictive like you suggest. Passwords: I actually did up the password requirements (to require 10 characters), but the CEO wants me to pull back to just 8 characters. And no special characters. Our conversion rate went down a tiny bit, could be the restrictive password requirements that did it. Keeping a record of common IP addresses and enforcing something like 2FA for strange logins: Love that idea. Would love to implement it. Can't, though. The CEO won't go for it, says 2FA lowers conversion and he's probably right about that.
  - nl 9y ago
    You can use the have I been pawned API to check for login which are dangerous, and captcha those.
e12e 9y ago
> What gets me is like: Password authentication SUCKS. There's a conference for that. Has been since 2010. Not quite sure about 2017/2018 > My boss (CEO) didn't want to implement any kind of 2 factor authentication, because it's cumbersome and will lower conversion rates True. "Security is not a convenience". But there's no easier way to get people authenticating off of something at least a bit secure than straight totp 2fa.
angry_octet 9y ago
It is quite feasible to try to brute force you own users passwords. Run a background job that tests against common passwords and close variants, and if it is too simple, either force a harder one, or force email auth if any of their metadata changes (IP, ASN, browser fingerprint). C.f. https://haveibeenpwned.com/Passwords https://haveibeenpwned.com/Passwords
kogepathic 9y ago
> Anyway. What gets me is like: Password authentication SUCKS. It's a terrible terrible authentication strategy. It's awful. It should not be relied upon. It would be good if humans didn't reuse passwords. But we do. So it sucks. There's an easy solution for this! Have I been Pwned offers a free API [0] you can use to check if a password has been in a previous breach. If the password was previously disclosed in a breach, simply inform the user politely "Sorry, this password has previously been previously disclosed and will likely be used by attackers to compromise your account" and make them pick a new one. No need for 2FA. Simply add a check to prevent users from re-using passwords. (If you don't want to start sending user passwords to HIBP, you can also download the list and use it internally) [0] https://haveibeenpwned.com/API/v2#PwnedPasswords https://haveibeenpwned.com/API/v2#PwnedPasswords
- dyu 9y ago
  This looks interesting, and I wasn't aware of this API before. The downside is you have to send a plaintext or at best SHA1 to that server, which creates another problem even if the server is not logging and can be trusted.
  - methodover 9y ago
    It seems like an incredibly bad idea to use their API. It seems like one should just use the downloadable database. And besides, it'll be faster to just use the database rather than making an API call anyway.
raesene9 9y ago
What you're describing gets covered by OWASP as "credential stuffing" You can find it as A2 in the current (RC2) draft OWASP Top 10 and on the wiki here https://www.owasp.org/index.php/Credential_stuffing https://www.owasp.org/index.php/Credential_stuffing
- methodover 9y ago
  Wow! Thanks. I should read the site more carefully. Thanks for the correction. Btw I feel like this should be in the top 10 list right? With mitigation strategies?
gspetr 9y ago
> What got us wasn't anything on the top ten list. From my PoV nothing got to you. Yes, you may have got some bad publicity, but the fault lies with users who have poor (non-existent?) OPSEC. Just as there are 2 types of people: those who don't have backups and those who will make backups, there are those who don't have password compartmentalization and those who will have it.
- ryanlol 9y ago
  Do you really think that their card processor would give a shit about whose fault all the fraudulent charges were?

sixhobbits 9y ago

You can mitigate this almost completely by finding that "database of usernames/passwords sourced from elsewhere" (they're not hard to find) and blacklisting them. Users should not be allowed to use any breached password when they register. A simple message saying "this message was included in a recent password breach and is therefore not secure" should suffice to prevent users getting annoyed that they can't use their favourite password on your site. Enforcing a minimum length of 10 or even 12 is a great way to eliminate nearly all previously leaked passwords from being used on your site, and it further encourages users to use password managers. Passwords are shit, but they're here to stay for a while still.

wongarsu 9y ago
HaveIBeenPwned makes this really easy by publishing a list of hashed passwords that have been observed in breaches [1]. The list is by no means complete, but it should cover a lot and is very easy to setup. 1: https://haveibeenpwned.com/Passwords https://haveibeenpwned.com/Passwords
- methodover 9y ago
  That... is a great idea. I'll do it. Thank you!
methodover 9y ago
About the minimum password length: I did change it to 10 the day of the attack. The CEO wants us to change it to 8. We've seen a small dip in conversions (like 1%), and the longer password requirement could be why, he thinks.
- tirant 9y ago
  That should be easy to prove with some basic A/B testing.

alexk 9y ago

Have you considered more conservative rate-limiting external IPs on login endpoints? E.g. with nginx it will make sense to set a custom rate limiting zone to prevent many requests from the same IP specifically to the login page: https://www.nginx.com/blog/rate-limiting-nginx/ https://www.nginx.com/blog/rate-limiting-nginx/ This will not fix the root cause, but will make it considerably harder to scan for matching passwords. You can also set up Fail2Ban to block IPs that failed to authenticate many times: https://serverfault.com/questions/421046/how-to-limit-nginx-auth-basic-re-tries https://serverfault.com/questions/421046/how-to-limit-nginx-...

methodover 9y ago
We do have fail to ban type logic in place. The attacker used a botnet; requests came in from many different IP addresses.
- greenyouse 9y ago
  Have you tried rate limiting based on the user account? That should block a distributed attack since each login would count against the rate limit independent of the IP address.
  - ScottBurson 9y ago
    How would that help in this situation? The attacker had a database of user/password pairs they were trying; they weren't trying to brute-force a particular account.
    - greenyouse 9y ago
      Oops, you're completely correct. I was thinking of the brute force scenario.

skywhopper 9y ago

Passwords suck, okay. What alternative is there?

toast0 9y ago
Oauth helps, because people are more likely to have a decent password already for Gmail/Facebook/whatever than to make a new good password on the spot to try your site. If you require an email address, you could send an email with a link (or code) to login and set a long loved cookie to keep users logged in. Again, it's likely that users will have a good password on their email. Both of these options have negatives in that they tie your user to an external identity provider.
- methodover 9y ago
  There could be a great opportunity for Google or Facebook to take the lead in managing user logins. The rest of the world could just piggy-back on one of those two. I'd love to never have to deal with password bullshit ever again as an app creator: I'll just make my users log in with Google/FB/etc.
  - majewsky 9y ago
    And lose all those users who deleted their Facebook account, and don't want their Google account to be associated with more data than necessary.
- abritinthebay 9y ago
  > people are more likely to have a decent password already for Gmail/Facebook/whatever Oh man... do I have news for you.

iraklism 9y ago

We deal with this almost every week, as in, we get into systems by searching through email:password leaks and use them. There are a number of mitigating controls that can be applied here. Most will hamper usability, some will not. There is a “simple” solution. Enforce 2FA. If not at the login, then before “dangerous” actions (transfer funds , change password , buy X/Y/Z )

methodover 9y ago
That was one of the ideas that we pitched to the CEO. Only sensitive actions would require 2FA. CEO shot it down, saying it would require too much work on the part of the customer.
- busterarm 9y ago
  Users will go through the work if you incentivize the behavior.
- orf 9y ago
  Why not give customers the option to choose to enable it then?
  - marcins 9y ago
    The kind of customer that reuses passwords is probably the kind that won't enable 2FA if it's optional.
    - orf 9y ago
      Sure, but then it's on them rather than you.
lphartley 9y ago
That's a simple solution from a security perspective. From a business perspective the most simple solution is guaranteeing that you'll cover all the damage customers might possibly suffer.

busterarm 9y ago

Make 2FA mandatory for users who were breached or are using passwords that are in known password lists. I don't know how much you spent in support, but U2F Zeros are dirt cheap. You could probably just proactively mail them to your clients and encourage them to use 2FA. Or offer discounts or other perks to users with 2FA.

methodover 9y ago
It's interesting that you went for TRUE 2FA. I was thinking about going for email/text-based 2FA. I know it's not true second factor authentication, but it seems so much more accessible than requiring a separate device. Our customers are not at all tech savvy, generally speaking; there's no way they'd go for a dongle.

k4ch0w 9y ago

Hey man, Did you add a captcha to your login? You could add one after 2 failed attempts from an ip. I understand not wanting 2FA for lower conversion rates, but I'd recommend implementing it in the future. It's one of the best ways to mitigate an attack against knowing your password. The session cookie is another story. As for the passwords, I'd enforce standards around what a user is allowed to use. You want to have it be at least 10+ chars with special characters. If they reuse the password on another site and that gets compromised there really isn't much you can do. However, if we assume the reused password from the other site is hashed with a salt and you enforce more chars, they are much harder to crack and thus prevent the plaintext password from being recovered and accessing your site. A great way to demo what more chars does to a password here https://www.betterbuys.com/estimating-password-cracking-times/ https://www.betterbuys.com/estimating-password-cracking-time... You should also maintain a record of the common IP address an account uses and require an email/text to allow access that would further mitigate your problem. I know your a startup, so this is more of a luxury feature.

methodover 9y ago
I like the idea of asking for a captcha after 2 failed logins. We had set up a pretty restrictive Cloudflare rule which blocked access to the site for some amount of time (60 minutes?) from IP addresses that got more than 5 (IIRC) failed logins within a short, sliding window of time (2 minutes?). But the botnet had a ton of machines on it. The rule didn't block nearly as many requests as I hoped it would. A captcha, though, could be set to be pretty restrictive like you suggest. Passwords: I actually did up the password requirements (to require 10 characters), but the CEO wants me to pull back to just 8 characters. And no special characters. Our conversion rate went down a tiny bit, could be the restrictive password requirements that did it. Keeping a record of common IP addresses and enforcing something like 2FA for strange logins: Love that idea. Would love to implement it. Can't, though. The CEO won't go for it, says 2FA lowers conversion and he's probably right about that.
- nl 9y ago
  You can use the have I been pawned API to check for login which are dangerous, and captcha those.

e12e 9y ago

> What gets me is like: Password authentication SUCKS. There's a conference for that. Has been since 2010. Not quite sure about 2017/2018 > My boss (CEO) didn't want to implement any kind of 2 factor authentication, because it's cumbersome and will lower conversion rates True. "Security is not a convenience". But there's no easier way to get people authenticating off of something at least a bit secure than straight totp 2fa.

angry_octet 9y ago

It is quite feasible to try to brute force you own users passwords. Run a background job that tests against common passwords and close variants, and if it is too simple, either force a harder one, or force email auth if any of their metadata changes (IP, ASN, browser fingerprint). C.f. https://haveibeenpwned.com/Passwords https://haveibeenpwned.com/Passwords

kogepathic 9y ago

> Anyway. What gets me is like: Password authentication SUCKS. It's a terrible terrible authentication strategy. It's awful. It should not be relied upon. It would be good if humans didn't reuse passwords. But we do. So it sucks. There's an easy solution for this! Have I been Pwned offers a free API [0] you can use to check if a password has been in a previous breach. If the password was previously disclosed in a breach, simply inform the user politely "Sorry, this password has previously been previously disclosed and will likely be used by attackers to compromise your account" and make them pick a new one. No need for 2FA. Simply add a check to prevent users from re-using passwords. (If you don't want to start sending user passwords to HIBP, you can also download the list and use it internally) [0] https://haveibeenpwned.com/API/v2#PwnedPasswords https://haveibeenpwned.com/API/v2#PwnedPasswords

dyu 9y ago
This looks interesting, and I wasn't aware of this API before. The downside is you have to send a plaintext or at best SHA1 to that server, which creates another problem even if the server is not logging and can be trusted.
- methodover 9y ago
  It seems like an incredibly bad idea to use their API. It seems like one should just use the downloadable database. And besides, it'll be faster to just use the database rather than making an API call anyway.

raesene9 9y ago

What you're describing gets covered by OWASP as "credential stuffing" You can find it as A2 in the current (RC2) draft OWASP Top 10 and on the wiki here https://www.owasp.org/index.php/Credential_stuffing https://www.owasp.org/index.php/Credential_stuffing

methodover 9y ago
Wow! Thanks. I should read the site more carefully. Thanks for the correction. Btw I feel like this should be in the top 10 list right? With mitigation strategies?

gspetr 9y ago

> What got us wasn't anything on the top ten list. From my PoV nothing got to you. Yes, you may have got some bad publicity, but the fault lies with users who have poor (non-existent?) OPSEC. Just as there are 2 types of people: those who don't have backups and those who will make backups, there are those who don't have password compartmentalization and those who will have it.

ryanlol 9y ago
Do you really think that their card processor would give a shit about whose fault all the fraudulent charges were?