4 ms·

I have said it before: I have a hard time trusting Linode after the major March 2012 security incident (HN thread: https://news.ycombinator.com/item?id=3654110

by mrb 14y ago

I have said it before: I have a hard time trusting Linode after the major March 2012 security incident (HN thread: https://news.ycombinator.com/item?id=3654110 https://news.ycombinator.com/item?id=3654110 ; my comment: https://news.ycombinator.com/item?id=5339799 https://news.ycombinator.com/item?id=5339799) At the time they did not give much information. They did not do a follow up. They did not discuss plans to prevent the same type of breach. I am not surprised that today, they got breached again! sigh And again, they are making the same mistakes. They are not giving much information. They are not going to do a follow up. Etc.

flebron 14y ago
Why do you think in both cases this was the result of some sort of mistake on Linode's part, versus an oversight on the part of the target? That is, you use the expression 'they {linode} got breached', versus the more correct 'they {the customers} got breached'. How do you know that in both cases this was the case?
- raquo 14y ago
  If it was one customer's fault, why reset all Linode users' passwords?
  - flebron 14y ago
    If you knew there was a cracker actively using your network, would you not take any other precautions other than changing the one password you know he used? One could argue Linode was overzealous in resetting all passwords. Reasonable minds can differ.
    - JshWright 14y ago
      That depends completely on the method of compromise... If the compromise was due to a weak password that was brute forced, or obtained from the end user in some other way, then there is absolutely no need to reset every user's password. The only time a 'reasonable mind' would think to reset everyone's password is if the attacker exploited some flaw in the underlying system, which could have given them access to arbitrary passwords.
      - StavrosK 14y ago
        Yeah, it does sound like there's something to the story that they aren't disclosing, and it doesn't inspire confidence.
    - mnordhoff 14y ago
      Malicious traffic doesn't make routers burst into flame. Customers get compromised on a daily basis at any large provider due to bad SSH passwords or WordPress vulnerabilities or whatever. Assuming the virtualization system is trustworthy (which it is), that's not a problem. Assuming it's not trustworthy, they shouldn't be running a virtual hosting company and letting random strangers sign up in the first place!
    - ceejayoz 14y ago
      Any decently sized web host is almost certain to have crackers actively using their network at any time. AWS probably has thousands of instances running old WP installs that've been compromised.
- seany 14y ago
  The lack of transparency from them makes it impossible to know one way or the other. The xen security issue is another example of this kind of opaqueness.
- asdfaoeu 14y ago
  The wording they use makes it sound like the attackers compromised some Linode service to gain access to a specific customers machine rather than that customer themselves being exploited. If it was just a compromise of that customer there would obviously be no need for this announcement or resetting of passwords. At the very least they don't know which it was.
- mrb 14y ago
  In March 2012, Linode acknowledged it was a breach of their internal systems, not the customers'. Today, they are reseting the password of all customers, making it pretty certain this is another breach at the Linode level, not at the customer's level.
akkartik 14y ago
It just occurred to me that https://www.schneier.com/blog/archives/2012/04/password_securi.html https://www.schneier.com/blog/archives/2012/04/password_secu... was in response to last year's breach.
- jameswyse 14y ago
  One of the comments[1] on that post has an interesting idea: > "A server that I use for mail etc., deals with protecting its users from themselves by regularly attempting dictionary and bruteforce attacks upon its own password file. Any passwords that get broken get expired (forcing a change next login) and the user gets a (mostly polite) email explaining why their password was expired and how to avoid it happening again." Thoughts? [1] http://www.schneier.com/blog/archives/2012/04/password_securi.html#c742926 http://www.schneier.com/blog/archives/2012/04/password_secur...
  - harshreality 14y ago
    If they're using pbkdf2, scrypt, bcrypt, etc., there's a much better option than attempting to brute force a database of salted passwords. You could check a lot (depending on language speed) of common passwords whenever a password is chosen, or during login, when you still have the password in cleartext. When it passes the checks, set a flag for that user so that the check doesn't have to be repeated. If it takes to long to do during login, have a counter that keeps track of where in a list of passwords the checker is at, and do N more each login.
    - kijin 14y ago
      According to [1], the 100 most frequently used passwords are used by 40% of people, and the 500 most frequently used passwords are used by 71% of people. So you can increase security for over half of your users simply by disallowing the 200 most commonly used passwords. This set of passwords is small enough to fit in an in-memory list, so you don't even need to use a database for that. I don't think there's much need to check for common passwords at each login, provided that none of your users are allowed to use such passwords in the first place. You might get more security by banning IPs that try more than a few of those common passwords in a short time (it's a useful way to detect brute-force attempts), but in fact you should be banning any IP that generates more than a few failed logins in a short time. [1] http://xato.net/passwords/more-top-worst-passwords/ http://xato.net/passwords/more-top-worst-passwords/ Another measure I use on my webapps is calculating how many bits of entropy a password contains, and ban anything less than ~40 bits (the threshold depends on the target demographic). An 8-digit number has 10^8 = 26.5 bits of entropy. An 8-char lowercase string has 26^8 = 37.6 bits of entropy (unless it's a common word). A mixed-case 8-char string that also contains numbers has (26+26+10)^8 = 47.6 bits of entropy. This method has the advantage that I don't need to specify annoying rules like "one lowercase, one uppercase, one symbol, etc." Simpler passwords can be allowed if they are long enough, and complex passwords can be banned if they are too short.
harshreality 14y ago
In addition, the web interface lacks: - 2-factor auth [1] - X-Frame-Options [2] - HSTS [2] - Redirect from http to https prior to login. (if someone bookmarks the http url, for example, they can be mitm'd anytime they use that bookmark) [2] These have been pointed out on the forums long ago (see cites) but have not been implemented. [1] http://forum.linode.com/viewtopic.php?f=7&t=7639 http://forum.linode.com/viewtopic.php?f=7&t=7639 [2] http://forum.linode.com/viewtopic.php?f=7&t=8764 http://forum.linode.com/viewtopic.php?f=7&t=8764
- JshWright 14y ago
  It doesn't always redirect... If you visit http://manager.linode.com/session/index http://manager.linode.com/session/index it'll happily let you log in via HTTP.
zhoutong 14y ago
As a former developer of Bitcoinica, the most affected customer in the previous Linode incident at that time, I have to say that Linode's communication was extremely unresponsive. It took them 15 days to write a reply that is still full of ambiguity. In the end they offered Bitcoinica one-year worth of services, which are basically nothing compared to the huge losses. I can expect 7 other customers to be treated similarly. There were no details available at all. Not only they didn't follow up to the public, there's a shockingly lack of transparency to their affected customers as well.

by mrb 14y ago

flebron 14y ago

Why do you think in both cases this was the result of some sort of mistake on Linode's part, versus an oversight on the part of the target? That is, you use the expression 'they {linode} got breached', versus the more correct 'they {the customers} got breached'. How do you know that in both cases this was the case?

raquo 14y ago
If it was one customer's fault, why reset all Linode users' passwords?
- flebron 14y ago
  If you knew there was a cracker actively using your network, would you not take any other precautions other than changing the one password you know he used? One could argue Linode was overzealous in resetting all passwords. Reasonable minds can differ.
  - JshWright 14y ago
    That depends completely on the method of compromise... If the compromise was due to a weak password that was brute forced, or obtained from the end user in some other way, then there is absolutely no need to reset every user's password. The only time a 'reasonable mind' would think to reset everyone's password is if the attacker exploited some flaw in the underlying system, which could have given them access to arbitrary passwords.
    - StavrosK 14y ago
      Yeah, it does sound like there's something to the story that they aren't disclosing, and it doesn't inspire confidence.
  - mnordhoff 14y ago
    Malicious traffic doesn't make routers burst into flame. Customers get compromised on a daily basis at any large provider due to bad SSH passwords or WordPress vulnerabilities or whatever. Assuming the virtualization system is trustworthy (which it is), that's not a problem. Assuming it's not trustworthy, they shouldn't be running a virtual hosting company and letting random strangers sign up in the first place!
  - ceejayoz 14y ago
    Any decently sized web host is almost certain to have crackers actively using their network at any time. AWS probably has thousands of instances running old WP installs that've been compromised.
seany 14y ago
The lack of transparency from them makes it impossible to know one way or the other. The xen security issue is another example of this kind of opaqueness.
asdfaoeu 14y ago
The wording they use makes it sound like the attackers compromised some Linode service to gain access to a specific customers machine rather than that customer themselves being exploited. If it was just a compromise of that customer there would obviously be no need for this announcement or resetting of passwords. At the very least they don't know which it was.
mrb 14y ago
In March 2012, Linode acknowledged it was a breach of their internal systems, not the customers'. Today, they are reseting the password of all customers, making it pretty certain this is another breach at the Linode level, not at the customer's level.

akkartik 14y ago

It just occurred to me that https://www.schneier.com/blog/archives/2012/04/password_securi.html https://www.schneier.com/blog/archives/2012/04/password_secu... was in response to last year's breach.

jameswyse 14y ago
One of the comments[1] on that post has an interesting idea: > "A server that I use for mail etc., deals with protecting its users from themselves by regularly attempting dictionary and bruteforce attacks upon its own password file. Any passwords that get broken get expired (forcing a change next login) and the user gets a (mostly polite) email explaining why their password was expired and how to avoid it happening again." Thoughts? [1] http://www.schneier.com/blog/archives/2012/04/password_securi.html#c742926 http://www.schneier.com/blog/archives/2012/04/password_secur...
- harshreality 14y ago
  If they're using pbkdf2, scrypt, bcrypt, etc., there's a much better option than attempting to brute force a database of salted passwords. You could check a lot (depending on language speed) of common passwords whenever a password is chosen, or during login, when you still have the password in cleartext. When it passes the checks, set a flag for that user so that the check doesn't have to be repeated. If it takes to long to do during login, have a counter that keeps track of where in a list of passwords the checker is at, and do N more each login.
  - kijin 14y ago
    According to [1], the 100 most frequently used passwords are used by 40% of people, and the 500 most frequently used passwords are used by 71% of people. So you can increase security for over half of your users simply by disallowing the 200 most commonly used passwords. This set of passwords is small enough to fit in an in-memory list, so you don't even need to use a database for that. I don't think there's much need to check for common passwords at each login, provided that none of your users are allowed to use such passwords in the first place. You might get more security by banning IPs that try more than a few of those common passwords in a short time (it's a useful way to detect brute-force attempts), but in fact you should be banning any IP that generates more than a few failed logins in a short time. [1] http://xato.net/passwords/more-top-worst-passwords/ http://xato.net/passwords/more-top-worst-passwords/ Another measure I use on my webapps is calculating how many bits of entropy a password contains, and ban anything less than ~40 bits (the threshold depends on the target demographic). An 8-digit number has 10^8 = 26.5 bits of entropy. An 8-char lowercase string has 26^8 = 37.6 bits of entropy (unless it's a common word). A mixed-case 8-char string that also contains numbers has (26+26+10)^8 = 47.6 bits of entropy. This method has the advantage that I don't need to specify annoying rules like "one lowercase, one uppercase, one symbol, etc." Simpler passwords can be allowed if they are long enough, and complex passwords can be banned if they are too short.

harshreality 14y ago

In addition, the web interface lacks: - 2-factor auth [1] - X-Frame-Options [2] - HSTS [2] - Redirect from http to https prior to login. (if someone bookmarks the http url, for example, they can be mitm'd anytime they use that bookmark) [2] These have been pointed out on the forums long ago (see cites) but have not been implemented. [1] http://forum.linode.com/viewtopic.php?f=7&t=7639 http://forum.linode.com/viewtopic.php?f=7&t=7639 [2] http://forum.linode.com/viewtopic.php?f=7&t=8764 http://forum.linode.com/viewtopic.php?f=7&t=8764

JshWright 14y ago
It doesn't always redirect... If you visit http://manager.linode.com/session/index http://manager.linode.com/session/index it'll happily let you log in via HTTP.

zhoutong 14y ago

As a former developer of Bitcoinica, the most affected customer in the previous Linode incident at that time, I have to say that Linode's communication was extremely unresponsive. It took them 15 days to write a reply that is still full of ambiguity. In the end they offered Bitcoinica one-year worth of services, which are basically nothing compared to the huge losses. I can expect 7 other customers to be treated similarly. There were no details available at all. Not only they didn't follow up to the public, there's a shockingly lack of transparency to their affected customers as well.