7 ms·

Why LivingSocial’s 50 million password breach is graver than you may think

45 points by darxius 13y ago

ExpiredLink 13y ago
I wonder why password tables are readable at all. An application only needs to call a stored procedure that compares the computed hash value against the stored. This stored procedure needs the rights to read the data. No application, no administrator should to be able to read that data.
- misnome 13y ago
  I suspect it's a side-effect of the fact that web frameworks tend to encourage "pass the database login once and let the framework manage everything", so there is no protection if you manage to get into whatever is running the site.
  - sk5t 13y ago
    The "one database login" rule (a.k.a. trusted platform design) doesn't necessitate reading the raw password hashes; the app account can be granted only permission to run a stored procedure that does the comparison yielding a yes/no answer, and denied 'select' on the views/tables/etc.
    - shabble 13y ago
      You should remember to use constant-time comparison when checking for equality in your stored procedure though; do databases typically provide functions/operators for that? It's something just as easy to get wrong in application code, but good frameworks already do, and abstract it away where you might not realise.
- Drakim 13y ago
  The issue is how you enforce this. Obviously if somebody has root on your server they can just disregard any rules for how the database should be accessed. Maybe by having the database on external hardware with an API, but I can imagine this is hard to scale. What if you need to sync it across several load balancing servers, or what if you want to make backups?
  - ExpiredLink 13y ago
    > Obviously if somebody has root on your server ... ... then it's game over for you. But I guess intruders typically use either SQL-injection or somehow get the application password to access the password table. In both cases an unreadable table would thwart the intruder.
    - Drakim 13y ago
      But is it? If the only way your production server can check passwords is by sending hash(salt+pw+salt) and getting a true\false reply from a piece of external hardware...then even if an intruder got complete control though some exploit or injection, he wouldn't be able to dump the database for later cracking with rainbow tables.
      - shabble 13y ago
        You may have just re-invented the Hardware Security Module(HSM)[1] :-) See also [2] for a cute implementation using an Arduino to secure an AWS API key [1] https://en.wikipedia.org/wiki/Hardware_security_module https://en.wikipedia.org/wiki/Hardware_security_module [2] http://stefan.arentz.ca/signing-aws-requests-with-your-arduino.html http://stefan.arentz.ca/signing-aws-requests-with-your-ardui...
  - ippisl 13y ago
    >> what if you want to make backups? One can design a special hardware, with a port for just comparing passwords and a port for copying. A Hacker would need physical access to copy the database. >> Load balancing Could be part of the design.Say card in a box, with internal physical connection inside the box for copying.
- tptacek 13y ago
  This isn't a bad idea at all, but if you lose control of your database, you need to assume you lost control of your app servers and your database server.
- marquis 13y ago
  postgres provides encryption functions but you still have to take care of your keys.
jgrahamc 13y ago
One way they could have upgraded their database to make it more secure: http://blog.jgc.org/2012/06/one-way-to-fix-your-rubbish-password.html http://blog.jgc.org/2012/06/one-way-to-fix-your-rubbish-pass...
- kyrra 13y ago
  I would vote for PBKDF2-HMAC-SHA512. The SHA512 part makes it a little harder to do in GPU as key lengths are 64-bit vs 32-bit for sha256. Pbkdf2 has been recommended by NIST, while the main crypto algorithm for bcrypt hasn't seen as much attention (though it has been around for a while and is probably safe). Edit: Though, it looks like PBKDF2-HMAC-SHA512 is not FIPS compliant.
  - tptacek 13y ago
    So, this isn't a bad recommendation, in that PBKDF2-XXX for any XXX is much better than "salted hashes", but I'm confused about this meme that recommends PBKDF2 above bcrypt and scrypt, both of which are better, marginally in bcrypt's case, and significantly in scrypt's. SHA2-512 is designed to be fast on hardware.
    - kyrra 13y ago
      True true, all of them are better than just salted hashes. It is much harder to brute-force bcrypt and scrypt in hardware (ASIC, GPUs, etc...). It's just important to understand the differences and trust behind the different algorithms. PBKDF2 is NIST recommended, which I would say speaks to its strength (unlikely it will be easily cracked). It can also be used with any input size. Though, calculation requires little memory, so it can easily be implemented in hardware. bcrypt has been around since 1999 has is still going strong, but is limited to input sizes of 55-characters. scrypt is fairly new and I don't feel has been wrong long enough to be vetted by the security community.
      - tptacek 13y ago
        You say it's "unlikely it will be easily cracked", but, obviously, it's demonstrably easier to crack than bcrypt and scrypt.
hashmymustache 13y ago
Are tech companies that store a fair amount of personally identifying information on millions of people ever subject to security audits? Not saying it would help, but it seems like as attacks get more prevalent and old security practices become obsolete that users might be able sue companies for negligence by not using optimal data encryption. Of course the tech company would counter sue for stupid passwords. And even if they want to restrict users from dictionary passwords with symbol replacements, hey, it's patented.
- jaredmcateer 13y ago
  I think in N. America if you do credit card processing you have to conform to PCI standards, unfortunately those standards are loosely audited and relies mostly on self-reporting. More than a few of the security protocols are cargo culty and even detrimental to security.
  - deleted 13y ago
    [deleted]
    - imbriaco 13y ago
      Before you decide to be condescending, you should probably freshen up yourself. The vast majority of merchants fall into PCI-DSS Level 2-4. Those merchants complete a self-assessment questionnaire (SAQ), in other words, self-reporting. They also have to pass a rather rudimentary quarterly automated scan from an approved vendor. You don't get to external auditors until you get to Level 1, which is a vanishingly small percentage of the merchants in the world.
- fizx 13y ago
  My understanding is that the first time you are big enough and screw anything up (leak some PII, mess up your privacy policy, etc), the FTC gets involved on behalf of the citizens, and then you settle with them, the terms of which include ISO-whatever compliant audits for the next few dozen years. So Facebook, Twitter, Google, Dropbox are very likely getting audited yearly. I don't know whether e.g. Pinterest has screwed up yet, so they might not be.
casca 13y ago
Everyone should be salting and hashing correctly - it's trivially easy to implement and the cost is trivial. However, once the user database is stolen in its entirety, there are other factors that become as important: - Did the user use a common passwords? If they did, then it will quick to brute-force. - Did the user reuse the same credentials on other sites? If they did then the value of recovering the password becomes worth plowing through well-salted scrypt. The real problem is that far too many people use the same credentials for LivingSocial and their Gmail.
- misnome 13y ago
  It was my understanding that hashing and salting is now obsolete, and doesn't really increase security that much. AFAIK, the best 'recommended' way currently is to use bcrypt.
  - sbov 13y ago
    My understanding was that bcrypt uses a salt and is a hash.
    - carbocation 13y ago
      In the same way the US Army consists of a soldier, so too is bcrypt a salt and a hash.
  - lockes5hadow 13y ago
    No, in fact they are a part of bcrypt, along with other great things.
  - UnoriginalGuy 13y ago
    Your understand is completely wrong. Plus bcrypt is used as a hashing function and includes a salt. So even if what you said was true, it would make bcrypt obsolete. Yes, I know, bcrypt uses a block cipher but block cipher can be easily used for cryptographic hashes (see DES for one example).
mistercow 13y ago
>LivingSocial engineers should be applauded for adding cryptographic salt Should they? Should they really? Should we also give them a cookie for properly dressing themselves in the morning? Uniquely salting and hashing passwords is the bare minimum for acceptable security practices. It is not a sign of expertise, but of competence.
swanson 13y ago
I don't get why they used SHA-1. I mean, LivingSocial has been sending people to speak and attend a lot of developer conferences. They hire a lot of bright people. Did none of the Rails devs raise an alert of like "Uhh hey guys, why don't we use Rails-community standard bcrypt?". I suppose it could be corporate bureaucracy or something, but it seems weird that a company with at least 30+ developers (based on public members on the github org) no one would point out that this is a Bad Idea.
- cheald 13y ago
  If I had to guess - it was done long ago when SHA-1 was what you used because it wasn't MD5, and nobody thought to change it since it was working.
  - swanson 13y ago
    Maybe - but you would think that a company with "Tech Leads" would think to check that kind of stuff. Or some new hire would come in and ask innocently "Hey what do we use for password hashing". I find it pretty hard to believe that no one would ever poke around the User model in a Rails app :)
  - lawnchair_larry 13y ago
    There has really never been a time where SHA1 was appropriate.
    - cheald 13y ago
      Sure there was - before the debut of high-powered GPUs. Try cracking SHA1-hashed passwords on CPUs - even modern ones - and let me know how far that gets you. By comparison, today's bcrypt passwords at today's work factors are tomorrow's ASIC snacks. Doesn't mean that bcrypt at reasonable work factors are wrong today.
- rmrfrmrf 13y ago
  I could think of a few reasons: 1. user management is a pretty trivial thing to code and one of the first things you do when you set up any site. It could have been a simple oversight before a rushed launch. 2. Playing off of 1), once you have any users in a database using a particular system, you have to go through a bit of a process to convert people over (you can't convert from one hashing algo to another yourself since you don't have the original passwords). That involves creating a concurrent password table, creating login hooks that force the user to upgrade their password on login (especially a pain if they're using native apps -- i'm not sure if they are), etc. Making sure that 50 million users migrate over without a hitch is a potentially site-breaking undertaking that could have serious impact on the site's success. Given the rocky climate LS currently faces, you'd have to have nerves of steel to lead a n initiative like that. 3. Depending on how LivingSocial has their code management structure set up, there may only be a few devs who actually have access to the user login system (you really don't want an intern or freelancer to be able to modify those). Given that the same devs who have that access probably have more immediate issues to deal with day-to-day, something like the current hashing algorithm might not be on the top of their priority list. Not that any of these are very good excuses, but I could see how it could happen.
FooBarWidget 13y ago
Hearing about all these recent hacks (LinkedIn, RubyGems, LivingSocial) really makes me nervous and wonder how I can avoid hacks. Are there any details on how these systems were hacked in the first place, so that the rest of us can learn from the lessons? I already use as many security good practices as I can (bcrypt, processes and apps running under different user accounts and database accounts, read-only accounts, principle of least privilege, firewall, fail2ban, key-only SSH logins, file modification monitoring, forwarding logs to a log host, etc) but I can't shake off the feeling that I may be forgetting something.
- larrys 13y ago
  "and wonder how I can avoid hacks" (I'll leave it to others more knowledgeable to answer this but keep what I say next in mind.) My question to you is, how big of a target are you? I mean the security you would use if you are Madonna is not the same as a "B" list actor or the person working in the local coffee shop. You can put untold hours into this but unfortunately security is a full time job and can also detract from other things you need to be concerned with that might be more important for your particular situation. "I may be forgetting something" Sure but the same concern is in play when taking a trip or with anything. What are you forgetting? Where are you going? How easy will it be to solve the problem and how much impact will forgetting dental floss have? Are you going to an uninhabited island or Manhattan where you can buy almost anything 24/7?
  - laumars 13y ago
    What you're advocating is bordering dangerously into the realm of "security through obscurity" - a very bad principle to base any kind of security on. While the former member might not be a high profile target for the moment (and let's assume that he isn't for the sake of devils advocate), who's to say that next week a story hits the news about his wife or children or even a company he's affiliated with and then reporters decide to start poking about on forums and social media for cheap information to write sensationalist reports on? Or that hackers use that news as an excuse to target the poor guy? Or even who's to say that next month he annoys someone on HN who decides to play some pranks? While you are right that that security is a full time job, you should never advocate the numbers game because as remote as the odds might be, people do win the lottery.
mattberg 13y ago
Sorry if this has been asked, but I haven't seen anything on it yet. What about users that only signed in via Facebook? Any actions I need to take?
CoachRufus87 13y ago
Does anyone actually know how to close a livingsocial account? I scoured their site but saw nothing that pointed me in the right direction.
sabat 13y ago
Although LivingSocial is guilty of pretty poor decision-making (SHA-1? really?), I found this article more sensational and scare-mongering than substantial. salting in no way makes cracking more difficult Oh really? Since when? Salting isn't a silver bullet, but it certainly does making cracking more difficult.

45 points by darxius 13y ago

I wonder why password tables are readable at all. An application only needs to call a stored procedure that compares the computed hash value against the stored. This stored procedure needs the rights to read the data. No application, no administrator should to be able to read that data.

misnome 13y ago
I suspect it's a side-effect of the fact that web frameworks tend to encourage "pass the database login once and let the framework manage everything", so there is no protection if you manage to get into whatever is running the site.
- sk5t 13y ago
  The "one database login" rule (a.k.a. trusted platform design) doesn't necessitate reading the raw password hashes; the app account can be granted only permission to run a stored procedure that does the comparison yielding a yes/no answer, and denied 'select' on the views/tables/etc.
  - shabble 13y ago
    You should remember to use constant-time comparison when checking for equality in your stored procedure though; do databases typically provide functions/operators for that? It's something just as easy to get wrong in application code, but good frameworks already do, and abstract it away where you might not realise.
Drakim 13y ago
The issue is how you enforce this. Obviously if somebody has root on your server they can just disregard any rules for how the database should be accessed. Maybe by having the database on external hardware with an API, but I can imagine this is hard to scale. What if you need to sync it across several load balancing servers, or what if you want to make backups?
- ExpiredLink 13y ago
  > Obviously if somebody has root on your server ... ... then it's game over for you. But I guess intruders typically use either SQL-injection or somehow get the application password to access the password table. In both cases an unreadable table would thwart the intruder.
  - Drakim 13y ago
    But is it? If the only way your production server can check passwords is by sending hash(salt+pw+salt) and getting a true\false reply from a piece of external hardware...then even if an intruder got complete control though some exploit or injection, he wouldn't be able to dump the database for later cracking with rainbow tables.
    - shabble 13y ago
      You may have just re-invented the Hardware Security Module(HSM)[1] :-) See also [2] for a cute implementation using an Arduino to secure an AWS API key [1] https://en.wikipedia.org/wiki/Hardware_security_module https://en.wikipedia.org/wiki/Hardware_security_module [2] http://stefan.arentz.ca/signing-aws-requests-with-your-arduino.html http://stefan.arentz.ca/signing-aws-requests-with-your-ardui...
- ippisl 13y ago
  >> what if you want to make backups? One can design a special hardware, with a port for just comparing passwords and a port for copying. A Hacker would need physical access to copy the database. >> Load balancing Could be part of the design.Say card in a box, with internal physical connection inside the box for copying.
tptacek 13y ago
This isn't a bad idea at all, but if you lose control of your database, you need to assume you lost control of your app servers and your database server.
marquis 13y ago
postgres provides encryption functions but you still have to take care of your keys.

jgrahamc 13y ago

One way they could have upgraded their database to make it more secure: http://blog.jgc.org/2012/06/one-way-to-fix-your-rubbish-password.html http://blog.jgc.org/2012/06/one-way-to-fix-your-rubbish-pass...

kyrra 13y ago
I would vote for PBKDF2-HMAC-SHA512. The SHA512 part makes it a little harder to do in GPU as key lengths are 64-bit vs 32-bit for sha256. Pbkdf2 has been recommended by NIST, while the main crypto algorithm for bcrypt hasn't seen as much attention (though it has been around for a while and is probably safe). Edit: Though, it looks like PBKDF2-HMAC-SHA512 is not FIPS compliant.
- tptacek 13y ago
  So, this isn't a bad recommendation, in that PBKDF2-XXX for any XXX is much better than "salted hashes", but I'm confused about this meme that recommends PBKDF2 above bcrypt and scrypt, both of which are better, marginally in bcrypt's case, and significantly in scrypt's. SHA2-512 is designed to be fast on hardware.
  - kyrra 13y ago
    True true, all of them are better than just salted hashes. It is much harder to brute-force bcrypt and scrypt in hardware (ASIC, GPUs, etc...). It's just important to understand the differences and trust behind the different algorithms. PBKDF2 is NIST recommended, which I would say speaks to its strength (unlikely it will be easily cracked). It can also be used with any input size. Though, calculation requires little memory, so it can easily be implemented in hardware. bcrypt has been around since 1999 has is still going strong, but is limited to input sizes of 55-characters. scrypt is fairly new and I don't feel has been wrong long enough to be vetted by the security community.
    - tptacek 13y ago
      You say it's "unlikely it will be easily cracked", but, obviously, it's demonstrably easier to crack than bcrypt and scrypt.

hashmymustache 13y ago

Are tech companies that store a fair amount of personally identifying information on millions of people ever subject to security audits? Not saying it would help, but it seems like as attacks get more prevalent and old security practices become obsolete that users might be able sue companies for negligence by not using optimal data encryption. Of course the tech company would counter sue for stupid passwords. And even if they want to restrict users from dictionary passwords with symbol replacements, hey, it's patented.

jaredmcateer 13y ago
I think in N. America if you do credit card processing you have to conform to PCI standards, unfortunately those standards are loosely audited and relies mostly on self-reporting. More than a few of the security protocols are cargo culty and even detrimental to security.
- deleted 13y ago
  [deleted]
  - imbriaco 13y ago
    Before you decide to be condescending, you should probably freshen up yourself. The vast majority of merchants fall into PCI-DSS Level 2-4. Those merchants complete a self-assessment questionnaire (SAQ), in other words, self-reporting. They also have to pass a rather rudimentary quarterly automated scan from an approved vendor. You don't get to external auditors until you get to Level 1, which is a vanishingly small percentage of the merchants in the world.
fizx 13y ago
My understanding is that the first time you are big enough and screw anything up (leak some PII, mess up your privacy policy, etc), the FTC gets involved on behalf of the citizens, and then you settle with them, the terms of which include ISO-whatever compliant audits for the next few dozen years. So Facebook, Twitter, Google, Dropbox are very likely getting audited yearly. I don't know whether e.g. Pinterest has screwed up yet, so they might not be.

casca 13y ago

Everyone should be salting and hashing correctly - it's trivially easy to implement and the cost is trivial. However, once the user database is stolen in its entirety, there are other factors that become as important: - Did the user use a common passwords? If they did, then it will quick to brute-force. - Did the user reuse the same credentials on other sites? If they did then the value of recovering the password becomes worth plowing through well-salted scrypt. The real problem is that far too many people use the same credentials for LivingSocial and their Gmail.

misnome 13y ago
It was my understanding that hashing and salting is now obsolete, and doesn't really increase security that much. AFAIK, the best 'recommended' way currently is to use bcrypt.
- sbov 13y ago
  My understanding was that bcrypt uses a salt and is a hash.
  - carbocation 13y ago
    In the same way the US Army consists of a soldier, so too is bcrypt a salt and a hash.
- lockes5hadow 13y ago
  No, in fact they are a part of bcrypt, along with other great things.
- UnoriginalGuy 13y ago
  Your understand is completely wrong. Plus bcrypt is used as a hashing function and includes a salt. So even if what you said was true, it would make bcrypt obsolete. Yes, I know, bcrypt uses a block cipher but block cipher can be easily used for cryptographic hashes (see DES for one example).

mistercow 13y ago

>LivingSocial engineers should be applauded for adding cryptographic salt Should they? Should they really? Should we also give them a cookie for properly dressing themselves in the morning? Uniquely salting and hashing passwords is the bare minimum for acceptable security practices. It is not a sign of expertise, but of competence.

swanson 13y ago

I don't get why they used SHA-1. I mean, LivingSocial has been sending people to speak and attend a lot of developer conferences. They hire a lot of bright people. Did none of the Rails devs raise an alert of like "Uhh hey guys, why don't we use Rails-community standard bcrypt?". I suppose it could be corporate bureaucracy or something, but it seems weird that a company with at least 30+ developers (based on public members on the github org) no one would point out that this is a Bad Idea.

cheald 13y ago
If I had to guess - it was done long ago when SHA-1 was what you used because it wasn't MD5, and nobody thought to change it since it was working.
- swanson 13y ago
  Maybe - but you would think that a company with "Tech Leads" would think to check that kind of stuff. Or some new hire would come in and ask innocently "Hey what do we use for password hashing". I find it pretty hard to believe that no one would ever poke around the User model in a Rails app :)
- lawnchair_larry 13y ago
  There has really never been a time where SHA1 was appropriate.
  - cheald 13y ago
    Sure there was - before the debut of high-powered GPUs. Try cracking SHA1-hashed passwords on CPUs - even modern ones - and let me know how far that gets you. By comparison, today's bcrypt passwords at today's work factors are tomorrow's ASIC snacks. Doesn't mean that bcrypt at reasonable work factors are wrong today.
rmrfrmrf 13y ago
I could think of a few reasons: 1. user management is a pretty trivial thing to code and one of the first things you do when you set up any site. It could have been a simple oversight before a rushed launch. 2. Playing off of 1), once you have any users in a database using a particular system, you have to go through a bit of a process to convert people over (you can't convert from one hashing algo to another yourself since you don't have the original passwords). That involves creating a concurrent password table, creating login hooks that force the user to upgrade their password on login (especially a pain if they're using native apps -- i'm not sure if they are), etc. Making sure that 50 million users migrate over without a hitch is a potentially site-breaking undertaking that could have serious impact on the site's success. Given the rocky climate LS currently faces, you'd have to have nerves of steel to lead a n initiative like that. 3. Depending on how LivingSocial has their code management structure set up, there may only be a few devs who actually have access to the user login system (you really don't want an intern or freelancer to be able to modify those). Given that the same devs who have that access probably have more immediate issues to deal with day-to-day, something like the current hashing algorithm might not be on the top of their priority list. Not that any of these are very good excuses, but I could see how it could happen.

FooBarWidget 13y ago

Hearing about all these recent hacks (LinkedIn, RubyGems, LivingSocial) really makes me nervous and wonder how I can avoid hacks. Are there any details on how these systems were hacked in the first place, so that the rest of us can learn from the lessons? I already use as many security good practices as I can (bcrypt, processes and apps running under different user accounts and database accounts, read-only accounts, principle of least privilege, firewall, fail2ban, key-only SSH logins, file modification monitoring, forwarding logs to a log host, etc) but I can't shake off the feeling that I may be forgetting something.

larrys 13y ago
"and wonder how I can avoid hacks" (I'll leave it to others more knowledgeable to answer this but keep what I say next in mind.) My question to you is, how big of a target are you? I mean the security you would use if you are Madonna is not the same as a "B" list actor or the person working in the local coffee shop. You can put untold hours into this but unfortunately security is a full time job and can also detract from other things you need to be concerned with that might be more important for your particular situation. "I may be forgetting something" Sure but the same concern is in play when taking a trip or with anything. What are you forgetting? Where are you going? How easy will it be to solve the problem and how much impact will forgetting dental floss have? Are you going to an uninhabited island or Manhattan where you can buy almost anything 24/7?
- laumars 13y ago
  What you're advocating is bordering dangerously into the realm of "security through obscurity" - a very bad principle to base any kind of security on. While the former member might not be a high profile target for the moment (and let's assume that he isn't for the sake of devils advocate), who's to say that next week a story hits the news about his wife or children or even a company he's affiliated with and then reporters decide to start poking about on forums and social media for cheap information to write sensationalist reports on? Or that hackers use that news as an excuse to target the poor guy? Or even who's to say that next month he annoys someone on HN who decides to play some pranks? While you are right that that security is a full time job, you should never advocate the numbers game because as remote as the odds might be, people do win the lottery.

mattberg 13y ago

Sorry if this has been asked, but I haven't seen anything on it yet. What about users that only signed in via Facebook? Any actions I need to take?

CoachRufus87 13y ago

Does anyone actually know how to close a livingsocial account? I scoured their site but saw nothing that pointed me in the right direction.

sabat 13y ago

Although LivingSocial is guilty of pretty poor decision-making (SHA-1? really?), I found this article more sensational and scare-mongering than substantial. salting in no way makes cracking more difficult Oh really? Since when? Salting isn't a silver bullet, but it certainly does making cracking more difficult.