14 ms·

Because there's two types of storing passwords in plain text. There's the "your password is stored in plaintext in the database" way which everyone agrees is 10

• dcow 8y ago
  The point perhaps is that they’re both terrible security designs and the impact to the user is the same regardless of how the “accident” happened. Nobody in FB’s position gets a pass for being irresponsible and negligent.
  • EnFinlay 8y ago
    I disagree that they are both terrible security designs. We expect every company to not use plaintext for auth. We do not expect every company to have infra/ops setup to prevent logging on login requests.
    • dcow 8y ago
      That’s certifiably depressing.
      • EnFinlay 8y ago
        It might be depressing, but I think it's true. What do you think?
        • EpicEng 8y ago
          It is not true in my experience. It may seem true to the typical CRUD app web dev, but do they really know what it's like to be a custodian of so much sensitive information?
    • aequitas 8y ago
      By extending this logic, a car manufacturer should be blamed for not designing proper brakes for their car. But if a worker then accidentally installs the breaks wrong they are not responsible? Imho, a company (especially as big as facebook) should have the right process and procedures to prevent these kind of problems and ensure developers have proper training to make them aware of the consequences of their actions.
      • EnFinlay 8y ago
        Interesting analogy and I see your point. If a car manufacturer's process made it easy to install the brakes wrong they would be held responsible (probably with a recall or damages for lives lost due to faulty brakes). I guess part of this is that passwords aren't considered that important to many people :(
    • EpicEng 8y ago
      Who is "we" in this scenario? Governing bodies do. Engineers I've worked with in health care do, as do our PM's, security officers, etc. I designed a clinical testing platform a couple of years ago. Our initial requirements stated very clearly that PHI and PII were not to appear in logs. This is basic stuff for anyone who actually works at this scale / level of sensitivity.
    • code_duck 8y ago
      I would expect FB to have that, though.
    • deleted 8y ago
      [deleted]
    • nck4222 8y ago
      >We do not expect every company to have infra/ops setup to prevent logging on login requests. What? I absolutely expect every company to not log my password in plain text. In my 15 years as a developer across several companies and industries, I have never seen anybody log passwords, or advocate for logging passwords. I'm struggling to think why any employee of any company should be able to view a plain text password in any form. Why would there not be an expectation here?
      • viraptor 8y ago
        You're taking about expectation not to log the password. That's fine. The parent was taking about expectation about infrastructure that validates this. This is both very uncommon and impossible to do 100% correctly. You can scan logs for a prefix (password=), you can do entropy counting, you can try to decode hex values in text. But if you find a base64 encoded hex string representing "foobar" - how do you even know it's a password? Short of trying all possible decodings of all possible substrings against your full password database, this is an impossible task. (You can do best-effort things though)
        • nck4222 8y ago
          Ah ok thanks, I misunderstood.
• kitsune_ 8y ago
  Data anonymization and de-identification is nothing new. Especially when it comes to logging. I don't get why a lot of you are downplaying it in this thread. This is just as bad as plain text passwords.
  • cflewis 8y ago
    Because we all know we are one bad morning away from doing it ourselves. People are distinguishing between extreme incompetence (storing plaintext passwords in databases) vs people trying to do their jobs. It's the same reason when there are large outages, where the comments are split between the enraged customers, and then the ones that are "Man, sucks to be them" as know they could easily have been the one that got the config push wrong.
    • 908087 8y ago
      If a single developer is able to push something like this to production without scrutiny in "one bad morning" on a platform with billions of users, the company itself is the problem.
    • kitsune_ 8y ago
      The fact that sensitive data like this is just one config push away from being exposed means it's a brittle design (not just the code, but the entire design /review / deployment process - this is a platform with 1 billion users, not your 1-dev WordPress webpage). That said, this is further aggravated by the simple fact that for whatever reason, 20k of their employees had access to these logs. All in all it just paints a picture of an irresponsible company where their house is not in order.
      • lostapathy 8y ago
        Exactly. It's one thing not to have enough controls in place to catch something like this on the forum for your warcraft guild. It's another thing entirely when you operate at the scale of Facebook.
    • untog 8y ago
• rhacker 8y ago
  Maybe the HN crowd needs to re-think login security. About 25 years ago challenge-response was a HUGE thing in login security. As https gained momentum and traction that went away. But the nice thing about challenge-response (at least for Javascript enabled clients) is that the password is one-way hashed before sending to the server. I wonder if it's time to up our game again and go back to that model.
  • pferde 8y ago
    I was just thinking the same thing. There is no reason the password itself even has to be sent over the wire.
• pwg 8y ago
  Different causes, identical security implications for the user's passwords that were improperly stored. Both are security fails, regardless of the cause of the fail. And a company the size of, and with the resources of, facebook, most certainly should not get a pass for the "oops, we logged more than we should have" cause. A corp. of their size, and with their resources, should be doing password handling correctly, every single time, no exceptions, no excuses.
• scriptkiddy 8y ago
  > and the other is a very easy mistake to make which can go unnoticed for a long time (because you can be attempting to log things completely unrelated to logins). This is not an excuse. If you're logging all request data, you need to strip or encrypt sensitive information in that request data. Handling Persistence of sensitive data is web development 101. Just because it's not in a database doesn't give you a pass to leave it unprotected. This level of incompetency is unacceptable.
• voidlogic 8y ago
  All the more reason to make the client side send HMAC(HMAC(username + password) + Unix Epoch rounded to last 5 min block)) over the wire in its POST to the auth endpoint. All the transport encryption and DB encryption/hashing/salting won't protect you from this kind of logging mistake, but the above would. P.S. There are ways to make the above even better by adding a nonce that has to be requested from the server before POST etc.
• dunham 8y ago
  This happened to Apple on the desktop, CVE-2014-1317 - In that case, they were logging the body of failed API requests (hex encoded) to a file in /var/log. It turned out that the login request occasionally hit an error, logging your AppleID password. It was easy to overlook, since as you said, it was intended to log something else (failed API requests), it only happened in the case of an error, and it was hex encoded. I only happened to stumble across it out of curiosity.
  • captainredbeard 8y ago
    The Apple ID team at Apple is one of the worst.
    • pmart123 8y ago
      Interesting. Care to elaborate?
      • eppsilon 8y ago
        Anecdotally, Apple ID auth on iOS/macOS has been a mess for me. Changing my password would result in multiple login prompts per device. Sometimes inputting the new password works, sometimes another prompt comes up after a few minutes. Also, though the flexibility of being able to use a different ID for iCloud, home sharing, iTunes, App Store, Messages, etc. is neat, it's pretty annoying to need to set it in each of those places. (And TBH it seems like being able to share purchases/access/etc between IDs is more useful than being able to have separate ones, yet the iCloud family stuff took a while to arrive.) The issues I've seen aren't as bad now as they used to be, but they haven't left a good impression.
• BryantD 8y ago
  The logging issue is something we've become aware of more recently, but it's just as bad. There is no intrinsic difference between the statement "don't store passwords in plain text" and "don't log PII or passwords in plain text."
• EpicEng 8y ago
  I've worked in healthcare / biotech for more than a decade and I can promise you that the FDA would see no difference between the two types of gaffes. As custodians of sensitive information it was our responsibility to ensure that said information didn't leak, period. I don't know anything about FB's infrastructure, but when I was lead I would have viewed leaks in logs as far worse than something in the DB because our DB's were harder to gain access to. I get what you're saying, but it's irrelevant. Easy to screw up, hard to screw up, doesn't matter; just don't screw up because the result is the same. This stuff is security 101. If you're logging requests then you need to ensure they don't contain sensitive info.
  • toomuchtodo 8y ago
    This is how professional software engineering and infrastructure implementation is done, and I appreciate you pointing it out so forcefully.
    • arkitaip 8y ago
      Corollary: Most web development simply lack the rigor of engineering.
      • toomuchtodo 8y ago
        Most software engineers aren’t. Engineering requires licensing and exams through a governing body. Can’t have the prestige of the title without the responsibility. Software developers want to have their cake and eat it too (power and respect with no oversight and governance). Pile on the regulation (GDPR for starts, more PII protection to follow up, extremely painful fines for failures).
        • EnFinlay 8y ago
          I couldn't agree more.
        • jjeaff 8y ago
          In the US, you only need certifications and testing for a "professional engineer". Most electrical, chemical, mechanical, or civil engineers are not certified PEs. Yet I don't think anyone is claiming they aren't engineers.
• munk-a 8y ago
  I think they are indeed the same outcome but I'd count plain-text DB passwords as grossly negligent while as this was merely negligent. Clearly both are terrible and deserve a shaming, but not hashing passwords going into a DB is a decision that someone pretty much necessarily had to have the full scope of and made a bad decision on. This logging issue could potentially be attributed to a mistake in team communication (i.e. the people we hired to auto-log requests didn't realize that `/login` should be blacklisted for logging and for some reason we never explicitly told them to do so). So I agree that these actions are indeed same but different. They ultimately speak to a failure at Facebook, but it doesn't speak to utter incompetence at Facebook.
  • cflewis 8y ago
    Yeah, there's so many ways this could happen. The one that came to mind for me was an engineer who thought passwords were being filtered by the time it got to their part of the stack, but somehow didn't notice they weren't because the payload was minified or because they tested in a test environment where they knew there wasn't the filtering but didn't look at prod (where there was also not the filtering). So many ways this could have gone down, so easy to do.
• samstave 8y ago
  >the other is a very easy mistake to make which can go unnoticed for a long time Sure, then the question for anyone at facebook would be from the Bobs: "What exactly is it, you'd say, you do around here?? FB claims to have "top minds" in essentially every discipline... Heck, they are IMO a revolving door to the .gov/nsa/infosec community... So... I call BS on your statement as "whoops! Easy mistake!" WTF is it that you'd say you do around here, Mr. FB-Security-Guy??
• captainredbeard 8y ago
  If they were following best practices, the server would never have access to plaintext passwords. The client / frontend would hash the password contents and send that across the wire.
  • javagram 8y ago
    That’s hardly a best practice and if followed consistently just means that the hash is your actual password, which means if someone steals the hash from a log file they can still impersonate you. Unless you actually mean some sort of challenge response scheme which is rather uncommon to see, e.g. http “digest” authentication or SRP.
    • qpiox 8y ago
      The big problem of a service having your true password (instead of the hash of the password) is that many users use the same password for a multitude of services (read Gmail, Hotmail, Yahoo, Amazon, ...) So it's bad practice to keep or transfer the users cleartext password. It should never leave her browser/client. Period.
      • javagram 8y ago
        Password managers and service specific 2FA solve that problem quite nicely for now (edit: although yes most users aren’t willing to do that).
      • aeorgnoieang 8y ago
        If the service has only the hash of the password, then that's the password, which would then be subject to the same problems as a "cleartext password".
  • throwawaymath 8y ago
    That's actually not a best practice; on the contrary it's extremely uncommon. Not because it's bad, but because it just doesn't actually add a meaningful security improvement. On the other hand, it does add non-negligible complexity to your authentication system. In particular, it would have done absolutely nothing to prevent this specific vulnerability. If you hash your users' passwords using a key-derivation algorithm on the client-side, each user's password simply becomes the original password's digest. From the server's perspective nothing has changed. Moreover the server will need to re-hash the password digest sent over the wire, because if the server is compromised the password digests can be directly replayed to the server to compromise corresponding accounts. Additionally, since the shared secret between the client and server is the user's password digest, the password needs to be hashed using the same salt every time the user authenticates. So each user's password digest is still a de facto unique password which will be sent over the wire anyway. This scheme basically retrofits the server's job onto the client with added complexity. It's like the (slightly) faster horses version of password authentication, when we could really be experimenting with developing cars (like two/multi-factor authentication, more robust server-side controls and provable correctness). That's not to say the scheme has no benefits whatsoever. It does mean that user passwords will be more complex, because the actual token stored by the server (the de facto password) is a digest. But there are two drawbacks - with enough users you'll still see many duplicated digests in your password database, even if you randomly generate salts on the client side. More importantly, you're offloading hashing to the client side in JavaScript. JavaScript can be very fast in 2019, but companies like Facebook and Google still maintain very low latency, substantially stripped-down versions of their websites[1] because client-side hashing isn't going to be nearly as fast as server-side hashing for a huge number of people. There's also a sizable population of people who don't even have JavaScript enabled, or who might have incompatible browsers. tl;dr - Client-side hashing is not a best practice (and not widely deployed) because it comes with a nontrivial complexity increase, lower client compatibility and negligible security benefits. It also would not have prevented this vulnerability. _______________________ 1. For example, mbasic.facebook.com.
• rubicon33 8y ago
  The irony of this post is that most people would probably beef up the security of their DB, far beyond whatever layer of security they prop up for their logs.
• SonicSoul 8y ago
  there's two types of storing passwords in plain text nope. there are n types of storing passwords in plain text (including storing them in memory). you just named 2. all n types fall under infosec responsibility to prevent, and none are acceptable. there is no pass because logs were less intentional than storing in a table
• ineedasername 8y ago
  Just because it's an easier oversight to make doesn't mean FB should be cut any more slack over it.
• ashelmire 8y ago
  The second is way worse. You don’t need access to the database for it. Most systems make it difficult to log a plaintext password; it should be filtered in your logs. This is day 1 shit.
• dboreham 8y ago
  >Same same but different. They're both issues with which any moderately competent engineer would be very familiar. Heck I've seen commercial contracts that specified that audits be done to ensure no password and PII leaks into log files. Everywhere I've worked in the past 20 years that logged stuff conducted audits on a regular basis to check that sensitive data wasn't being logged or in inadvertently stored in a database. The grey area might be for example when some data gets logged as a blob of HEX that it turns out has a password in it if you run the right decoder function on it. I've seen cases like that though show up in audits -- you see a blob of HEX or MIME and go find out what might be in it. Anyway, it really isn't accurate to say that this is a kind of <slaps forehead> <doh, why didn't we think of that> thing. It gets thought about all the time.
• bitL 8y ago
  A company that has lower acceptance rate than Harvard is "accidentally logging passwords"? Yeah, sure :D
• jammygit 8y ago
  What I'm hearing is that, because they collected it and stored the passwords as part of their reckless attempt to log every action that every human makes online in order to manipulate them, its more excusable than if it was in a specific and protected user credential database? Its only an easy mistake when you're vacuuming up everything people do.
• zeckalpha 8y ago
  What are logs but a database with a particular schema?