17 ms·
On a scale of Facebook with literally billions of users this is pretty much unexcusable. This community usually scoffs at entities, which store passwords in pl
by CaptainZapp 8y ago
On a scale of Facebook with literally billions of users this is pretty much unexcusable.
This community usually scoffs at entities, which store passwords in plain text. Why do you give Facebook a pass?
- WrtCdEvrydy 8y agoDude, you'd be surprised how much data your average company leaks in logging. Ask any HIPPA compliant company how they scrub their logs from errors that include medical data and you'll truly see how bad things are.
- kitsune_ 8y agoThis is, supposedly, one of the top 4 global tech companies. Not some regional med-tech enterprise shop with a fleet of ageing WindowsNT servers. It is literally inexcusable.
- lojack 8y agoIt happens at fortune 5 medical companies. When you sign up to Facebook you know you’re signing up to Facebook. When you go to the doctors office you are probably completely unaware that your PHI is being shared with a couple dozen other companies often intentionally and sometimes unintentionally. If you are a human in the United States there is a high probability your PHI is sitting in a log file on a server owned by some company you’ve never even heard of. Fwiw none of this is necessarily a breach of hipaa laws.
- WrtCdEvrydy 8y agoWait till you get into companies that allow doctors to have other doctors review your charts... your data gets silently shipped to a third party company who shares it with doctors... and in most cases you don't even get told it happens.
- tzs 8y agoOT: how much of a pain is it under HIPPA to deal with medical data provided by the patients themselves through channels that aren't supposed to be used for such data? I've never dealt with medical data, but do deal with credit card data so have to deal with PCI. With credit card data we run into customers who send emails, or use online chat, or leave voice mails telling us they have a new card and giving the number. We never ask for them to do this, and indeed everything we tell them says never to send us a credit card by such means, but they do it anyway. That brings all those systems into scope for PCI, which is a pain in the ass. We use helpdesk and support chat software licensed from a third party. We had to write scripts that understand its DB schema and data formats and can find and remove credit card information, and keep them up to date as the helpdesk and chat software is updated. A few years ago, the vendor tried to discontinue the stand-alone version and move its users to their cloud service version, but had to drop that plan when they found out that a lot of their customers were doing the same thing we were, and absolutely could not move to a cloud service unless that cloud service handled PCI issues. Apparently they didn't want to deal with making their cloud service handle that, and there had been no talk since then of dropping the stand alone version. Do people dealing with HIPPA run into similar issues?
- WrtCdEvrydy 8y agoYes, this is why people build in HIPPA compliant clouds like AWS... but I left that business years ago. You should always audit your third part services to see what data is sent in debug logs.
- EnFinlay 8y agoBecause there's two types of storing passwords in plain text. There's the "your password is stored in plaintext in the database" way which everyone agrees is 10 different kinds of stupid, and then there's the "we accidentally logged the body of all requests that went through this system, and it turns out login requests came through here" kind. One is a bad security decision (because you must decide how to store passwords in the DB), and the other is a very easy mistake to make which can go unnoticed for a long time (because you can be attempting to log things completely unrelated to logins). Same same but different.
- dcow 8y agoThe point perhaps is that they’re both terrible security designs and the impact to the user is the same regardless of how the “accident” happened. Nobody in FB’s position gets a pass for being irresponsible and negligent.
- polote 8y agoso yeah, dev have access to the password of your account, but really they dont need it, they probably have access to the hash in the database and login anyway. So really NO harm have been done
- p_rude 8y agoIt is neither a standard practice nor a best practice to give your devs access to your production auth databases.
- icedchai 8y agoPerhaps not in more mature organizations, but it's standard practice at every startup I've ever worked for. One place had the dev office VPN'd into production at all times.
- EnFinlay 8y agoHaving (read) access to the hash shouldn't give any access to the account. Edit: You've made this comment twice in these threads. It's wrong.
- polote 8y agoYeah I was wrong, it is only true if the dev has access to the prod database
- savanaly 8y agoWe scoff at companies that intentionally store the passwords in plain text, then compare that stored value with the credentials provided to authenticate a session. This is a common mistake for novice engineers. We are dismayed by but I wouldn't quite say scoff at accidental logging of secure data, which is a much more common accident and perpetrated by otherwise educated and advanced engineers every day.
- oldmanbythesea3 8y agoPeople scoff at companies that store passwords in plain text on purpose. Inadvertently logging them is not the same thing. And Facebook’s scale is why it’s understandable how it could happen.
- code_duck 8y agoSomeone had to write the code and then use the logs. Individuals, like any company. Then the article says it was actually searchable and used for something by employees. Why does the entire company’s scale make this excusable?
- Ajedi32 8y agoThis isn't a case of a company intentionally storing passwords in plaintext in their database. That's a big security no-no, because password hashing is such a basic and fundamental part password-based authentication that it's nearly impossible to miss if anyone involved in maintaining that system is remotely security-conscious. This situation, on the other hand, seems to be more of an unintentional capture of passwords by their logging system. That's much harder to notice and more akin to a bug or security vulnerability, which is something pretty much every system suffers from sooner or later. In such cases I'm more concerned with the company's response to the vulnerability than the fact that it exists at all. Here it seems like Facebook noticed the problem and proactively fixed it before it was exploited, which to me is a good sign.
- pfortuny 8y agoImpossible is just a couple of mistakes away. Paper holds whole worlds.
- lacey 8y agoI don’t think “pretty much”, it’s just inexcusable.
- edoceo 8y agoI had this kind of issue, a sloppy logging tool, when I had <100 customers. It was fixed before we got to 200. Early FB (c2005) could get the excuse, new team, just getting started. But, no excuse after the first round of funding
- jonnycomputer 8y agothe_duke is not "the community" so when you say, "the community usually scoffs at entities which store passwords in plaint text. Why do you give Facebook a pass" is a fallacy of composition.
- blr246 8y agoAgree this is pretty much inexcusable. Logging request or response payloads without an explicit whitelist should raise flags for any developer. There are very few cases where you can assert that not only in the present but also for all future use cases of a system, the entirety of a payload will not contain sensitive user data. Only a whitelist will suffice to maintain good security. It's common for developers to attach sensitive data for debugging and other use cases under arbitrary paths. Systems can improve further by adding patterns and other heuristics to drop values from the whitelist that look like sensitive data.
- adavis321 8y agoJust as alarming for me is that Facebook engineers don't seem to understand risk management: "In this situation what we’ve found is these passwords were inadvertently logged but that there was no actual risk that’s come from this. We want to make sure we’re reserving those steps and only force a password change in cases where there’s definitely been signs of abuse." Inadvertently logging passwords is a risk. If those logs were accessed then that's a bigger risk. Signs of abuse is an issue. There is no such thing as an "actual risk", there are just probabilities (and possible consequences). Once a consequence happens, it is no longer a risk -- then it's actual.
- dimgl 8y agoYeah I agree this is pretty much inexcusable. I'm not sure why the pass.
- ikeyany 8y agoFacebook didn't mean to log failed sign-in information. It just kind of happened.
- ivanhoe 8y agoBecause storing plain-text passwords in DB is a sign of pure incompetence, while accidentally logging a password is a huge screw up, but unintentional one - and frankly a type of mistake that is not unimaginable to happen to any of us. You need to log a full requests to urgently debug some issue, and in all this mess you forget to remove the logger afterwards... and boom, you've got yourself a log full of passwords, credit cards, and all other kinds of secrets. In real world it's just like that, humans make errors, that's why your logs should always be heavily protected and regularly audited.
- cbhl 8y agoI would argue that an organization the scale of Facebook is the most likely place for this mistake to happen. Log in and sign up pages -- especially on mobile web -- are being constantly iterated on by "growth" and "emerging markets" teams whose first priority is getting graphs to go up and to the right, not making sure the pages are secure. Their entire mission is to make things work with yesterday's technology (feature phones, old Android tablets, and so forth). On the other hand, the dedicated security folks are focusing on ensuring there are multiple factors protecting high-profile targets, so they're working on SMS as a second factor, Login Codes, Security Keys, and primarily focused on Desktop Web, iOS, and Android. Or they're doing fuzzing, or building robots to chase down engineers for buffer overflows and the like.
- EGreg 8y agoI don’t get it!!! Facebook is supposed to be hiring the best developers. It has tons of money. This is the most basic security consideration - hashing passwords on the client at least. At LEAST. I am constantly surprised by how basic practices were ignored by these corporations that got so big, while the small guys implement them. I guess people really were "dumb fucks" to trust Zuckerberg with their passwords.