6 ms·
> 3) Storing said passwords and others in plaintext. [2] in logs.
by wnevets 8y ago
> 3) Storing said passwords and others in plaintext. [2]
in logs.
- munk-a 8y agoIf you're a small business - I get it, it's good to whitelist whatever information you're logging explicitly, but for smaller teams a hard to diagnose issue might lead the team to "log everything so we can sort it out later". Facebook is Facebook, whether this decision was the product of the corporation as a whole, a small dev team, or a highly paid consultant/third party, Facebook is a big enough company that they don't have the "I didn't realize..." excuse anymore.
- fuzz4lyfe 8y agoSeems like a pretty trivial automated test for so many PHD's to miss: create_user('Bob', 'BobPassword123') assert "BobPassword123" not in logfile
- munk-a 8y agoHrm, so you checked the apache access logs, or maybe an error log, what about the system logs? Does the login request spin up a bash script and pass the password to it? I don't think it's trivial to guarantee non-existence.
- jlarocco 8y ago> I don't think it's trivial to guarantee non-existence. I disagree in this case. Log messages don't spontaneously appear in arbitrary places. If the developers understand what their software is doing and how their systems are configured then they should know where to check for the logging messages.
- noselasd 8y agoUntil someone turned on logging of the full request body on the load balancer/proxy or otherwise unknown-to-you middlebox that was the TLS termination point in production that you did not know about
- fuzz4lyfe 8y agoSo their incompetence lies in lacking configuration management in production as opposed to a lack of testing? Why are they manually configuring production servers anyway at that scale?
- munk-a 8y agoThey are managing a lot of things, they have full responsibility to be aware of these things as a company but I'd guess that with the scale of ops Facebook has regular developers never interact with the team that's managing their attempt at high availability by scattering servers across different hosts, data centers, legal jurisdictions etc. in fact I'd imagine at a op the size of Facebook that sort of problem has a dedicated department. That said, someone should have been watching for this stuff and failed to do so (or to exist), so I'm not excusing them - but this is not a trivial thing to protect against.
- stickfigure 8y agoYou speak with the hubris of a man that's never been in the arena.
- fuzz4lyfe 8y agoFacebook hires some of the top software developers and engineers on the planet, if not leaking plaintext passwords is too high of an expectation for them then nothing that isn't public knowledge should ever be put into any computer system. As a profession we should demand our peers do better than this.
- munk-a 8y agoWe're all people of varying skills, I would never assume an innate high bar for any activity a human does - only by requiring the bar be maintained at a level and regularly checking and enforcing that requirement can we be reasonably sure it is - and this isn't just `echo $password` the way these passwords got into the log file is (from what I've been able to discern) pretty obscure and round about - Facebook is absolutely responsible and needs to be held to account, but the mistake is understandable.
- heavenlyblue 8y agoRun the deployment in docker, check every file changed in the container for it’s contents.
- wavefunction 8y agoyeah well we're not running apache >wut is this post?
- kortilla 8y agoSeems like a useless test considering how many actual production systems don’t log to local disk.
- nkozyra 8y agoAh yes, the single, non-distributed, file-backed log Facebook uses.
- fuzz4lyfe 8y agoYou have no problem with the create_user function? Obviously it's pseudocode my point is there are a finite number of log locations and checking for an instance of a known string among them isn't difficult.
- UncleMeat 8y agoThere are a finite number of log locations now. How do you intend to operate an integration test across the entirety of Facebook's stack to detect new log locations?
- fuzz4lyfe 8y agoChange requests and a change review board staffed by professionals. Subject: Change Request Body: I would like to log the body of authentication requests in production. Subject Re: Change Request Body: How will you ensure personal data is not stored that shouldn't be? Subject: Re: Re: Change Request Body: I will add configuration Y to logging system X.
- hw 8y agoBureaucracy like this is what kills teams and products. There's no guarantee it works, and every change or commit should already have privacy and data protection in mind anyway. You also don't know what you don't know - you could for all your knowledge think that nothing is being logged, when in fact there's a subsystem outside your scope that's actually doing the logging. You assert that it's trivial, yet you're adding more layers to protect against something like that from happening. It's the naivety that all problems are trivial is what gets people and companies into trouble in the first place
- 8y ago
- gnulinux 8y agoWhat if it logs "BobPassw[...]"? Is that more acceptable? What if there is a bug and some other function logs "[...]word123"?
- skybrian 8y agoIf it turns out to be that simple to write a test, it probably didn't happen by accident. Many systems are more difficult to write tests for. Of course we (and they) should do it anyway, but it does often take an investment in making things testable.
- beat 8y agoI've worked for multiple Fortune 25 companies, and that excuse does not fly. Not in banking or healthcare, where breaches of privacy/confidentiality are actually illegal, rather than merely distasteful. Small teams and careless devs doing that sort of bad logging will be caught and corrected by strict security oversight. This is the sort of thing that leads the HN crowd to sneer at the old, slow ways of the enterprise world.
- deleted 8y ago[deleted]
- gjs278 8y agohey just so you know your https ssl certificate is broken for congruence.io so much for how careful you are with computers. it’s telling you worked for multiple fortune 25’s.
- Aunche 8y agoThe passwords in plaintext wasn't a breach or a leak. If you punish Facebook for it, they would be less inclined to share such information in the future.
- tkxxx7 8y agoThat's how you handle children, though. No one is going to ignore wrongdoings in hopes the perp will continue to play nicely.
- munk-a 8y agoIf we can't trust a company to audit themselves fairly (in fact, we can't generally) then they should bear the burden of paying for an external audit of their processes, many "boring" industries have these sorts of requirements.
- jyrkesh 8y agoThere might not have been a large-scale public data dump of passwords, but if 20,000[1] employees had access to logs with plaintext passwords, there is no guarantee I could ever accept that zero of them read or used customer passwords for personal purposes. What's to stop a malicious ex-lover from grabbing a FB password and reading that person's private messages? If FB didn't even know there were passwords in plaintext, they very likely weren't auditing log access as much as was needed. [1] https://www.theverge.com/2019/3/21/18275837/facebook-plain-text-password-storage-hundreds-millions-users https://www.theverge.com/2019/3/21/18275837/facebook-plain-t...
- wavefunction 8y agoI'm on a small team (5 engineers) that is GDPR compliant so no PII is logged. It's not hard if you care.
- UncleMeat 8y agoIts a lot easier on a small team. Security isn't a stable state.
- wavefunction 8y agoI'm replying to the parent who made the opposite assertion I can't f'in believe I have to explicate such simple things to supposedly intelligent and thoughtful people
- organsnyder 8y agoGreat—so potentially subject to even less security/privacy controls compared to other places they could've stored it.
- taurath 8y agoWhich is the NUMBER ONE MOST BASIC rule of handling users confidential data. It reflects incredibly poorly on the engineering practices of Facebook that this managed to get through. It should be a criminal liability.
- wnevets 8y agoI'm not arguing this wasn't a bad thing but comment made it sound like they were storing plaintext passwords in the database circa 2002.
- taurath 8y agoStoring them in logs is the same thing IMO. Logs are usually more easily accessible than a user database.
- wnevets 8y agoGonna have to disagree. I think it would be way worse if we found out they were storing passwords were plaintext in the database in 2019. Even if the security implications are the same/worse, the policy/decision making of such a revelation would be beyond terrible. edit: To put it another way, remote code execution flaws are terrible but they can happen. However it would be way worse if someone put in a static username/password backdoor. The security outcome may be the same but one is beyond terrible policy/decision making.
- taurath 8y agoAs is the policy/decision making that results in logs having the passwords - you can still have very locked down database access. Logs tend to get spread around all sorts of systems, and access control for logs is almost ALWAYS lower than DB keys. They're also cached for search-ability on any number of elasticsearch or business intelligence platforms, so getting rid of them after the fact is even harder. At the very least its an equivalent problem.
- whateveracct 8y agoeven worse
- TallGuyShort 8y agoThat's still a pretty elementary error for a company that gets off on using CS-y riddles in interviews like Google used to. Move fast and break things, and then get a $5B fine.
- OrgNet 8y ago> in logs. ... for seven years.
- chopin 8y agoSo, it's not in plaintext if it is in logs? Storing passwords in logs is even worse than on a database since access more likely is less restrictive. Every programmer worth its salt knows about this problem (I surely wrote code to prevent this). Sorry, but it is like saying "there is no SQL injection, only bad input validation".
- golergka 8y agoThe first thing I did after setting up Jenkins for our small dev shop was to find and install and test a plugin that removes all kinds of credentials from the logs. And I'm just a developer, and I never claimed nor think that I have any devops or administration qualifications - just common sense. If Facebook devops engineers, who are probably among the best trained and highest paid on the planet, lack the same common sense for their users passwords... I can't even come up with an ending to this sentence that would properly express my emotions right now.