11 ms·
I've worked in healthcare / biotech for more than a decade and I can promise you that the FDA would see no difference between the two types of gaffes. As custod
by EpicEng 8y ago
I've worked in healthcare / biotech for more than a decade and I can promise you that the FDA would see no difference between the two types of gaffes. As custodians of sensitive information it was our responsibility to ensure that said information didn't leak, period.
I don't know anything about FB's infrastructure, but when I was lead I would have viewed leaks in logs as far worse than something in the DB because our DB's were harder to gain access to.
I get what you're saying, but it's irrelevant. Easy to screw up, hard to screw up, doesn't matter; just don't screw up because the result is the same. This stuff is security 101. If you're logging requests then you need to ensure they don't contain sensitive info.
- toomuchtodo 8y agoThis is how professional software engineering and infrastructure implementation is done, and I appreciate you pointing it out so forcefully.
- arkitaip 8y agoCorollary: Most web development simply lack the rigor of engineering.
- toomuchtodo 8y agoMost software engineers aren’t. Engineering requires licensing and exams through a governing body. Can’t have the prestige of the title without the responsibility. Software developers want to have their cake and eat it too (power and respect with no oversight and governance). Pile on the regulation (GDPR for starts, more PII protection to follow up, extremely painful fines for failures).
- EnFinlay 8y agoI couldn't agree more.
- jjeaff 8y agoIn the US, you only need certifications and testing for a "professional engineer". Most electrical, chemical, mechanical, or civil engineers are not certified PEs. Yet I don't think anyone is claiming they aren't engineers.
- tsbertalan 8y agoI have a PhD in chemical engineering, but not a PE, since I didn't go the industry route. I wouldn't feel comfortable calling myself an Engineer without it, and that was the attitude expressed by my professors and cohort coming out of undergrad.
- mfoy_ 8y agoIt's my opinion that the title "Engineer" should be on par with things like "Doctor" or "Lawyer". It should require certification and there should be consequences for claiming to be one if you aren't.
- mcbits 8y agoI don't think anyone would get in trouble for job titles like "computer doctor" or "network protocol lawyer". Practicing medicine or practicing law (or in some cases, practicing as an engineer) without the required qualifications, regardless of job title ... different story.
- gridlockd 8y ago> Most software engineers aren’t. "Software Engineer" is just a job title. > Engineering requires licensing and exams through a governing body. "Software Engineering" doesn't, it's just a job title. > Can’t have the prestige of the title without the responsibility. There is no real-world prestige associated with having "Software Engineer" as a job title (as opposed to "Software Developer"). > Software developers want to have their cake and eat it too (power and respect with no oversight and governance). "Power and respect" is not actually something that is commonly associated with a guy sitting at a desk for a salary. > Pile on the regulation (GDPR for starts, more PII protection to follow up, extremely painful fine for failures). You just learned that Facebook stored passwords in plaintext for years and nothing bad happened. In fact, it's hard to conceive that anything particularly disastrous could have happened. Yet, you call for more regulation that will drive up costs for everything. Whole industries could become unprofitable and those high-paid "Software Engineers" would become redundant. I have the opposite opinion: The fact that computer security is generally poor teaches people how to properly use computers. They need to assume that there are no secrets, because that is the plain reality. No amount of regulation will make computers secure. No certification or best-practice encryption scheme will prevent users from writing their password on post-it notes, or from opening "sexy.jpg.exe", or from answering that call from the "Microsoft Service Center".
- darkpuma 8y ago> "Software Engineer" is just a job title. And like the job title "sandwich engineer", it should probably be derided. Apologies to anybody working in the honorable trade of sandwich crafting. I love sandwiches and appreciate your work. They bring me much more enjoyment than software.
- hombre_fatal 8y agoSame with electrical, chemical, mechanical, and civil engineers since almost none of them are PEs? Seems a bit silly.
- randomdata 8y agoI agree. The job title is literally _engine_er. Unless you oversee the operation of engines (as in locomotives), you are not an engineer. But seriously, the definition of engineer in regular lexicon is someone who designs, builds, or maintains complex systems. I struggle to think how software does not tick all of those boxes. Sandwiches are a different story. Perhaps you are confusing engineer with professional engineer? I can see how it might be easy to mix them up if you are not paying attention, but professional engineer carries a different meaning. Most software engineers are not professional engineers but are unquestionably engineers.
- vonmoltke 8y ago> Engineering requires licensing and exams through a governing body. Offering engineering services to the public requires licensure. Working at places like Boeing, General Motors, Texas Instruments, and Caterpillar does not (for the vast majority of positions); It just requires an accredited engineering degree. Note that this is independent of the software industry needing more engineering rigor, which it desperately does.
- balls187 8y ago> Software developers want to have their cake and eat it too (power and respect with no oversight and governance). This doesn't come from software developers. This comes from the SV culture of "move fast and break things."
- toomuchtodo 8y agoI painted too broad strokes. I agree, and admit my mistake.
- mfoy_ 8y agoJust replace "software developer" with "self-proclaimed software engineer" in your comment and you'd be spot-on.
- balls187 8y agoNothing about engineering really covers security best principles and practices. I went through classical engineering school, and I'd summarize with three categories: * Newtonian Physics & The Mathematics Behind It (i.e. Differential Equations) * Ethics * Technical Communications
- jammygit 8y agoI think that was actually invented at Facebook in the early days, unless I'm mistaken. Its a good mantra unless its security related
- mfoy_ 8y agoA lot of HNers probably have the job title "software engineer" without any of the qualifications of an "engineer"... that's probably why you're being down-voted. But you're not wrong.
- wlesieutre 8y agoI don't think anyone's arguing that it's ok, just that it's less "easy to screw up" vs "hard to screw up" and more "did something really stupid on purpose" and "did something really stupid by accident." But yes, either way the end result is the same and you shouldn't do it.
- dumbfounder 8y agoYes but how many people looked at these logs, saw the passwords, and said nothing? If it was logged presumably the logs were viewed from time to time. The mistake is easy to make, but it's also easy to correct if there is a culture to report such mistakes.
- wlesieutre 8y agoThat's true. Kerbs article says: > My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
- meitme 8y agoThose numbers are pretty meaningless without some concept of what queries they were making. When I was working with large logs I'd regularly pull huge amounts of logs to only look at maybe a handful of lines. Even then often only scanning for the things I'm specifically working on. Not saying this is what happened just saying those 2,000 employees almost definitely were not doing careful analysis on the complete result of their 9 million queries(And shouldn't have!).
- pnutjam 8y agoThis is exactly how Zuck was able to steal login information in his stories about founding facebook. I find it hard to believe this is accidental.
- kerng 8y agoDo you have a reference? I was wondering the same but have no reference.
- freeflight 8y agohttps://www.businessinsider.com/henry-blodget-okay-but-youve-got-to-admit-the-way-mark-zuckerberg-hacked-into-those-email-accounts-was-pretty-darn-cool-2010-3?IR=T https://www.businessinsider.com/henry-blodget-okay-but-youve...
- arrty88 8y agoYep i remember this story. So clever and yet so dirty.
- human 8y agoThis is borderline illegal, if not simply illegal. What if he entered into Paypal accounts or bank accounts? I don't want a world where if I enter the wrong password, I have companies pouring over my other accounts trying to breach them. Let alone if I reuse a password by accident.
- ArtDev 8y agoWow! That does not look accidental at all!
- oldmanbythesea3 8y ago> I don't know anything about FB's infrastructure This explains a lot of the comments here. Facebook’s scale is not like anything most engineers have worked on. Facebook probably has logs in the 100s of terabytes. Ensuring that sensitive data isn’t logged takes more than some occasional greps.
- EpicEng 8y agoMy point is that it's irrelevant. Have more data? Need more auditing. Any system which touches sensitive data is subject to security review. Yes, FB systems are massive. They should have massive oversight as well. You may as well be defending a nuclear meltdown because desiging nuclear power plants is hard. >Ensuring that sensitive data isn’t logged takes more than some occasional greps. Right; it requires investment into process, requirements, testing, and oversight. Most importantly, it requires a company wide, top down mentality that your customer's privacy and protection is more important than your margins. If you can't (won't) dedicate the resources required to ensure your customers data is protected then you have no business operating at such a scale.
- ryandrake 8y agoExactly. Can every piece of data logged be tied back to a legitimate business purpose? What’s needed here is a mentality change: These logs should be thought of as liabilities rather than assets. You should log only what you need, while you need it, and then turn off the log when you’re done. If your mentality is “log everything, always, because maybe we’ll need it later” then these privacy and security trash fires should be expected.
- viraptor 8y agoLog everything: you don't care about privacy. Treat logs as liabilities: why can't you solve the issue I experienced yesterday? You can't win. Either you log more than you think you need right now or you can't do engineering investigations on past data. You're going to end up somewhere in the middle realistically.
- 8y ago
- viraptor 8y agoThe problem is, you can't ensure that. You can do best effort filtering/scanning only. Let's say you have an API and someone writes a client with a memory corruption bug which results in password and some other field being sent swapped 1 in 10k times. Now you have password logged. You maybe can't even tell exactly where and it's close to impossible to detect automatically. "just don't screw up" is not possible in this case and the chance you'll learn about this soon is really small. Or if you think a client bug is a stretch: If you're running a large enough website which logs usernames on failed logins, I bet you have at least one password or a concatenated usernamepassword in your logs. Just because someone wasn't paying attention to the text box focus.
- EpicEng 8y agoYou can never ensure anything completely in engineering. Perfection is never the goal, but you can do a much better job than FB has here. You're talking in hypotheticals. Nowhere did I say "nothing ever goes wrong if you know what you're doing". What about the issue at hand? What's your opinion on passwords being logged on every request for years?
- viraptor 8y ago> This stuff is security 101. If you're logging requests then you need to ensure they don't contain sensitive info. If what you meant by that is "best effort" then sure. The issue at hand is only obvious now that we're reading about it. It could easily be missed depending on how their infrastructure works.
- EpicEng 8y ago>The issue at hand is only obvious now that we're reading about it No, it's not. It's completely obvious to anyone who has ever worked on a system which deals with sensitive information. It can only "easily be missed" if you're not auditing your code and logs. This stuff is basic to anyone who works in industries where the data is a liability.
- btown 8y ago
- deleted 8y ago[deleted]
- tptacek 8y agoI work fairly extensively in health care technology and have since ~2005 and am not aware of the particular regulation that would cause the "FDA" to react to a password storage security gap. Could you be more specific about the regulations you're referring to? Unsafe logging of PHI would be explicitly problematic, but passwords aren't PHI.
- EpicEng 8y agoI never said the FDA was specifically interested in passwords. The FDA was brought up in reference to my work with PHI. The FDA does care about how you secure access to sensitive data however. If you leak passwords which allow unauthorized access to PHI, you're going to have a problem. The analogy was intended to focus on practices which should be employed by companies which handle sensitive data, not the specifics of what the FDA is looking for.
- tptacek 8y agoThese aren't "leaked" passwords; they're logged in plaintext internally, which is not great, but is not the same thing as having leaked them.
- EpicEng 8y ago"leaked" in the sense that they're somewhere they shouldn't be. I could have chosen a better word. if I were to store database passwords on a file share the entire company had access to we would consider that a "leak" even though they weren't exposed externally.
- tptacek 8y agoHIPAA/HITECH would not necessarily; the analysis would have to include compensating controls and who had access. The fact pattern in this Facebook case would probably not support a violation. Again: crappy stored passwords aren't PHI.
- 8y ago
- smrtinsert 8y agoWorked in two data sensitive industries as well. There is no excuse, completely agree. This is amateur and completely indicative of a broken dev process