21 ms·

Facebook Stored Hundreds of Millions of User Passwords in Plain Text for Years

• icpmacdo 8y ago
  Zuck should resign
  • CaptainZapp 8y ago
    He controls 60% of the vote, so he probably wont. But wait a few days, when he crawls out of his hidy hole, to ensure us, once again, that this mistake is totally on him and to promise betterment. On a less sarcastic note: It troubles me that their new "privacy mindedness" is taken at face value in some of the "serious" press. That's despite Facebook's history of outright lies.
    • mrosett 8y ago
      I'm skeptical that they can turn their culture around, but I do think they do see damage to their reputation as an existential threat (either via users leaving the platform or via regulatory pressure.)
  • EnFinlay 8y ago
    Because some developer typed: > log.info(req.body) and no one realized that login data would get captured? Get real.
    • Bhilai 8y ago
      Smart companies typically overload loggers to do tokenization over many types of requests automatically to avoid this kind of inadvertent exposure.
  • snaky 8y ago
    Why? Revenue is growing.
• ravenstine 8y ago
  We're sorry.
  • jayess 8y ago
    We take security very seriously.
    • CaptainZapp 8y ago
      Nah; That's Equifax!
    • snaky 8y ago
      > There is nothing more important to us than protecting people’s information, and we will continue making improvements as part of our ongoing security efforts at Facebook," Pedro Canahuati, the company's vice president for engineering security and privacy, wrote in the post.
• the_duke 8y ago
  (Edit: reworded a bit to make it clear I don't think this is acceptable) Sounds a lot like some service was logging the full body of a signup/login request, which then was readable for anyone with access to the logging/tracing infrastructure. Dumb mistake but it's not hard to imagine this happening, considering that FB probably has a bunch of services involved in the login/signup flow to prevent bots/spam, abuse, etc. Not to imply this is acceptable, especially at a IT company like FB with vast resources and know how. Raw passwords are an especially big screw-up. There are a lot of failures here, from actually logging something so sensitive over giving access to so many employees to not noticing this for years. (Assuming this was actually log data). BUT if we are honest, anonymizing log data is rarely a priority. Even if it is, leaking sensitive data can happen easily in a lot of different points in the infrastructure. In actual application code, client + server exception tracing (just imagine a deserialization exception which contains part of the input) , web server, load balancer, proxies, service mesh... There is a lot of interesting stuff hiding in the logs at pretty much every company. This is a good time to look in the mirror and audit your logging and tracing data. Unless you are in a highly regulated field like finance/healthcare or there is a strong company-wide culture for security/privacy with regular audits already, I can almost guarantee you will find at least one data point that should not be where it is. Protecting sensitive data needs to be a big consideration for every dev, ops and especially management, which has to allocate enough time for security reviews and audits.
  • good_guy 8y ago
    LOL, way to defend a crappy design.
  • CaptainZapp 8y ago
    On a scale of Facebook with literally billions of users this is pretty much unexcusable. This community usually scoffs at entities, which store passwords in plain text. Why do you give Facebook a pass?
    • WrtCdEvrydy 8y ago
      Dude, you'd be surprised how much data your average company leaks in logging. Ask any HIPPA compliant company how they scrub their logs from errors that include medical data and you'll truly see how bad things are.
• deleted 8y ago
  [deleted]
• taneq 8y ago
  Move fast and break things, am I right?
• krisrm 8y ago
  Posted this on the other thread from Facebook, but at what point do we start imposing strict fines on companies that are found to have done this? Granted, I guess we wouldn't be hearing about this instance at all if there was to be some sort of fine attached - it would have just been swept under the rug - so maybe that's not a good idea. I'm just tired of the "oops we stored your passwords in plaintext lol" from companies with engineers that should clearly know better.
  • Angostura 8y ago
    It's likely to have been a breach of GDPR, so if this situation had existed when GDPR was in force, the answer to your question would be "at this point".
    • krisrm 8y ago
      I guess that's true, yeah. Will be interesting to see if there are fines from the EU.
      • jeltz 8y ago
        It all depends on if one can prove negligence.
  • segmondy 8y ago
    When we can start fining you for your mistakes, when developers can start getting fired for any mistakes immediately. I don't care about Facebook, but storing user passwords is the plain is not "privacy violation" without the password they still have access to all your data. storing user passwords by logging it is stupid amateur security mistake. I can understand if Facebook stored the password and used it to access your other accounts with permissions to invite users then sure fine em. But mistakes are mistakes, they owned up to it.
    • krisrm 8y ago
      Storing passwords in plain text is beyond a simple slap-on-the-wrist mistake, and it has real security implications.
• dimtion 8y ago
  This seems like the passwords were inadvertently written into some log files. While this is a very bad security issue for a company like Facebook, I am pretty sure that this type of bug is much more prevalent in the industry than we would like to assume.
  • EnFinlay 8y ago
    It's just such an easy mistake to make. Some new ops guys logs all the traffic through a subsystem and boom - plaintext passwords.
    • Stuckinsofa 8y ago
      Yeps. And at Facebook-scale, when you realize the issue after it has went live you have already "stored hundreds of millions of user passwords"
      • kerng 8y ago
        That's a good point in favor of less hacking things together and more engineering - Facebooks seems to have not embraced strict engineering processes which are the only way to tackle these things. Facebook has to change it's own principles to succeed long term.
        • EnFinlay 8y ago
          I would say that this needs to change on a broader scale within the software development community than just with Facebook.
    • LeifCarrotson 8y ago
      Why does traffic going through the subsystem contain plaintext passwords? It was my understanding yhat my password, on a properly configured login page, never left my browser, much less crossed multiple machines that had the ability to read it.
      • 8y ago
• polote 8y ago
  Why is that an issue ? devs will also likely have access to the hash of the password of a user, and then access his account. We dont care if passwords are leaking, this is your issue to not use the same password on all websites. Password are not really a private thing for developers inside a company
  • EnFinlay 8y ago
    It's an issue for the same reason storing passwords in plaintext in the database is an issue. Reading the hash != access to the account. Passwords should be private to everyone except the account owner. If devs have access to all user passwords then you have a fucking issue.
    • icedchai 8y ago
      On many systems, "real passwords" are just a tcpdump away (most sites aren't doing end-to-end TLS. It's TLS terminated at the load balancer / proxy level, everything else is in the clear.)
      • EnFinlay 8y ago
        Also true.
      • jeltz 8y ago
        That is a much smaller issues than passwords stored in logs, because the people who have access to running tcpdump usually also have the ability to intercept traffic in other ways (e.g. by updating the application to log all traffic or by inspecting RAM). On the other hands access to logs is usually handed out to a much bigger group of people.
      • the_duke 8y ago
        True, but thankfully service meshes like Istio are increasingly used which encrypt the traffic between each service.
  • okmokmz 8y ago
• Ajedi32 8y ago
  Yet another example of why its important to use _unique_ passwords for every site you have an account on. Even if the site you're using does password storage properly, that's no guarantee plaintext credentials couldn't leak through other means, such as, in this case, improperly configured logging systems. In the future, WebAuthn may be able to solve this problem for good, as sites will only have access to a unique public key rather than a plaintext password. Until then, a password manager is your best defense against this type of issue.
  • xtracto 8y ago
    This is so prevalent in technology companies that it is funny to read this thread with everybody throwing mud at Facebook without considering that most probably the company they are working on has had (or maybe even currently has) the same issue.
    • nabakin 8y ago
      That's a fallacy my dude. You can throw shade at your own company and Facebook for the same reason. It doesn't and shouldn't minimize what Facebook has done.
• pizza 8y ago
  > My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords. This imo is the truly alarming takeaway. FB employees were retrieving user passwords? Around two thousand FB employees? How in God's name is Zuckerberg going to perform his usual performative contrition about that one? I'm just trying to imagine the data structures that were being retrieved from databases. Either they stored something like a big user account data type that contained their password in plaintext, which imo is a really weird design choice, or logs for other services were being mixed in with logs leaking the user/pass combos. Surely one of the engineers could have noticed and said 'wait a minute... those are logins' over the course of the years? We hear all the time that people want FB to follow a responsible social practices (the debate on what those are rages on, which is great imo), but can't FB at least wrangle its own code base? On the other hand, we shouldn't take the stance that heads should roll, imo - it would just create a chilling effect that would deter other companies from ever going public about their own security mishaps. edit: I should probably tone it down in this comment but I'll leave it for posterity
  • rhacker 8y ago
    This should be the main takeaway. If true, then 2000 people decided not to raise the alarm. And 9,000,000 queries with results that had passwords. This completely invalidates the other thread of people discussing giving facebook a pass because accidental logging could happen to any company.
    • weka 8y ago
      How much do you want to bet that some of those FB employees decided to login by turning off "location detection" flag and in an incognito browser? I would not be surprised. Some of those 9,000,000+ queries and 2,000+ employees must have had some nefarious use. Statistically speaking...
      • kitsune_ 8y ago
        Judging by the maturity level of the discourse on teamblind this precisely what happened.
• jdlyga 8y ago
  My facebook feed has really become stale in the past few years. It used to be a good way to see what your friends are up to, and see pictures. But now it's ads, news stories, posts from my old university, bad memes, and that's about it. Even instagram has become full of influencers, models, brands, etc. I just want a platform to stay in touch with friends and that's it.
• hammock 8y ago
  Reminder that Facebook also stores(stored?) 3 separate hashes for your password: passwoRd, PASSWOrD, and PasswoRd. They weren't super transparent about this fact. Source: https://www.zdnet.com/article/facebook-passwords-are-not-case-sensitive-update/ https://www.zdnet.com/article/facebook-passwords-are-not-cas...
  • i_cant_speel 8y ago
    I don't really see a reason they needed to be super transparent about that fact. It doesn't really have a significant security impact. It's just a small QOL improvement they implemented.
  • g45y45 8y ago
    You don't need to store 3 hashes, infact, you would need many hashes for all possible cases. It is easier to drop all cases pre hashing, and only store a lower case hashed value.
    • dymk 8y ago
      Hashing the the lowercase version of the password string is not the same as what they’re doing. What you’re suggesting greatly reduces the security of the password, what they’re doing only divides the search space by 3 (which is nothing).
  • anticensor 8y ago
    Good to know. Someone in Facebook probably writes all his emails in all caps.
  • crazygringo 8y ago
    Honestly, given the possibility of accidental caps lock, and how mobile keyboards try to capitalize the first letter... ...that seems like a clever feature, and it doesn't reduce security in any meaningful way at all. I would never bother to program that, out of my own laziness, but I respect that they did. It's really thinking about users.
• scriptkiddy 8y ago
  How do companies keep messing this up. The cardinal rule of web application security is to NEVER store plain text passwords anywhere. The only time your application should have access to plain text passwords is when it is hashing the password or verifying the password against a hash. If you need to log all request data for some reason, strip the passwords out. It really isn't difficult.
  • qpiox 8y ago
    In properly built software, the clear text password never leaves the client (browser in this case). There is no real need to have the password sent over the net.
• Bhilai 8y ago
  So they logged plaintext passwords, what a dumb mistake. To top it all it sounds like 20K employees had access to these infra components. I have had my doubts on FB's internal access control model for a while now. Specially after one particular story where an employee claimed to have internal access to private information and used that to stalk people on Tinder - https://www.wsj.com/articles/facebook-fires-employee-who-bragged-on-tinder-about-his-access-to-user-data-1525294488 https://www.wsj.com/articles/facebook-fires-employee-who-bra...
• duncans 8y ago
  Twitter found they did the same thing last year https://news.ycombinator.com/item?id=16989534 https://news.ycombinator.com/item?id=16989534
  • tqi 8y ago
    Same for Github: https://news.ycombinator.com/item?id=16974851 https://news.ycombinator.com/item?id=16974851 1 comment, "Crazy level of transparency, they noticed, fixed it and notified the users even though this was an internal leak. I bet most businesses would just sweep it under the rug."
• arduinomancer 8y ago
  To me it’s no surprise these things happen considering these companys only care about hiring leetcode experts. Who knew data structure and algorithms doesn’t qualify you to build a secure real world web application? Knowledge of practical things like security has zero value to these companies when hiring engineers.
  • tschellenbach 8y ago
    Well if you hire enough people, and you're not 100% successful at selecting the good ones... Also even good people make mistakes at times. Almost impossible to prevent. The most actionable thing you can do is build less things, and focus on using of the shelve micro services (either in-house or third party) for everything
    • therealdrag0 8y ago
      But as someone else said, even though it maybe a common mistake (I've seen similar mistakes at my work), how is it that a number of those thousands of engineers didn't raise the alarm and get traction sooner on securing it? Companies need to incentivize engineers to not have a "not my problem" attitude.
• terracatta 8y ago
  I think the technical security problem here (properly whitelisting parameters in logs) is just a symptom and not the core underlying concern (as the article mentions Twitter and Github just dealt with similar issues) To me, it seems very likely someone before 2019 laid eyes on these logs and either: a) Decided not to report it (implies serious security culture issue) b) Reported it and no action was taken (implies serious security process issue) c) Didn't even acknowledge it was inappropriate (implies a serious security training failure) If you've already become complicit towards regularly violating the privacy of your end-users, one can easily understand an employee devaluing the seriousness of clear-text passwords in a log. Are FB employees so regularly exposed to sensitive data that they have become desensitized to the seriousness of clear-text passwords in an internally accessible log?
  • Bhilai 8y ago
    This is very basic stuff, all user-identifying information should be tokenized before being logged. Controlling access to production login logs or exposing them on only a need to know basis is another basic security principle. Sounds like FB is actually a wild wild west internally.
    • sydd 8y ago
      What is now a common practice was nonexistent a few years ago. 20 years ago you could bypass Windows login with a few clicks. 15 years ago CORS was nonexistent. 15 years ago it was common to send sensitive data unencrypted..and so on. Ex Facebook people told me that until around 2010 FB management turned a blind eye on its employees digging around their databases. Then they released a warning that people should stop and started locking prod data down. A few months later those who still peeked around prod data were let go.
• MagicPropmaker 8y ago
  With 20,000 employees who potentially had access, there's no way of knowing if any password was abused. They should message their users to change their password, especially if they use it on other sites, just to be sure. And because many sites use Facebook OAuth, there's a lot of room for abuse.
  • dexterdog 8y ago
    With 20,000 employees who potentially had access, we can rest assured that passwords were abused.
    • MagicPropmaker 8y ago
      Exactly! It's nearly certain that at least one of them had an enemy, ex, etc, that he/she wanted to exploit.
      • dexterdog 8y ago
        Or just a person saving the data for a rainy day
• Zenst 8y ago
  Sadly many companies did in their early days from that period (even when it was known to be bad). Blackberry had plaintext passwords stored in a postgres dB up until 2.0 of BES (circa 2007). More so as no constraints (yes many many people had a password of "pass","god" and the like). So let us not forget legacy systems we have all encountered or read nostalgic stories about how old XYZ kit is still in use today. Or to put it another way - the amount of work that went into Y2k, just to fix a date field, fixing a plaintext password is more work and we have had no password2k like event and any event is local with some company falling foul and by that - getting hacked and data shared publicly. So even today, you can guarantee that their are still systems out there using plaintext password storage. Some will be legacy hangover reasons, but not all. But it sure does make you think, how many legacy systems are still in use today and by legacy, they don't even have to be that old in some fields or work. Sure does make you wonder about satellite security given the age of some of those still operational and in orbit.
• Shivetya 8y ago
  Oh the fun that is scanning source repositories, pretty much a guarantee at any large company you are going to find plain text passwords, as in hard coded. as for more fun in plain text passwords it is no worse than any system which permits an admin to extract the user's current password and those systems still exist
• deleted 8y ago
  [deleted]
• 0xmohit 8y ago
  Did someone say that it happened because Facebook believes in transparency?
• AJRF 8y ago
  Facebook posted a response with the title "How we secure passwords". The euphemism of company press releases always make me laugh.
• tschellenbach 8y ago
  If it happens to a company with the best engineers, focus on security and almost endless budget...
  • qpiox 8y ago
    No, this did not happen by mistake. It is plain and simple that they don't have focus on security. There is no true need for the cleartext password to leave the users browser.
    • dymk 8y ago
      So if it's not by mistake, do you think they're intentionally trying to reduce security? Are you saying that every website that hashes on the server (almost 100% of the internet) isn't just negligent, but consciously sabotaging user security?
• deleted 8y ago
  [deleted]
• nedwards72756 8y ago
  What I find more in more in life is that those that get ahead are doing so by not doing things the right way. We have such a large population now that it seems those at the top have got there because they have done something wrong in order t get there.
• sebringj 8y ago
  Personally as a developer I'm not thinking Facebook is bad but rather that in general we are bad and we pass over plain text passwords to endpoints (through SSL) and just trust the data will not be abused (logging) but would be cool to have client-side encryption prior to sending to any 3rd party so figuring out a way to do that cleanly on different devices where a password, although the same everywhere possibly but is always encrypted uniquely for a particular company prior to going over the wire so even if compromised internally would only be for that particular service. This is probably just a legacy problem in the end where a new approach would simply make this problem obsolete like a yubikey or built in protocol at some point.
  • sebringj 8y ago
    All the answers I see on stack say there is no gain to do hash or encrypt prior to sending. I'm not sure I can agree with this. Let's say you hash a password and some attacker gains this. They can now log into that website BUT NOT everyone else for your username if you happen to use the same password. Jeez am I wrong here?
    • johnnyRose 8y ago
      I have looked into this in the past and came to the same conclusion. Essentially, the "password" sent to the website is the client-side salted/hashed version of the user's actual password. The server could then salt/hash the "password" another time before storing it. This could result in the same issue if the "password" is logged, but it protects the user's true password from being discovered. Maybe a security expert can weight in on this, because I don't understand why this wouldn't be the standard.
      • lixtra 8y ago
        Ideally the server doesn’t need to remember the salt it sent to the client, so it should be signed together with a timestamp to avoid reuse. While you’re at it you can also add some hash puzzle to be solved by the client increasing difficulty with failed logins.
• DGAP 8y ago
  Not a great look for Stamos.
• __ralston3 8y ago
  I find it pretty shocking that other commenters are looking at this as excusable. I mean, is that OK/excusable at your company? Logging payloads/bodies of sensitive requests in plain text - 0 obfuscation. That's ok? Wow. Other commenters are saying "it's logging so it's a forgivable mistake". Is it though? Obviously the world won't end because of these decisions, but holy hell I can't believe this wasn't caught/brought up in some type of code review. This seems pretty 101-ish
  • city41 8y ago
    This kind of thing can often be hard to catch in a code review, because often it's the combination of several systems that cause this to happen. Tracing the user's password from submission form all the way to logger would probably require jumping through several layers, most of which are just handed black box blobs that they hand to the next system.
• Mugwort 8y ago
  Is it even possible for this to be the result of negligence when Facebook has some of the best programming talent in the world? Maybe, but I don't think so. I think this was INTENTIONAL. The question at least is why? IDK. Some FB employees probably didn't mind because it gave them unbelievable spying powers. Another thing to consider is how the NSA had a strong relationship with FB for years and the odds of them not knowing about FB's clear text practices is zero. They knew. People within law enforcement probably knew as well. Given how nothing happens at FB without Zuck's apporoval there's no doubt he knew as well.
• bookmarkacc 8y ago
  Feels like this falls ln the infosec team. Our team has a scanner that parses all* logs for passwords and PII. We know that the infosec team at Facebook was a second class citizen. So, its no surprise that this got through. *sampling on large sets
  • bogomipz 8y ago
    What's PII here? I'm not familiar with this acronym.
    • creatornator 8y ago
      Personally Identifying Information
• Mugwort 8y ago
  I hope Amazon, Apple and Google are doing better. How can we know for sure? It's not unreasonable to ask because I would have thought it completely NUTS to think FB used clear text but apparently they did (or still do). What about everyone else? Does anyone know how to find out?
  • javagram 8y ago
    > Does anyone know how to find out? Join the ops team at each of those companies, work your way to a position where you can analyze log files, then start analyzing them and see if you can find data that should have been obscured. I don’t see any way to find out otherwise, we are talking about querying TBs of internal, company specific private logs to see if someone made a mistake. The best anyone could tell you is “I work for company X and I don’t think we’ve had this problem.” Edit: just using a password manager with unique password for every site will solve most of the problems from a customer perspective. My Facebook password is unique to Facebook and I have 2FA so even an engineer with my password from a log couldn’t login to my account.
    • puzzle 8y ago
      At Google you don't get to read random log files, sometimes even from your own project. There are entire pipelines for raw and sanitized logs, with expiration dates and corresponding access controls. Unless you're on the security team, crawling randomly through logs for no good reason is a quick way to get in trouble.
      • lstodd 8y ago
        > At Google you don't get to read random log files > crawling randomly through logs for no good reason is a quick way to get in trouble. So do you get to or do you not?
        • puzzle 8y ago
          I meant crawling your own logs
• fcantournet 8y ago
  How the hell are plaintext passwords ever touching a FB server in 2019 ? oO You should always hash the pwd client-side first... wtf ?
  • javagram 8y ago
    > You should always hash the pwd client-side first... wtf ? This is very rare. What production systems do you know of that use SRP or similar mechanisms? As far as I know the vast majority of companies and applications use server side hashing.
• coldtea 8y ago
  >In this situation what we’ve found is these passwords were inadvertently logged but that there was no actual risk that’s come from this. We, because of all those 2000 devs who had access to them, nobody could never had stolen them...
• gfody 8y ago
  > an ongoing investigation has so far found no indication that employees have abused access to this data what sort of indication would they be looking for? presumably it wouldn't be hard for an employee to have made a copy for themselves without leaving a trace of evidence.
  • ashelmire 8y ago
    I wonder this every time I see it in a report. It’s not like every file access is recorded for 10 years. Or at all. If you’re lucky you know who accessed a machine since the last time you rotated logs. But let’s say the data was mounted and accessible to all internal machines; literally anyone could have looked at it and done whatever they wanted without anyone knowing.
• qpiox 8y ago
  The overpaid Facebook software engineers are ignorant and under-educated on the most basic practices regarding keeping and checking user passwords, that have been standard for at least 20 years. Or maybe somebody did it intentionally?
• misiti3780 8y ago
  This is really inexcusable, but so much of what facebook has done or does is at this point I'm not surprised. I blocked all their IPs and do not have an account. If i had to guess, it was probably legacy code that causes this. Mark Z. started this with raw PHP, if he had access to or had use an framework, they would have provided salted passwords for free. (it is basically impossible to do this in django, rails, etc). That does not make it Ok since they have 1000s of engineers are qualified to fix it, but they probably have/had higher priorities, clearly security is not one of them
• mmcclellan 8y ago
  Facebook's press release https://newsroom.fb.com/news/2019/03/keeping-passwords-secure/ https://newsroom.fb.com/news/2019/03/keeping-passwords-secur... on the matter is pretty nonchalant: >Keeping Passwords Secure >As part of a routine security review in January, we found that >some user passwords were being stored in a readable format >within our internal data storage systems. Nothing to see here folks.
• jordache 8y ago
  What the Fck FB? Why would you ever store plaintext password?
• packet_nerd 8y ago
  It's a shame we're still using passwords, even more so that they are sent to the server. FIDO UAF or U2F is the way this should be done.
• pansinghkoder 8y ago
  but can you code the DAG in 30 minutes?
• madrox 8y ago
  As the article mentions, Twitter disclosed a similar mistake a year ago...and GitHub before them. These are just the organizations responsible enough to say something. Can we take a moment to acknowledge that this is an easy mistake to make? A logger doesn't care if it's a password or not. Strings are strings. As long as the answer is "humans should be more careful" we'll be seeing these kinds of disclosures regularly across the industry. My best attempt to address this in my teams has been to use different data types for different data classifications. Naked strings must be loaded in one of these data types after input sanitization. That makes it easier to catch accidental inappropriate use. This is useful for managing PII as well.
  • skybrian 8y ago
    Yes, this is an industry-wide problem. Many logging API's are designed for convenience, not security. We need systematic solutions that are still easy to use (or they won't be used).
  • gwbas1c 8y ago
    That's why modern languages need something like a typedef! At least C# has a concept of a SecureString for things like passwords.
    • saagarjha 8y ago
      How does SecureString actually work? Presumably it's possible to coerce one into a "normal" string, and I'm sure that someone will end up doing this in the codebase because it's more convenient to work with.
      • gwbas1c 8y ago
        The point of SecureString is to ensure that sensitive information gets removed from RAM in a timely manner. It's just a wrapper around some native memory that gets zeroed when the SecureString object is disposed. Otherwise, when you use standard C# strings, you don't know when the garbage collector will collect them. (Granted, you could do similar things with pinning a char[] to a char* and zeroing it before you end the pin, but who wants to jump through those hoops?)
• C14L 8y ago
  Wasn't it up to 6% of worldwide annual revenue for each individual case, according to the GDPR? In reality though, I would be very surprised if this resulted in any fine at all.
  • devilmoon 8y ago
    4% for each European user, theoretically. Realistically, nothing is gonna happen and they will keep happily earning a fuckton of money from unaware users.
• crankylinuxuser 8y ago
  Better question: As a system administrator, how does one find issues like "improper logging that leads to credential capture"? And how do es one find credentials for internal services crammed in and hidden in old crufty source tomes? Obviously, when you find it, you nuke the creds, change them, and fix it. But you can't automate, because that requires the raw data. And if you hash the passwords, you end up having to hash everything in order to perhaps find the bad hashes. And when dealing with MBs/s of log, you can't offset-hash everything to scan... and you're back to raw text-matching passwords.. How do others do this, so we can do the right thing?
  • prophesi 8y ago
    > Obviously, when you find it, you nuke the creds, change them, and fix it. The status quo is to toggle a boolean on their account. If they sign in and this boolean is true, then don't actually sign them in. Tell them they need to change their password first. Then you'd send out an email to the account holder to change their password. If they actually own the email, and aren't an imposter, then you're good to go. So in terms of automation, it's literally just a boolean check during log ins, and make sure you're actually doing password resets correctly.
    • crankylinuxuser 8y ago
      You've reduced the scope of my original question, to that of compromised accounts. My question is a magnitude larger than that. For example, someone checks in code and left a config file semi-populated which included a live login credential. Obviously the answer is "Dont do that!" but we all are human. People accidentally post apikeys, credentials and other secure things. It happens. But how do you automatically find issues like this? If PII is radioactive, credentials are gamma-emitters. We solved this at a job long ago with a list of plaintext passwords on a machine you couldn't log into except local (butt-in-seat), and it took your SVN commit, scanned it, and gave a pass/fail. That methodology at least stopped our internal shared service accounts from being accidentally compromised.
      • prophesi 8y ago
• rhegart 8y ago
  My Instagram was compromised a couple of years ago. My password was very secure and completely unique to Instagram so I found this extremely odd. Someone had logged in from the Bay Area. I believe it was compromised twice within 24 hours and then my Facebook was too with a completely unique password as well. Facebook emailed me to change all passwords. I have perhaps over 50 acquaintances working at Facebook. Could someone internally have accessed it as a prank? I don’t remember all the details but I did change my bank account immediately after as well just in case.
• redsavagefiero 8y ago
  I've seen this so many times in self-rolled auditing kludges that it has become kind of a running joke. I'm now surprised if anyone does anything more sophisticated than sym encryption of master credentials used to guard secrets without using the same secret for both as a first pass. Embedded master passwords abound in most legacy OSS configurations.
• OrgNet 8y ago
  Somewhat unrelated but if you have two Facebook accounts with very similar login names/email address, and different passwords, it doesn't matter which login name that you use, it will login into the account based on the password only. I contacted Facebook about it and they say that it is an intentional "feature".
  • devilmoon 8y ago
    I can actually log into Facebook with a password I used on there roughly two years ago, which to me seems like a VERY bad security issue. I'm hoping they are at least fingerprinting my devices so that if someone else tries to access my account with an old password it won't let them, but still...
    • julienreszka 8y ago
      What do you mean by "here"
• prophesi 8y ago
  What irks me is that they're not going to issue a mandatory password reset, since they only do so "in cases where there’s definitely been signs of abuse." Even if it's an internal server, having passwords stored in plain-text always merits a password reset. Storing it on a file-server unencrypted somewhere is just as bad (and arguably worse), than storing them in plaintext in the database.
• morningmoon 8y ago
  Another reason to use passwordless login. Just send an one-time login code via email. That’s what my companies do and there’s no way we can leak passwords... there aren’t any! I honestly don’t understand the point of using passwords when a website allows reset by email.
• residentfoam 8y ago
  This is just not acceptable. I deleted my Facebook account over a year ago and I am doing just fine. I don't miss a bit of it. At a company at the scale of Facebook mistakes like this are not acceptable. It looks like engineers there don't have too much experience in the field, but they are probably very good at flipping binary trees on a white board.
• thefifthsetpin 8y ago
  Why is it normal to not hash passwords in the browser before sending them over the wire? Seems like it's opening the door to issues such as this one, or even more serious ones like when cloudflare leaked web server memory into other sites. I understand that you'd still need another hashing layer before inserting it into the db, but wouldn't some hashing client-side be useful?
  • gol706 8y ago
    For all intents and purposes that hash becomes the users password at that point. Also you can't really salt that hash in a meaningful way.
  • throwawaysec101 8y ago
    Yes we should be hashing to avoid these sort of password logging issues. This is one of those old internet tech-debt that should be resolved in the future either via JS or HTML standard where password is hashed before sending to the server. Reasons for this are: - Internet services have grown and it's too much of a burden for user to have different password for every service. - Even if users have different password, they use the same pattern and add some number of special character at the end which defeats the purpose if password pattern is revealed. In order to protect the pattern of password across multiple services these login services should use client-side hashing. Think of it like something along the lines of SSL green icon in chrome, services not using client-side hashing are leaking it somewhere (no way to tell if they are not leaking).
  • mic47 8y ago
    This does not help at all. If you hash client side,then the hash will became your password. If you accidentally log this hash, anyone who have access to that hash can log in as you easily by constructing logun request.
    • deleted 8y ago
      [deleted]
• tareqak 8y ago
  Facebook's newsroom post on the story: https://newsroom.fb.com/news/2019/03/keeping-passwords-secure/ https://newsroom.fb.com/news/2019/03/keeping-passwords-secur... .
• hetspookjee 8y ago
  Meanwhile the stock of facebook has risen with 0.5%. How does this even work? This is enormous lack of oversight and proper processes. In addition the original post of facebook is rich in media speak: First they speak of _some_ users and _readable format_ and next paragraph they speak of _hundreds of millions_ We estimate that we will notify hundreds of millions of Facebook Lite users, tens of millions of other Facebook users, and tens of thousands of Instagram users. As part of a routine security review in January, we found that some user passwords were being stored in a readable format within our internal data storage systems. With this technique, we can validate that a person is logging in with the correct password without actually having to store the password in plain text.
• lern_too_spel 8y ago
  Is anybody surprised? This is the same company that served its login form over unencrypted http for many years. Prior to using a password manager, I used my password for "likely incompetent" companies for my Facebook account.
• dontbenebby 8y ago
  I wonder if this is related to the fact Facebook passwords are (were?) not case sensitive?[1] (Ex: some script checks for hashes of the password, but not hashes of the permutations that are also accepted?) [1] https://www.zdnet.com/article/facebook-passwords-are-not-case-sensitive-update/ https://www.zdnet.com/article/facebook-passwords-are-not-cas...
  • dymk 8y ago
    Facebook passwords are case sensitive. The only case swapping that happens is to correct leaving your caplocks key on, or if your keyboard auto-capitalizes the first letter (like some mobile keyboards). This reduces the search space by a factor of 3 (aka nothing), not by all permutations of capitalization.
    • dontbenebby 8y ago
      >Facebook passwords are case sensitive. The only case swapping that happens is to correct leaving your caplocks key on That's not the definition of "case sensitive" I've seen used by non-FB engineers.
• runesoerensen 8y ago
  Facebook's own announcement is titled "Keeping Passwords Secure"[0] Facebook comms: You can't really use the word "keep" when the entire post is about how you have failed to keep passwords secure, by storing it in plaintext. Or at least, you probably shouldn't use that word. It signals that you intend to keep doing what you're doing, which wasn't just keeping passwords insecure up until sometime in January. It's up until today when you effectively disclosed that you haven't notify hundreds of millions of users that their passwords were compromised for months after discovering it. I know they "were never visible to anyone outside of Facebook", but that group still includes some +25K people. [0] https://newsroom.fb.com/news/2019/03/keeping-passwords-secure/ https://newsroom.fb.com/news/2019/03/keeping-passwords-secur...
  • ggggtez 8y ago
    And they found 2000 employees accessed the data. I don't care how secure they think the passwords are in those 2000 people's hands. You can bet those have been sold on the black market if they leaked to that many people. Just takes 1 unscrupulous person, which is why you encrypt passwords so even if it leaks, the leaker can't get the original!
  • tonyjstark 8y ago
    Why informing them and not resetting their password as a standard security measurement? I think the headline makes it even worse because it's so tone deaf.
  • CobrastanJorji 8y ago
    Facebook comms knows that. That's why Facebook comms used the word "keep." It's also why they said "some" passwords and not "hundreds of millions" of passwords. Facebook comms also knows that "there is nothing more important to us than protecting people’s information" is a bald-faced lie, since Facebook is literally in the business of selling people's information to companies like Cambridge Analytica, but sometimes it's okay to lie as long as it sounds like a meaningless platitude so that nobody would be expected to seriously believe it.
• lateralux 8y ago
  Orwellian Cyberpunk Idiocracy
• craftoman 8y ago
  I remember when I was in Greek college and we were building a login app with PHP and MySQL. We actually were storing every password in plain text and I asked my professor why shouldn't we hashed them for better security and he was like "that's not necessary cause we overload the server hashing all those passwords". For a moment I thought I was paranoid but then I start thinking how vulnerable some systems may be.
• diogenescynic 8y ago
  Facebook seems to be the biggest cyber-threat and insecurity to the entire United States. When does Facebook get shut down for being a national security threat?
• softdevfromerth 8y ago
  Some number of Facebook employees ABSOLUTELY knew this was going on, quietly talked amongst themselves, maybe brought it up, but then it was knocked back down. 100% I guarantee it. One company I worked for had plain-text passwords and it was quietly discussed (or not discussed) until we EVENTUALLY fixed it (years). Fuck facebook with a rusty pitchfork. This is the tip of the iceberg with their data-handling practices. I bet their systems are fucked throughout.
• lkrubner 8y ago
  Possibly this is the biggest deal of all: > and searchable by thousands of Facebook employees Any time we use a big Web service we understand that there must some employee at that service who could see our activity, but we expect the number of employees who could see our data must be strictly limited. Thousands of Facebook employees could see user passwords? If true, that is an amazing number. I know that when I use Gmail, there must be some employee at Google who could potentially see what I've written. But I've also read that Google has strict controls about who could see my email, and the activity of their employee is logged. What Facebook allowed sounds serious, if only because they were so lackadaisical about enforcing strict policies around data access.
• elken 8y ago
  As the saying goes. Never trust a PHP developer!
  • dang 8y ago
    Could you please stop posting unsubstantive comments to Hacker News?
• julienreszka 8y ago
  I'm shocked.
• turc1656 8y ago
  One of the many, many reasons to not ever use the "login with your Facebook account" (or Google or whatever) on any site. And that's before you even take into consideration what sites like Facebook and Google are doing with that information.
• xwvvvvwx 8y ago
  this is why it’s a good idea to delete your old logs
• uvu 8y ago
  How about private posts? Every time when you type something on Facebook, it's all logging to logs?
• caseymarquis 8y ago
  I'm surprised more sites don't hash passwords with a per user salt and hash/match with a per session salt before transmitting them.
• known 8y ago
  Criminal negligence that can invite class action suit
• duchenne 8y ago
  Let's say that this problem occurred because they logged the body of post requests (including the passwords). Could this problem be mitigated by hashing the password on the front-end (and a second time on the back-end)? Like this, the server would never see the plain text password. Also, if a facebook employee intercepts the hashed password, he could only use it to login to a facebook account. But, I guess he could do that anyway if he has access to the db. However, this employee would have no way to know the real password and use it on another website.
• DBYCZ 8y ago
  To facebook's credit: I created a brand new email address and unique password to be used exclusively for my Facebook account, and 6 years later (with no password changes), I have received no spam and had no discernable security issues. Maybe I was just fortunate, but everyone should use a burner email if they choose to use Facebook.
• laacz 8y ago
  I do wonder if there was a devops feature built on top of this.