12 ms·

"...various errors seem to have caused Facebook’s systems to log some passwords in plain text since as early as 2012." 1) As a software engineer I can't imagin

• verelo 8y ago
  "1) As a software engineer I can't imagine how such errors could possibly have entered production code accidentally" Like i'd love to agree, but frankly, i'm not surprised. I feel like I've seen crazier things happen on production environments...
• eatmyass 8y ago
  >1) As a software engineer I can't imagine how such errors could possibly have entered production code accidentally, especially after code review. If precise details of these errors are released I am open to have my mind changed. You can't imagine? Really? Error occurs, request that caused error gets logged.
• throwaway2016a 8y ago
  Not too far fetch... For example, I've seen people put in place logging systems that logged the whole request body and then someone changed a field name and all the sudden as a side effect the code that sanitizes the body to remove the sensitive information no longer does it's job.
  • manojlds 8y ago
    I liked the suggestion of logging known password to detect leaks
    • danek 8y ago
      It’s a good start at least. You still have to worry about the password being in the middle of a base64 encoded string and weird cases like that but at least you could grep for some known things.
• dagw 8y ago
  As a mere programmer I can very easily image dozens of ways such errors could easily enter production code accidentally.
  • megous 8y ago
    With my admin hat on, I don't see how I would miss leaking millions of passwords to logs, since 2012. Unless I never reviewed them.
    • int_19h 8y ago
      If they logged complete requests, the password might well be buried somewhere deep in request body, and not readily apparent to whoever is skimming the logs - especially if this doesn't happen for all requests, but only some subset of them.
      • thelittleone 8y ago
        Wouldn’t it be pretty simple (for FB) to create accounts with super unique password as part of their test process? Then they scan all their storeage for the super unique password? Or am I missing something?
        • slig 8y ago
          Do you know of any companies doing that?
          • shshhdhs 8y ago
            Yes, I do. They’re called canaries by the security teams. It’s especially useful for when you run automated tests when pushing updates in your pipeline. Scan the logs and database entries for the canaries.
            • slig 7y ago
              Thank you, didn't know what they're called.
• kevingadd 8y ago
  During a job interview at one mid-size startup one of the interview questions involved handing me a 2 page excerpt of server logs and asking me to identify bugs/issues from the messages and tracebacks in the logs. (neat idea!) The logs contained user credentials, and they hadn't noticed. I pointed it out to the CTO.
  • stevenwliao 8y ago
    Wow, their interview question (in the hands of the wrong interviewee) could have caused a data breach.
    • londons_explore 8y ago
      If it's a plaintext password, the minute it hits any human eyes, it is IMO compromised and the user should be required to change it. Employee, Interviewee, Hacker - doesn't matter.
      • itslennysfault 8y ago
        I'd go a step further. IMO the second it hits a hard drive it's compromised so just the act of having that in the log would count.
  • rhizome 8y ago
    Hey, in the wild! I've long advocated for companies to use actual bugs (that have been fixed) in testing candidates. Heck, as your anecdote suggests, they might find the candidate provides a fix better than their ninja rockstar did, and in this way could serve as part of a performance review for existing employees. ;)
• jarsin 8y ago
  I think Rails had this issue by default for some time.
• icedchai 8y ago
  This happens all the time, likely as the side-effect of some other activity - instrumentation related to exception handling / error monitoring, performance analysis, debugging, etc.
• UncleMeat 8y ago
  1. You decide that logging passwords is bad. 2. You create a system that detects high entropy content before being logged. 3. You don't want to drop all high entropy content, so you create some rules about where in requests to look for high entropy content. 4. Something about the request structure changes, breaking your log filtering. 5. There is nothing that notices the drop in the amount of content filtered out of logs. There's oodles of ways this could happen. I'd wager that more than half of all businesses that have a website that handles passwords has logged passwords in plaintext somewhere.
  • hn_throwaway_99 8y ago
    > You create a system that detects high entropy content before being logged. Unfortunately, as numerous password breaches have shown, most passwords aren't that high entropy.
    • mercer 8y ago
      Perhaps also checking logs for x of the most common passwords could work? On the scale of Facebook this might very likely trigger some positives with such a bug?
• itslennysfault 8y ago
  My assumption since the start of this is that it's telemetry / server log data. Honestly, I came really close to doing this exact thing. My engineer: "Should we log the body of api requests?" Me: "Yes, of course!....... waaaait a second. No." The result was manually white listing fields that SHOULD be logged on each end point. It's a pain compared to "just log everything" or even a black list, but it's far safer.