8 ms·
As a mere programmer I can very easily image dozens of ways such errors could easily enter production code accidentally.
by dagw 8y ago
As a mere programmer I can very easily image dozens of ways such errors could easily enter production code accidentally.
- megous 8y agoWith my admin hat on, I don't see how I would miss leaking millions of passwords to logs, since 2012. Unless I never reviewed them.
- int_19h 8y agoIf they logged complete requests, the password might well be buried somewhere deep in request body, and not readily apparent to whoever is skimming the logs - especially if this doesn't happen for all requests, but only some subset of them.
- thelittleone 8y agoWouldn’t it be pretty simple (for FB) to create accounts with super unique password as part of their test process? Then they scan all their storeage for the super unique password? Or am I missing something?
- slig 8y agoDo you know of any companies doing that?
- int_19h 8y agoTo implement, yes. But you'd have to think about this as a problem first. The very fact that we're having this discussion here now shows that it's not all that obvious, at least for your average dev. How their privacy and/or security review missed it is a more interesting question.
- Someone 8y agoThat test user also would have to hit every nook and cranny of their code. That’s hard, and may not be repeatably possible (some code may only run at first login, or when making your 1024th post, or if you’re posting a movie before you ever post a photo, or when a single server sees its millionth post, etc) They also would have to run it in such a way after every code update. Since every code update may introduce a new nook or cranny, that would slow down development too much. I wouldn’t even try doing that, but instead have a two-pronged defense: - code review of every single log statement in the code base by individuals whose _only_ job it is to prevent such problems. - permanent checking of every single line logged for a thousand or so common passwords. At Facebook’s scale, the second probably has lots of false positives, so I would do that at time of logging, when the location doing that logging is known, so that an alarm only gets triggered if a single log statement repeatedly logs passwords.
- Ayesh 8y agoIf someone logs a full request, the first thing that should come to their minds is the privacy. Someone who works as a Dev on Facebook should know that HTTP requests can contain sensitive data, session cookies, passwords, credit card information, etc.
- hn_throwaway_99 8y agoProbably because you don't work on systems anywhere near the scale of Facebook. Facebook has 2.3 billion MAUs, so if they had some edge case that hit 0.1% of users, that's already 2.3 million users.
- nihonde 8y agoAnd what does scale have to do with logging any plaintext passwords?
- hn_throwaway_99 8y agoBecause it was a bug. They most likely weren't doing this: log(request.password); But they had some code that did this: log(excludeSensitive(request. headers)); But then someone on a different team changed how passwords were sent so excludeSensitive was broken.
- nihonde 8y agoScrubbing passwords and other sensitive information from logs is such a basic requirement. It’s something you write a regression test for and never think about again.
- jrochkind1 8y agoAs a manager, I can tell you we would never do such a thing, and if we did, then we never will again, and if we do, never again after that.