7 ms·

> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contain

by pizza 8y ago

> My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords. This imo is the truly alarming takeaway. FB employees were retrieving user passwords? Around two thousand FB employees? How in God's name is Zuckerberg going to perform his usual performative contrition about that one? I'm just trying to imagine the data structures that were being retrieved from databases. Either they stored something like a big user account data type that contained their password in plaintext, which imo is a really weird design choice, or logs for other services were being mixed in with logs leaking the user/pass combos. Surely one of the engineers could have noticed and said 'wait a minute... those are logins' over the course of the years? We hear all the time that people want FB to follow a responsible social practices (the debate on what those are rages on, which is great imo), but can't FB at least wrangle its own code base? On the other hand, we shouldn't take the stance that heads should roll, imo - it would just create a chilling effect that would deter other companies from ever going public about their own security mishaps. edit: I should probably tone it down in this comment but I'll leave it for posterity

rhacker 8y ago
This should be the main takeaway. If true, then 2000 people decided not to raise the alarm. And 9,000,000 queries with results that had passwords. This completely invalidates the other thread of people discussing giving facebook a pass because accidental logging could happen to any company.
- weka 8y ago
  How much do you want to bet that some of those FB employees decided to login by turning off "location detection" flag and in an incognito browser? I would not be surprised. Some of those 9,000,000+ queries and 2,000+ employees must have had some nefarious use. Statistically speaking...
  - kitsune_ 8y ago
    Judging by the maturity level of the discourse on teamblind this precisely what happened.
    - sanderjd 8y ago
      Judging by working with hundreds of software engineers over a decade, this seems very unlikely to have happened.
- cellularmitosis 8y ago
  > If true, then 2000 people decided not to raise the alarm Have you ever worked at a software co, discovered <terrible software practice>, raised a flag, and were told "thanks, we've added a ticket to the backlog"? I'd bet that some of them did flag it.
  - rhacker 8y ago
    Still interesting that not one of them went all out and publicly derided the company. The whistleblower clause would probably have protected them. However our whistleblower laws could use a brush up.
  - deusofnull 8y ago
    You're probably right, and I would love to see that confirmed. FB wouldnt have to link to the Jira ticket or give the name or details of the ticket, but verify that was the case and then explain why nothing was done.
  - gwbas1c 8y ago
    Explicitly logging a password is one of those practices that doesn't sit on the backlog. It's probably a bit more complicated than that. Usually the things that I encounter have to do with how HTTP requests are logged. For example, putting sensitive information in a URL that's loaded over HTTPS is considered insecure because many companies have policies where they log every URL that their employees visit. (Think of a password reset link.) A lot of inexperienced programmers don't realize this, because they don't realize that you can man-in-the-middle yourself, and that most corporate computers come preconfigured to allow the employer to man-in-the-middle everyone. So, if a password reset link never expires, it means that some guy in IT can own an account that was reset on a corporate computer. (This, basically, is how they catch people viewing porn on their work computers.) Anyway, my point is that the problem is probably something where a junior programmer transmitted a password in a way that they didn't realize was being logged.
    - e7ebanlv 8y ago
      Back when I was sysadmin, I catched everyone watching porn by suddenly walking into their offices, but I'm sure your method is better.
memory_grep 8y ago
Maybe many people were just using SELECT * FROM ... and got plain text passwords they weren't necessarily looking for.
- pard68 8y ago
  I read the article and this is what I got from it too. I `SELECT *` all the time and see hashed passwords. It is very rare that I need that hash, usually I'm doing something entirely unrelated, but it's just easy to get all the rows and only use what you need, especially when you are troubleshooting and don't even know what you need yet.
jeltz 8y ago
Isn't it possible that those developers just wanted to look at something harmless like for example failed logins and did not even see or reflect on that the log messages also included the password? But if so it is a bit worrying that 2000 eyes did not spot the bug.
IshKebab 8y ago
No you're misrepresenting what (probably) happened. They had some system that logged API requests (fine and normal). Some API requests include plaintext passwords (also fine and normal). The issue is presumably that they had no exclusion to the logging for sensitive information like passwords, which is honestly very easy to overlook. So two thousand Facebook employees were not "retrieving passwords". They were looking at the API logs, which is a normal thing to do.
- planb 8y ago
  This. It's a bit baffling to me that even the HN audience doesn't come to this conclusion immediately - maybe people want to assume the worst because it's Facebook?
  - IshKebab 8y ago
    Definitely. This always happens when Facebook comes up here. People forget to use their brains and just rant about how evil Facebook is.
- C4stor 8y ago
  > Some API requests include plaintext passwords (also fine and normal) Wait, no ? How is that fine and normal ? It doesn't seem fine and normal all.
  - yogaboll 8y ago
    It's completely standard. If you encrypt the password in the browser, potential hackers have access to the encryption code which makes it useless. Check the network tab in dev-tools when logging in to hn for example and you'll see your password in the request body :)
- mesozoic 8y ago
  Maybe normal but still not best practice
oldmanbythesea3 8y ago
What that quote is saying is that the logs that contained the passwords were accessed by 2,000 engineers. Most of those engineers would only be looking at data relevant to their job, only a security engineer would be in a position to notice the passwords, which is what happened.
breischl 8y ago
>FB employees were retrieving user passwords? Around two thousand FB employees? This sounds to me like they were writing it into some log analysis tool, and people happened to pull it up. I have no knowledge of FB internals, but as an example let's say they have Splunk, which is a popular distributed log analysis tool (or some custom equivalent, because FB is huge). Splunk makes it very easy to pull up tons of logging for any app - that's what it's for. So write a poorly-constrained query and you can get millions of log lines from many different apps. It's also feasible that everyone (or nearly so) in the company has access to Splunk. So if somebody, somewhere accidentally writes passwords into a logfile, and it gets indexed into Splunk, a ton of people might theoretically see it, but just skim past it because it's not what they were looking for. Picture trying to hunt down an error, and wading through many screens full of logging from different apps - if there's a password in there you could easily not notice if you're not looking for it. That's all speculation, but it's also entirely plausible. All it takes is accidentally logging out those passwords to somewhere that Splunk can see. Also not trying to say this is in any way OK or acceptable, just that I can understand how a seemingly-small mistake could quickly result in thousands of people having access to passwords.
zxcvvcxz 8y ago
Probably some poor engineers trying to check if their partners were cheating on them. Tools like FB make it easier after all.

by pizza 8y ago

rhacker 8y ago

This should be the main takeaway. If true, then 2000 people decided not to raise the alarm. And 9,000,000 queries with results that had passwords. This completely invalidates the other thread of people discussing giving facebook a pass because accidental logging could happen to any company.

weka 8y ago
How much do you want to bet that some of those FB employees decided to login by turning off "location detection" flag and in an incognito browser? I would not be surprised. Some of those 9,000,000+ queries and 2,000+ employees must have had some nefarious use. Statistically speaking...
- kitsune_ 8y ago
  Judging by the maturity level of the discourse on teamblind this precisely what happened.
  - sanderjd 8y ago
    Judging by working with hundreds of software engineers over a decade, this seems very unlikely to have happened.
cellularmitosis 8y ago
> If true, then 2000 people decided not to raise the alarm Have you ever worked at a software co, discovered <terrible software practice>, raised a flag, and were told "thanks, we've added a ticket to the backlog"? I'd bet that some of them did flag it.
- rhacker 8y ago
  Still interesting that not one of them went all out and publicly derided the company. The whistleblower clause would probably have protected them. However our whistleblower laws could use a brush up.
- deusofnull 8y ago
  You're probably right, and I would love to see that confirmed. FB wouldnt have to link to the Jira ticket or give the name or details of the ticket, but verify that was the case and then explain why nothing was done.
- gwbas1c 8y ago
  Explicitly logging a password is one of those practices that doesn't sit on the backlog. It's probably a bit more complicated than that. Usually the things that I encounter have to do with how HTTP requests are logged. For example, putting sensitive information in a URL that's loaded over HTTPS is considered insecure because many companies have policies where they log every URL that their employees visit. (Think of a password reset link.) A lot of inexperienced programmers don't realize this, because they don't realize that you can man-in-the-middle yourself, and that most corporate computers come preconfigured to allow the employer to man-in-the-middle everyone. So, if a password reset link never expires, it means that some guy in IT can own an account that was reset on a corporate computer. (This, basically, is how they catch people viewing porn on their work computers.) Anyway, my point is that the problem is probably something where a junior programmer transmitted a password in a way that they didn't realize was being logged.
  - e7ebanlv 8y ago
    Back when I was sysadmin, I catched everyone watching porn by suddenly walking into their offices, but I'm sure your method is better.

memory_grep 8y ago

Maybe many people were just using SELECT * FROM ... and got plain text passwords they weren't necessarily looking for.

pard68 8y ago
I read the article and this is what I got from it too. I `SELECT *` all the time and see hashed passwords. It is very rare that I need that hash, usually I'm doing something entirely unrelated, but it's just easy to get all the rows and only use what you need, especially when you are troubleshooting and don't even know what you need yet.

jeltz 8y ago

Isn't it possible that those developers just wanted to look at something harmless like for example failed logins and did not even see or reflect on that the log messages also included the password? But if so it is a bit worrying that 2000 eyes did not spot the bug.

IshKebab 8y ago

No you're misrepresenting what (probably) happened. They had some system that logged API requests (fine and normal). Some API requests include plaintext passwords (also fine and normal). The issue is presumably that they had no exclusion to the logging for sensitive information like passwords, which is honestly very easy to overlook. So two thousand Facebook employees were not "retrieving passwords". They were looking at the API logs, which is a normal thing to do.

planb 8y ago
This. It's a bit baffling to me that even the HN audience doesn't come to this conclusion immediately - maybe people want to assume the worst because it's Facebook?
- IshKebab 8y ago
  Definitely. This always happens when Facebook comes up here. People forget to use their brains and just rant about how evil Facebook is.
C4stor 8y ago
> Some API requests include plaintext passwords (also fine and normal) Wait, no ? How is that fine and normal ? It doesn't seem fine and normal all.
- yogaboll 8y ago
  It's completely standard. If you encrypt the password in the browser, potential hackers have access to the encryption code which makes it useless. Check the network tab in dev-tools when logging in to hn for example and you'll see your password in the request body :)
mesozoic 8y ago
Maybe normal but still not best practice

oldmanbythesea3 8y ago

What that quote is saying is that the logs that contained the passwords were accessed by 2,000 engineers. Most of those engineers would only be looking at data relevant to their job, only a security engineer would be in a position to notice the passwords, which is what happened.

breischl 8y ago

>FB employees were retrieving user passwords? Around two thousand FB employees? This sounds to me like they were writing it into some log analysis tool, and people happened to pull it up. I have no knowledge of FB internals, but as an example let's say they have Splunk, which is a popular distributed log analysis tool (or some custom equivalent, because FB is huge). Splunk makes it very easy to pull up tons of logging for any app - that's what it's for. So write a poorly-constrained query and you can get millions of log lines from many different apps. It's also feasible that everyone (or nearly so) in the company has access to Splunk. So if somebody, somewhere accidentally writes passwords into a logfile, and it gets indexed into Splunk, a ton of people might theoretically see it, but just skim past it because it's not what they were looking for. Picture trying to hunt down an error, and wading through many screens full of logging from different apps - if there's a password in there you could easily not notice if you're not looking for it. That's all speculation, but it's also entirely plausible. All it takes is accidentally logging out those passwords to somewhere that Splunk can see. Also not trying to say this is in any way OK or acceptable, just that I can understand how a seemingly-small mistake could quickly result in thousands of people having access to passwords.

zxcvvcxz 8y ago

Probably some poor engineers trying to check if their partners were cheating on them. Tools like FB make it easier after all.