6 ms·

I don't think anyone's arguing that it's ok, just that it's less "easy to screw up" vs "hard to screw up" and more "did something really stupid on purpose" and

I don't think anyone's arguing that it's ok, just that it's less "easy to screw up" vs "hard to screw up" and more "did something really stupid on purpose" and "did something really stupid by accident." But yes, either way the end result is the same and you shouldn't do it.

dumbfounder 8y ago
Yes but how many people looked at these logs, saw the passwords, and said nothing? If it was logged presumably the logs were viewed from time to time. The mistake is easy to make, but it's also easy to correct if there is a culture to report such mistakes.
- wlesieutre 8y ago
  That's true. Kerbs article says: > My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
  - meitme 8y ago
    Those numbers are pretty meaningless without some concept of what queries they were making. When I was working with large logs I'd regularly pull huge amounts of logs to only look at maybe a handful of lines. Even then often only scanning for the things I'm specifically working on. Not saying this is what happened just saying those 2,000 employees almost definitely were not doing careful analysis on the complete result of their 9 million queries(And shouldn't have!).
    - mk89 8y ago
      Out of 2000 users, there was not one that had a look at username/password? Impressive. Also developers during dev. Seriously, this is not justifiable, it's pure laziness if we want to be positive.
    - anigbrowl 8y ago
      'Only scanning for the things one is working on' to the point of missing huge security holes or safety risks is exactly that's wrong with the industrial division-of-labor approach. Workers start to assume that even if they do notice a problem, someone else is either going to fix it or has examined and OKed the current procedure. Workers who speak up and assert that something they noticed actually looks more important than whatever task they were originally assigned to are typically punished rather than rewarded, and fired if they make too much of a fuss because otherwise managerial authority might be diminished. Most firms run internally like a dictatorship, and given Facebook's absurd ownership structure (where there are voting and non-voting shares Zuckerberg retains a majority of the voting shares) it's a recipe for dysfunction.
      - tptacek 8y ago
        This is true but not very interesting. Like literally every large tech company in the entire industry, Facebook has a specialized team of people that work on security --- it has one of the better versions of that team, for what it's worth. Just like ordinary Apple engineers, ordinary Google engineers, ordinary LinkedIn engineers, and ordinary Airbnb engineers, ordinary Facebook engineers don't have an end-to-end picture of how all of security at Facebook works. There's no evidence that there was whistleblowing suppression about this issue anywhere at Facebook, is there?
    - vijay_n 8y ago
      What if one among the 2000 was curious ?
- code_duck 8y ago
  The article indicates it was searchable and used for something by employees, so most of these comments to the effect of ‘they must have been doing this accidentally because Facebook is just so big therefore they are excused’ are invalid. Some group of people did this on purpose and knew it was happening.
  - chatmasta 8y ago
    You're probably being downvoted for speculation, but that sounds completely reasonable to me. At least one person had to have noticed a password in a log file they were viewing at some time. Most people viewing log files know that passwords should not be there. It would trigger alarm bells, and depending on the person, also excitement -- you have somebody's facebook password. It's conceivable that someone then told their friend at work, and a few of these 2,000 developers knew of this secret internal stash of passwords they could access whenever they wanted to "prank" someone on facebook...
    - YokoZar 8y ago
      Why would humans be reading these log files?
      - chatmasta 8y ago
        For plenty of reasons, I'm sure. From the article: > My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
        YokoZar 8y ago
        That doesn't sound like a human reading the log. It sounds like automation. A script counting the number of successful logins, for instance, could read the same data element that unintentionally contained passwords.
        8y ago
- rmason 8y ago
  Zuck doesn't believe in hiring employees over 30 because they're not 'smart' about tech. Great example why that's a mistake. If he did, someone would have seen that with 10-15 years of corporate experience and would have found that simply unacceptable. They also would have had the gravitas to know how to escalate things to the top and gotten it fixed.
EpicEng 8y ago
I understand that, but I believe they do not have the correct mentality here. I'm not certain how many of the people saying that have worked on systems as sensitive as FB. They think it's easier to screw this up because, for them, it is, but they don't operate at this level. This sort of thing should not be any easier to screw up when you operate at the scale of FB. If it happens then you have the wrong procedures in place. In my personal example our logs and logging practices/code were audited in the same way our DB layer was. There was no difference; if a system touches sensitive data, it is a massive vulnerability.
- Alex3917 8y ago
  This. If you have two or three engineers at a startup it's a very easy mistake to make. With thousands of engineers it's an impossible mistake to make, so it could have only been done purposely.
  - JohnFen 8y ago
    And even with a startup of two or three engineers, it's pretty much unforgivable.
    - Alex3917 8y ago
      It depends on how many users you have and what sort of data you're protecting. If you have a billion users then sending password reset tokens to your analytics provider is a very serious vulnerability. If you have 100 users and you're running a generic message board then it's not really a vulnerability because the difficulty of that method of attacking the assets under protection almost certainly exceeds their value.
      - marshray 8y ago
        > that method of attacking the assets under protection almost certainly exceeds their value This was most likely the reasoning Sony had used (prior to 2012) when deciding how to safeguard the account info of users who had registered for marketing promotions. It turned out that user credentials are unlike, say, office furniture, in that the cost of a data breach can vastly exceed the "value" of the protected "asset".

dumbfounder 8y ago

Yes but how many people looked at these logs, saw the passwords, and said nothing? If it was logged presumably the logs were viewed from time to time. The mistake is easy to make, but it's also easy to correct if there is a culture to report such mistakes.

wlesieutre 8y ago
That's true. Kerbs article says: > My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
- meitme 8y ago
  Those numbers are pretty meaningless without some concept of what queries they were making. When I was working with large logs I'd regularly pull huge amounts of logs to only look at maybe a handful of lines. Even then often only scanning for the things I'm specifically working on. Not saying this is what happened just saying those 2,000 employees almost definitely were not doing careful analysis on the complete result of their 9 million queries(And shouldn't have!).
  - mk89 8y ago
    Out of 2000 users, there was not one that had a look at username/password? Impressive. Also developers during dev. Seriously, this is not justifiable, it's pure laziness if we want to be positive.
  - anigbrowl 8y ago
    'Only scanning for the things one is working on' to the point of missing huge security holes or safety risks is exactly that's wrong with the industrial division-of-labor approach. Workers start to assume that even if they do notice a problem, someone else is either going to fix it or has examined and OKed the current procedure. Workers who speak up and assert that something they noticed actually looks more important than whatever task they were originally assigned to are typically punished rather than rewarded, and fired if they make too much of a fuss because otherwise managerial authority might be diminished. Most firms run internally like a dictatorship, and given Facebook's absurd ownership structure (where there are voting and non-voting shares Zuckerberg retains a majority of the voting shares) it's a recipe for dysfunction.
    - tptacek 8y ago
      This is true but not very interesting. Like literally every large tech company in the entire industry, Facebook has a specialized team of people that work on security --- it has one of the better versions of that team, for what it's worth. Just like ordinary Apple engineers, ordinary Google engineers, ordinary LinkedIn engineers, and ordinary Airbnb engineers, ordinary Facebook engineers don't have an end-to-end picture of how all of security at Facebook works. There's no evidence that there was whistleblowing suppression about this issue anywhere at Facebook, is there?
  - vijay_n 8y ago
    What if one among the 2000 was curious ?
code_duck 8y ago
The article indicates it was searchable and used for something by employees, so most of these comments to the effect of ‘they must have been doing this accidentally because Facebook is just so big therefore they are excused’ are invalid. Some group of people did this on purpose and knew it was happening.
- chatmasta 8y ago
  You're probably being downvoted for speculation, but that sounds completely reasonable to me. At least one person had to have noticed a password in a log file they were viewing at some time. Most people viewing log files know that passwords should not be there. It would trigger alarm bells, and depending on the person, also excitement -- you have somebody's facebook password. It's conceivable that someone then told their friend at work, and a few of these 2,000 developers knew of this secret internal stash of passwords they could access whenever they wanted to "prank" someone on facebook...
  - YokoZar 8y ago
    Why would humans be reading these log files?
    - chatmasta 8y ago
      For plenty of reasons, I'm sure. From the article: > My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.
      - YokoZar 8y ago
        That doesn't sound like a human reading the log. It sounds like automation. A script counting the number of successful logins, for instance, could read the same data element that unintentionally contained passwords.
        8y ago
rmason 8y ago
Zuck doesn't believe in hiring employees over 30 because they're not 'smart' about tech. Great example why that's a mistake. If he did, someone would have seen that with 10-15 years of corporate experience and would have found that simply unacceptable. They also would have had the gravitas to know how to escalate things to the top and gotten it fixed.

EpicEng 8y ago

I understand that, but I believe they do not have the correct mentality here. I'm not certain how many of the people saying that have worked on systems as sensitive as FB. They think it's easier to screw this up because, for them, it is, but they don't operate at this level. This sort of thing should not be any easier to screw up when you operate at the scale of FB. If it happens then you have the wrong procedures in place. In my personal example our logs and logging practices/code were audited in the same way our DB layer was. There was no difference; if a system touches sensitive data, it is a massive vulnerability.

Alex3917 8y ago
This. If you have two or three engineers at a startup it's a very easy mistake to make. With thousands of engineers it's an impossible mistake to make, so it could have only been done purposely.
- JohnFen 8y ago
  And even with a startup of two or three engineers, it's pretty much unforgivable.
  - Alex3917 8y ago
    It depends on how many users you have and what sort of data you're protecting. If you have a billion users then sending password reset tokens to your analytics provider is a very serious vulnerability. If you have 100 users and you're running a generic message board then it's not really a vulnerability because the difficulty of that method of attacking the assets under protection almost certainly exceeds their value.
    - marshray 8y ago
      > that method of attacking the assets under protection almost certainly exceeds their value This was most likely the reasoning Sony had used (prior to 2012) when deciding how to safeguard the account info of users who had registered for marketing promotions. It turned out that user credentials are unlike, say, office furniture, in that the cost of a data breach can vastly exceed the "value" of the protected "asset".