6 ms·

Data anonymization and de-identification is nothing new. Especially when it comes to logging. I don't get why a lot of you are downplaying it in this thread. Th

Data anonymization and de-identification is nothing new. Especially when it comes to logging. I don't get why a lot of you are downplaying it in this thread. This is just as bad as plain text passwords.

cflewis 8y ago
Because we all know we are one bad morning away from doing it ourselves. People are distinguishing between extreme incompetence (storing plaintext passwords in databases) vs people trying to do their jobs. It's the same reason when there are large outages, where the comments are split between the enraged customers, and then the ones that are "Man, sucks to be them" as know they could easily have been the one that got the config push wrong.
- 908087 8y ago
  If a single developer is able to push something like this to production without scrutiny in "one bad morning" on a platform with billions of users, the company itself is the problem.
- kitsune_ 8y ago
  The fact that sensitive data like this is just one config push away from being exposed means it's a brittle design (not just the code, but the entire design /review / deployment process - this is a platform with 1 billion users, not your 1-dev WordPress webpage). That said, this is further aggravated by the simple fact that for whatever reason, 20k of their employees had access to these logs. All in all it just paints a picture of an irresponsible company where their house is not in order.
  - lostapathy 8y ago
    Exactly. It's one thing not to have enough controls in place to catch something like this on the forum for your warcraft guild. It's another thing entirely when you operate at the scale of Facebook.
- untog 8y ago
  "people are distinguishing between extreme vs people trying to do their jobs" Those things are in no way exclusive. Storing passwords in plaintext is awful practise. Including the contents of form submissions in logging is a similarly awful practise. I really don't see what the difference is.
  - munk-a 8y ago
    I like logging and at our company we blanket log all GET parameters on every request, similarly I think it's a Good Idea™ to never log POST parameters, but there are some pages where we've started logging some whitelisted ones because they can help diagnose some errors we run across, but, in an environment where all data is coming into the server in JSON blobs attached in request bodies then it can be a bit harder to make any sort of blanket whitelist.
  - ApolloFortyNine 8y ago
    Earlier in the thread: >Because there's two types of storing passwords in plain text. There's the "your password is stored in plaintext in the database" way which everyone agrees is 10 different kinds of stupid, and then there's the "we accidentally logged the body of all requests that went through this system, and it turns out login requests came through here" kind. Basically the conscious decision to store passwords in plain text is worse than unintentionally doing so in logs. One is purposeful, one is not. Yes it's bad, but it's not as bad as if it was done on purpose. And generally, this how most laws are enforced or implemented.
    - untog 8y ago
      Can't both be attributed to ignorance? You could store passwords in plaintext in a database because you didn't know better. You can store plaintext passwords in logs because you don't know better. If I store passwords in plaintext am I being purposeful or am I being ignorant of best practises? I think people are drawing a distinction where one doesn't really exist.
      - ApolloFortyNine 8y ago
- pmontra 8y ago
  I'm working with Rails and Phoenix on two different projects right now and all of them filter out passwords in the log files [1] [2]. I'm also working on a Django project but we're not logging HTTP calls arguments there. I think we could use a filter like [3] but I'd rather have the framework to automatically take care of that. And if I'd be writing a web app from scratch, no matter if I've been doing web for 25 years, I'm sure I'll do a lot of silly mistakes. That's why I prefer to build on top of frameworks. [1] https://guides.rubyonrails.org/configuring.html#rails-general-configuration https://guides.rubyonrails.org/configuring.html#rails-genera... [2] https://hexdocs.pm/phoenix/Phoenix.Logger.html https://hexdocs.pm/phoenix/Phoenix.Logger.html [3] https://djangosnippets.org/snippets/2966/ https://djangosnippets.org/snippets/2966/
EnFinlay 8y ago
You're right, the impact is just as bad. I think people in this thread (including myself) are downplaying it because it's a very easy mistake to make, rather than a terrible design decision.
- newaccoutnas 8y ago
  A mistake that should have been found very quickly, given the eyes. Not for years without ever apparently being known (I find that hard to believe)
- staticassertion 8y ago
  The major difference is that we don't have standard patterns for protecting against one attack, vs the other. With password storage everyone knows the patterns, or we expect them to. With everything between the request and password storage, we don't. This type of attack could easily be prevented. When secrets come in, immediately store them (ideally at the web framework level) in a type that overrides print/debug formatting. Then add a "get_raw" to it, and you can now grep for that being used anywhere outside of storage to a DB (and your DB libs should take the Secret type too). Or don't use a `get_raw` and instead use a `hash` method that returns a safely hashed version of it. Further, your secret type could at the very least add a round of SHA256, maybe even with a pepper?, just to be sure. This isn't hard, I've done it before. The problem is that it isn't something that people feel embarrassed not to do, vs storing plaintext creds. Impact is the same - creds are plaintext in a DB. Attackers always expect sensitive data in logs, so it isn't as if you'd get lucky and they'd miss this.
  - deleted 8y ago
    [deleted]
  - aeorgnoieang 8y ago
    > When secrets come in, immediately store them (ideally at the web framework level) ... But, in practice, a lot of this stuff is being logged by things like proxies that could be several layers in front of the "web framework level".
hak8or 8y ago
The diffirence is intent and/or gross negligence. Storing passwords in plain text in a database requires a gross lack of security mentality in both design stages and implementation changes. It is also drilled into people's brains constantly about how bad of an idea it is. To put it simply, it cannot happen accidentally. Logging like this can easily be attributed to an accident. The person who implemented this logging should get hit with some repercussions because he surely tested the logging and must have seen the passwords when glancing by eye. But other than that, this was clearly a minor oversight.
- zrobotics 8y ago
  How is this in any way better or more excusable? Calling it a 'minor oversight' could also apply to the DB being in plaintext. Storing passwords in a plaintext DB heppes in the same manner-a dev is lazy or ignorant of the security reprocussions. Which is what happened here; barring evidence this was done maliciously we can assume that this was accidental. But that doesn't make it any more excusable, since it should be clear that logs can also contain sensitive data that needs to be protected/anonymized. Considering how selective FB is for hiring, I would hope we could expect a higher standard.
  - glenneroo 8y ago
    Not to mention 200-600 million users had their passwords exposed for many years. That must be a massive trove of log files.
- lacey 8y ago
  You assume that storing passwords in plaintext is intent as opposed to gross negligence. In many past instances it’s been intentional and gross negligence because the people making all the design and implementation choices were not knowledgeable about best practices.
- Aeolun 8y ago
  I see this differently. If someone stores passwords in plaintext in the database, they’re idiots that don’t know any better. If someone logs your password in plaintext but has it encrypted in the database that’s grossly negligent.

cflewis 8y ago

Because we all know we are one bad morning away from doing it ourselves. People are distinguishing between extreme incompetence (storing plaintext passwords in databases) vs people trying to do their jobs. It's the same reason when there are large outages, where the comments are split between the enraged customers, and then the ones that are "Man, sucks to be them" as know they could easily have been the one that got the config push wrong.

908087 8y ago
If a single developer is able to push something like this to production without scrutiny in "one bad morning" on a platform with billions of users, the company itself is the problem.
kitsune_ 8y ago
The fact that sensitive data like this is just one config push away from being exposed means it's a brittle design (not just the code, but the entire design /review / deployment process - this is a platform with 1 billion users, not your 1-dev WordPress webpage). That said, this is further aggravated by the simple fact that for whatever reason, 20k of their employees had access to these logs. All in all it just paints a picture of an irresponsible company where their house is not in order.
- lostapathy 8y ago
  Exactly. It's one thing not to have enough controls in place to catch something like this on the forum for your warcraft guild. It's another thing entirely when you operate at the scale of Facebook.
untog 8y ago
"people are distinguishing between extreme vs people trying to do their jobs" Those things are in no way exclusive. Storing passwords in plaintext is awful practise. Including the contents of form submissions in logging is a similarly awful practise. I really don't see what the difference is.
- munk-a 8y ago
  I like logging and at our company we blanket log all GET parameters on every request, similarly I think it's a Good Idea™ to never log POST parameters, but there are some pages where we've started logging some whitelisted ones because they can help diagnose some errors we run across, but, in an environment where all data is coming into the server in JSON blobs attached in request bodies then it can be a bit harder to make any sort of blanket whitelist.
- ApolloFortyNine 8y ago
  Earlier in the thread: >Because there's two types of storing passwords in plain text. There's the "your password is stored in plaintext in the database" way which everyone agrees is 10 different kinds of stupid, and then there's the "we accidentally logged the body of all requests that went through this system, and it turns out login requests came through here" kind. Basically the conscious decision to store passwords in plain text is worse than unintentionally doing so in logs. One is purposeful, one is not. Yes it's bad, but it's not as bad as if it was done on purpose. And generally, this how most laws are enforced or implemented.
  - untog 8y ago
    Can't both be attributed to ignorance? You could store passwords in plaintext in a database because you didn't know better. You can store plaintext passwords in logs because you don't know better. If I store passwords in plaintext am I being purposeful or am I being ignorant of best practises? I think people are drawing a distinction where one doesn't really exist.
    - ApolloFortyNine 8y ago
pmontra 8y ago
I'm working with Rails and Phoenix on two different projects right now and all of them filter out passwords in the log files [1] [2]. I'm also working on a Django project but we're not logging HTTP calls arguments there. I think we could use a filter like [3] but I'd rather have the framework to automatically take care of that. And if I'd be writing a web app from scratch, no matter if I've been doing web for 25 years, I'm sure I'll do a lot of silly mistakes. That's why I prefer to build on top of frameworks. [1] https://guides.rubyonrails.org/configuring.html#rails-general-configuration https://guides.rubyonrails.org/configuring.html#rails-genera... [2] https://hexdocs.pm/phoenix/Phoenix.Logger.html https://hexdocs.pm/phoenix/Phoenix.Logger.html [3] https://djangosnippets.org/snippets/2966/ https://djangosnippets.org/snippets/2966/

EnFinlay 8y ago

You're right, the impact is just as bad. I think people in this thread (including myself) are downplaying it because it's a very easy mistake to make, rather than a terrible design decision.

newaccoutnas 8y ago
A mistake that should have been found very quickly, given the eyes. Not for years without ever apparently being known (I find that hard to believe)
staticassertion 8y ago
The major difference is that we don't have standard patterns for protecting against one attack, vs the other. With password storage everyone knows the patterns, or we expect them to. With everything between the request and password storage, we don't. This type of attack could easily be prevented. When secrets come in, immediately store them (ideally at the web framework level) in a type that overrides print/debug formatting. Then add a "get_raw" to it, and you can now grep for that being used anywhere outside of storage to a DB (and your DB libs should take the Secret type too). Or don't use a `get_raw` and instead use a `hash` method that returns a safely hashed version of it. Further, your secret type could at the very least add a round of SHA256, maybe even with a pepper?, just to be sure. This isn't hard, I've done it before. The problem is that it isn't something that people feel embarrassed not to do, vs storing plaintext creds. Impact is the same - creds are plaintext in a DB. Attackers always expect sensitive data in logs, so it isn't as if you'd get lucky and they'd miss this.
- deleted 8y ago
  [deleted]
- aeorgnoieang 8y ago
  > When secrets come in, immediately store them (ideally at the web framework level) ... But, in practice, a lot of this stuff is being logged by things like proxies that could be several layers in front of the "web framework level".

hak8or 8y ago

The diffirence is intent and/or gross negligence. Storing passwords in plain text in a database requires a gross lack of security mentality in both design stages and implementation changes. It is also drilled into people's brains constantly about how bad of an idea it is. To put it simply, it cannot happen accidentally. Logging like this can easily be attributed to an accident. The person who implemented this logging should get hit with some repercussions because he surely tested the logging and must have seen the passwords when glancing by eye. But other than that, this was clearly a minor oversight.

zrobotics 8y ago
How is this in any way better or more excusable? Calling it a 'minor oversight' could also apply to the DB being in plaintext. Storing passwords in a plaintext DB heppes in the same manner-a dev is lazy or ignorant of the security reprocussions. Which is what happened here; barring evidence this was done maliciously we can assume that this was accidental. But that doesn't make it any more excusable, since it should be clear that logs can also contain sensitive data that needs to be protected/anonymized. Considering how selective FB is for hiring, I would hope we could expect a higher standard.
- glenneroo 8y ago
  Not to mention 200-600 million users had their passwords exposed for many years. That must be a massive trove of log files.
lacey 8y ago
You assume that storing passwords in plaintext is intent as opposed to gross negligence. In many past instances it’s been intentional and gross negligence because the people making all the design and implementation choices were not knowledgeable about best practices.
Aeolun 8y ago
I see this differently. If someone stores passwords in plaintext in the database, they’re idiots that don’t know any better. If someone logs your password in plaintext but has it encrypted in the database that’s grossly negligent.