4 ms·

Because we all know we are one bad morning away from doing it ourselves. People are distinguishing between extreme incompetence (storing plaintext passwords in

Because we all know we are one bad morning away from doing it ourselves. People are distinguishing between extreme incompetence (storing plaintext passwords in databases) vs people trying to do their jobs. It's the same reason when there are large outages, where the comments are split between the enraged customers, and then the ones that are "Man, sucks to be them" as know they could easily have been the one that got the config push wrong.

908087 8y ago
If a single developer is able to push something like this to production without scrutiny in "one bad morning" on a platform with billions of users, the company itself is the problem.
kitsune_ 8y ago
The fact that sensitive data like this is just one config push away from being exposed means it's a brittle design (not just the code, but the entire design /review / deployment process - this is a platform with 1 billion users, not your 1-dev WordPress webpage). That said, this is further aggravated by the simple fact that for whatever reason, 20k of their employees had access to these logs. All in all it just paints a picture of an irresponsible company where their house is not in order.
- lostapathy 8y ago
  Exactly. It's one thing not to have enough controls in place to catch something like this on the forum for your warcraft guild. It's another thing entirely when you operate at the scale of Facebook.
untog 8y ago
"people are distinguishing between extreme vs people trying to do their jobs" Those things are in no way exclusive. Storing passwords in plaintext is awful practise. Including the contents of form submissions in logging is a similarly awful practise. I really don't see what the difference is.
- munk-a 8y ago
  I like logging and at our company we blanket log all GET parameters on every request, similarly I think it's a Good Idea™ to never log POST parameters, but there are some pages where we've started logging some whitelisted ones because they can help diagnose some errors we run across, but, in an environment where all data is coming into the server in JSON blobs attached in request bodies then it can be a bit harder to make any sort of blanket whitelist.
- ApolloFortyNine 8y ago
  Earlier in the thread: >Because there's two types of storing passwords in plain text. There's the "your password is stored in plaintext in the database" way which everyone agrees is 10 different kinds of stupid, and then there's the "we accidentally logged the body of all requests that went through this system, and it turns out login requests came through here" kind. Basically the conscious decision to store passwords in plain text is worse than unintentionally doing so in logs. One is purposeful, one is not. Yes it's bad, but it's not as bad as if it was done on purpose. And generally, this how most laws are enforced or implemented.
  - untog 8y ago
    Can't both be attributed to ignorance? You could store passwords in plaintext in a database because you didn't know better. You can store plaintext passwords in logs because you don't know better. If I store passwords in plaintext am I being purposeful or am I being ignorant of best practises? I think people are drawing a distinction where one doesn't really exist.
    - ApolloFortyNine 8y ago
      >You can store plaintext passwords in logs because you don't know better. This whole conversation is about how it likely was not done on purpose, and that it was the result of logging HTTP requests and responses. If they wrote logger.info(password), yea I'd say that's as outrageous.
pmontra 8y ago
I'm working with Rails and Phoenix on two different projects right now and all of them filter out passwords in the log files [1] [2]. I'm also working on a Django project but we're not logging HTTP calls arguments there. I think we could use a filter like [3] but I'd rather have the framework to automatically take care of that. And if I'd be writing a web app from scratch, no matter if I've been doing web for 25 years, I'm sure I'll do a lot of silly mistakes. That's why I prefer to build on top of frameworks. [1] https://guides.rubyonrails.org/configuring.html#rails-general-configuration https://guides.rubyonrails.org/configuring.html#rails-genera... [2] https://hexdocs.pm/phoenix/Phoenix.Logger.html https://hexdocs.pm/phoenix/Phoenix.Logger.html [3] https://djangosnippets.org/snippets/2966/ https://djangosnippets.org/snippets/2966/

908087 8y ago

If a single developer is able to push something like this to production without scrutiny in "one bad morning" on a platform with billions of users, the company itself is the problem.

kitsune_ 8y ago

The fact that sensitive data like this is just one config push away from being exposed means it's a brittle design (not just the code, but the entire design /review / deployment process - this is a platform with 1 billion users, not your 1-dev WordPress webpage). That said, this is further aggravated by the simple fact that for whatever reason, 20k of their employees had access to these logs. All in all it just paints a picture of an irresponsible company where their house is not in order.

lostapathy 8y ago
Exactly. It's one thing not to have enough controls in place to catch something like this on the forum for your warcraft guild. It's another thing entirely when you operate at the scale of Facebook.

untog 8y ago

"people are distinguishing between extreme vs people trying to do their jobs" Those things are in no way exclusive. Storing passwords in plaintext is awful practise. Including the contents of form submissions in logging is a similarly awful practise. I really don't see what the difference is.

munk-a 8y ago
I like logging and at our company we blanket log all GET parameters on every request, similarly I think it's a Good Idea™ to never log POST parameters, but there are some pages where we've started logging some whitelisted ones because they can help diagnose some errors we run across, but, in an environment where all data is coming into the server in JSON blobs attached in request bodies then it can be a bit harder to make any sort of blanket whitelist.
ApolloFortyNine 8y ago
Earlier in the thread: >Because there's two types of storing passwords in plain text. There's the "your password is stored in plaintext in the database" way which everyone agrees is 10 different kinds of stupid, and then there's the "we accidentally logged the body of all requests that went through this system, and it turns out login requests came through here" kind. Basically the conscious decision to store passwords in plain text is worse than unintentionally doing so in logs. One is purposeful, one is not. Yes it's bad, but it's not as bad as if it was done on purpose. And generally, this how most laws are enforced or implemented.
- untog 8y ago
  Can't both be attributed to ignorance? You could store passwords in plaintext in a database because you didn't know better. You can store plaintext passwords in logs because you don't know better. If I store passwords in plaintext am I being purposeful or am I being ignorant of best practises? I think people are drawing a distinction where one doesn't really exist.
  - ApolloFortyNine 8y ago
    >You can store plaintext passwords in logs because you don't know better. This whole conversation is about how it likely was not done on purpose, and that it was the result of logging HTTP requests and responses. If they wrote logger.info(password), yea I'd say that's as outrageous.

pmontra 8y ago

I'm working with Rails and Phoenix on two different projects right now and all of them filter out passwords in the log files [1] [2]. I'm also working on a Django project but we're not logging HTTP calls arguments there. I think we could use a filter like [3] but I'd rather have the framework to automatically take care of that. And if I'd be writing a web app from scratch, no matter if I've been doing web for 25 years, I'm sure I'll do a lot of silly mistakes. That's why I prefer to build on top of frameworks. [1] https://guides.rubyonrails.org/configuring.html#rails-general-configuration https://guides.rubyonrails.org/configuring.html#rails-genera... [2] https://hexdocs.pm/phoenix/Phoenix.Logger.html https://hexdocs.pm/phoenix/Phoenix.Logger.html [3] https://djangosnippets.org/snippets/2966/ https://djangosnippets.org/snippets/2966/