5 ms·

How is something like this even possible? It just sounds so incompetent.

by 77pt77 9y ago

deleted 9y ago
[deleted]
thln666 9y ago
seriously? pretty easily. somebody probably left a debug log message in place or something. guaranteed that this happens all the time and most people don't report it.
- weavie 9y ago
  Indeed. This is probably a good reminder for every developer to just go and check through their logs to see what is there. It can be quite a shock sometimes to find how much can get dumped there..
- jonjojr 9y ago
  As developer, I can tell you this happens more often than I'd like to admit. debug logs is that necessary evil you need to troubleshoot pesky bugs. Unfortunately some of these debug tools need to be turned on in a live environment to capture those logs for debugging. But also Unfortunately, we are humans and we concentrate on fixing the bug and forget to turn off logging or log unnecessary data.
- chatmasta 9y ago
  I doubt anyone left something that logged the plaintext password. No reasonable architecture necessitates holding onto a plaintext password for more than one line of code. One possibility is an HTTP server on the request path after TLS termination. But then why is an HTTP server logging the request body? My guess would be some sort of instrumentation process was blindly reading data in memory without distinguishing what the data was, but produced logs that incidentally included passwords.
  - miragle 9y ago
    many reasonable architectures funnel the request body through multiple middlewares before it reaches the code that actually needs the password. there are so many places that request data can be logged in any reasonable system. this is completely ordinary. it's not like there's a special HTTP mechanism for transmitting passwords that keeps it out of the normal code flow. it's in the request. if you log requests, you're probably going to end up accidentally logging something you're not supposed to at some point.
  - testplzignore 9y ago
    In my experience, I've seen both of the following scenarios: POST request comes in from the client. Full URL and request body is logged. Sometimes for simply troubleshooting, sometimes for security reasons (e.g., wanting to know all data coming in so that it's possible to identify security holes after they've been exploited). POST request comes in from client. Frontend server makes a GET request to a backend server, and the password ends up in the standard request logs. In one case, I've seen this happen because the developer thought path variables were cool, so every API they wrote looked like /a/b/c/d/e. Sigh.
- bdamm 9y ago
  Absolutely this happens all the time. I personally have seen it happen twice at two different companies.
leggomylibro 9y ago
HTTPS form submissions should be encrypted while the data travels between the user's computer and the server, but the server will still need to decrypt them to perform the hashing. It's possible, and probably even common, for inexperienced or forgetful developers to add request logging for debugging or diagnosing service outages without adding extra logic to scrub sensitive fields.
- deleted 9y ago
  [deleted]
eterm 9y ago
Here's one potential scenario?: Over-zealous developers who think it's appropriate to log all function calls with parameters for trace level logs, or a framework with the same opinion automatically applies such tracing and logging over the whole code-base. No-one notices because no-one uses trace level logging, until one day another developer is tearing their hair out because they can't reproduce a bug that is only occuring on live. It's an urgent bug that needs resolution asap. So this developer turns on the trace level logging and eventually finds and resolves their bug. Being the careful person they are, they turn off the logging and go away happy. Meanwhile they've unknowingly produced a few gigabytes of log outputs which happen to include plaintext passwords. That's just one of many different scenarios where people acting in 'good faith' can still lead to bad outcomes. That is why a "PUNISH THEM!" attitude to this kind of incident is not helpful.
icedchai 9y ago
This happens constantly. You'd be surprised what's recorded in logs...
Jach 9y ago
I'm curious if anyone has details on using bcrypt/scrypt at scale. Specifically one way I could see this happening is something like login requests go to a load balancer that puts the requests on a queue to be picked up and validated by some hasher service, and the queue ends up writing the requests to logs to recover from certain kinds of failures.
- panarky 9y ago
  Is it insecure to bcrypt/scrypt on the client instead so the server never sees the plaintext password?
  - krallja 9y ago
    Then anyone with a copy of your DB can log in by sending the hash directly. It’s identical to storing plain text passwords.
    - chatmasta 9y ago
      You could hash it twice, once at the client and once at the server. So your database would store a hash of a hash. But without persistence on the client side you wouldn’t be able to do salting in the first hash (where do you store the salt?)
      - stordoff 9y ago
        Does it buy you anything at that point? A server-side issue, such as this, would still log the thing you need to log in, and a client-side issue would just intercept the hashed form or could derive the hashing mechanism from analysing the client. At most, it would seem to prevent weak passwords from being passed directly to bcrypt, but salting should solve that in a similar way anyway, and anyone brute-forcing a copy of the database can incorporate the same weak hashing logic
        chatmasta 9y ago
jonjojr 9y ago
No, it is just poor oversight. Not incompetence. These could be excellent developers that just overlooked a logging tool.
carapace 9y ago
That's exactly what it is: incompetence, rank incompetence. Something like nine out of ten people getting paid today as professional software "engineers" should be let go. Dr. Margaret Hamilton figured out most of what we need to do to develop reliable software during and after the Apollo 11 mission. She coined the term "software engineering". Unfortunately, her work suffered from bad languaging and languished. You'll notice you've been downvoted to hell and the comments in reply to yours are apologists and excuses. Not a coincidence. FizzBuzz

How is something like this even possible? It just sounds so incompetent.

by 77pt77 9y ago

How is something like this even possible? It just sounds so incompetent.

deleted 9y ago

[deleted]

thln666 9y ago

seriously? pretty easily. somebody probably left a debug log message in place or something. guaranteed that this happens all the time and most people don't report it.

weavie 9y ago
Indeed. This is probably a good reminder for every developer to just go and check through their logs to see what is there. It can be quite a shock sometimes to find how much can get dumped there..
jonjojr 9y ago
As developer, I can tell you this happens more often than I'd like to admit. debug logs is that necessary evil you need to troubleshoot pesky bugs. Unfortunately some of these debug tools need to be turned on in a live environment to capture those logs for debugging. But also Unfortunately, we are humans and we concentrate on fixing the bug and forget to turn off logging or log unnecessary data.
chatmasta 9y ago
I doubt anyone left something that logged the plaintext password. No reasonable architecture necessitates holding onto a plaintext password for more than one line of code. One possibility is an HTTP server on the request path after TLS termination. But then why is an HTTP server logging the request body? My guess would be some sort of instrumentation process was blindly reading data in memory without distinguishing what the data was, but produced logs that incidentally included passwords.
- miragle 9y ago
  many reasonable architectures funnel the request body through multiple middlewares before it reaches the code that actually needs the password. there are so many places that request data can be logged in any reasonable system. this is completely ordinary. it's not like there's a special HTTP mechanism for transmitting passwords that keeps it out of the normal code flow. it's in the request. if you log requests, you're probably going to end up accidentally logging something you're not supposed to at some point.
- testplzignore 9y ago
  In my experience, I've seen both of the following scenarios: POST request comes in from the client. Full URL and request body is logged. Sometimes for simply troubleshooting, sometimes for security reasons (e.g., wanting to know all data coming in so that it's possible to identify security holes after they've been exploited). POST request comes in from client. Frontend server makes a GET request to a backend server, and the password ends up in the standard request logs. In one case, I've seen this happen because the developer thought path variables were cool, so every API they wrote looked like /a/b/c/d/e. Sigh.
bdamm 9y ago
Absolutely this happens all the time. I personally have seen it happen twice at two different companies.

leggomylibro 9y ago

HTTPS form submissions should be encrypted while the data travels between the user's computer and the server, but the server will still need to decrypt them to perform the hashing. It's possible, and probably even common, for inexperienced or forgetful developers to add request logging for debugging or diagnosing service outages without adding extra logic to scrub sensitive fields.

deleted 9y ago
[deleted]

eterm 9y ago

Here's one potential scenario?: Over-zealous developers who think it's appropriate to log all function calls with parameters for trace level logs, or a framework with the same opinion automatically applies such tracing and logging over the whole code-base. No-one notices because no-one uses trace level logging, until one day another developer is tearing their hair out because they can't reproduce a bug that is only occuring on live. It's an urgent bug that needs resolution asap. So this developer turns on the trace level logging and eventually finds and resolves their bug. Being the careful person they are, they turn off the logging and go away happy. Meanwhile they've unknowingly produced a few gigabytes of log outputs which happen to include plaintext passwords. That's just one of many different scenarios where people acting in 'good faith' can still lead to bad outcomes. That is why a "PUNISH THEM!" attitude to this kind of incident is not helpful.

icedchai 9y ago

This happens constantly. You'd be surprised what's recorded in logs...

Jach 9y ago

I'm curious if anyone has details on using bcrypt/scrypt at scale. Specifically one way I could see this happening is something like login requests go to a load balancer that puts the requests on a queue to be picked up and validated by some hasher service, and the queue ends up writing the requests to logs to recover from certain kinds of failures.

panarky 9y ago
Is it insecure to bcrypt/scrypt on the client instead so the server never sees the plaintext password?
- krallja 9y ago
  Then anyone with a copy of your DB can log in by sending the hash directly. It’s identical to storing plain text passwords.
  - chatmasta 9y ago
    You could hash it twice, once at the client and once at the server. So your database would store a hash of a hash. But without persistence on the client side you wouldn’t be able to do salting in the first hash (where do you store the salt?)
    - stordoff 9y ago
      Does it buy you anything at that point? A server-side issue, such as this, would still log the thing you need to log in, and a client-side issue would just intercept the hashed form or could derive the hashing mechanism from analysing the client. At most, it would seem to prevent weak passwords from being passed directly to bcrypt, but salting should solve that in a similar way anyway, and anyone brute-forcing a copy of the database can incorporate the same weak hashing logic
      - chatmasta 9y ago

jonjojr 9y ago

No, it is just poor oversight. Not incompetence. These could be excellent developers that just overlooked a logging tool.

carapace 9y ago

That's exactly what it is: incompetence, rank incompetence. Something like nine out of ten people getting paid today as professional software "engineers" should be let go. Dr. Margaret Hamilton figured out most of what we need to do to develop reliable software during and after the Apollo 11 mission. She coined the term "software engineering". Unfortunately, her work suffered from bad languaging and languished. You'll notice you've been downvoted to hell and the comments in reply to yours are apologists and excuses. Not a coincidence. FizzBuzz