9 ms·
This has happened to basically every large web app company. Turn on debug logging in the app which logs HTTP request headers and likely doesn't strip out sensit
by ecnahc515 7y ago
This has happened to basically every large web app company. Turn on debug logging in the app which logs HTTP request headers and likely doesn't strip out sensitive information. Easy mistake, and hopefully it wasn't like this since the beginning, and was maybe only found to be an issue for a subset of Robinhood's logs or something.
- yread 7y agoplain text passwords in HTTP request headers? Wouldn't that be usually in POST body which almost never gets logged?
- dharmab 7y agoHeaders often include session tokens, though
- jrockway 7y agoWhile careless and problematic, that is less problematic than logging passwords. The user probably reuses their password, so as soon as some developer can just look at logs and get a bunch of email/passwords, they probably now have access to other systems that they shouldn't. That said, there are more than just logs to worry about. Is your service-to-service traffic encrypted? Then passwords are in the tcpdumps that you took to analyze a strange networking bug (you then copied the password to your workstation to look at the trace in wireshark). Do your programs segfault and core dump? Then passwords are probably in that core dump. Do you check the bounds of every memory read and write operation? Then passwords are probably in random variables (see: Heartbleed/Cloudbleed). Sessions, though, are less scary. Sure, if you steal someone's cookie internally, you can impersonate that user on your own services. Nobody wants that, because it probably bypasses all the internal auditing systems; the auditing system can't tell that request apart from a legitimate user request. But, cookies can be revoked and have a shorter lifetime than a passwords, so it's not quite as bad. Ultimately, it's all about limiting risk. If you have a database full of passwords, then someone compromising that database today gets all passwords ever. If you have a disk full of logs full of passwords, someone gets all the passwords that were used to log in within that log server's retention time period. If someone hacks in and starts tcpdumping your internal network and it's not encrypted, they only get passwords from users that log in while they're running tcpdump. If you encrypt network traffic, then someone only gets passwords that were in memory when your binary crashed. Nothing is perfect, but you can do more to add more security. Perfection is impossible, but not logging passwords is one step closer to perfection.
- londons_explore 7y agoSession cookies should be locked to a TLS channel ID. Then they are useless to anyone who steals them, since a TLS channel ID can't be recreated except with access to the TPM of the computer which initially created the cookie/session/TLS connection.
- daxterspeed 7y agoSmall enough POST bodies could still be worth logging to detect issues. Of course because of things like this it generally isn't a good idea, but there's still instances where it could be important to log POST bodies to eg. understand certain automated attacks.
- bigiain 7y agoI have, on a not insignificant number of occasions, used mod_dumpio in Apache to debug things. Which means I end up with all POST payloads in log files. Which has occasionally meant I ended up with plaintext passwords in log files... I am extremely careful when doing this. But I have seen cow orkers copy my debugging techniques and "forget" to switch off the dumpio logging - which has at least once been discovered when a prod server's log files started blowing out in size dramatically. With real user passwords in it... (Quite how mod_dumpio and it's config ended up on a prod server is another rant, which needs to be done over whisky, not in the office...)
- avip 7y agohttps://en.wiktionary.org/wiki/cow-orker https://en.wiktionary.org/wiki/cow-orker
- dunham 7y agoThis happened to Apple too, while logging API errors (CVE-2014-1317 - in this case it was hex-encoded). Given the frequency that this occurs, I wonder if it'd be a good practice to use a sentinel value for a password during testing and grep logs for it.
- patrickthebold 7y agoAnd the hex encoded version of it.
- naniwaduni 7y agobase64, too, in all four alignments.
- mikeash 7y agoMaybe we should avoid sending passwords altogether. With some client-aide code, you could build a challenge-response system that eliminates this risk entirely.
- snazz 7y agoHas there been any research into this or any companies that use it yet?
- willsr 7y agoAmazon use it in their Cognito offering.
- lonelappde 7y agoChip-and-pin credit cards use it. 2FA/Fido/Yubikey use it.
- cyc115 7y agoFido (and now w3c)'s WebAuthn come to mind
- deleted 7y ago
- redleggedfrog 7y ago"This has happened to basically every large web app company." That is a seriously sad reflection on the poor quality of the software cranked out today. Someone should be losing their job.
- icedchai 7y agoSure, people should lose their job because someone's diagnosing a network problem, happens to run tcpdump on a server behind a TLS-terminating load balancer, and sees username/passwords. /s This literally happens all the time.
- redleggedfrog 7y agoDepends, how did they get to the server (secure jumpbox)? Did anyone know it'd be a security related event when they were diagnosing the problem? Did the tech take precautions? In such a case, I don't think robinhood.com would need to notify users. They tcpdump would be handled securely, scrubbed properly, troubleshooting done, and that'd be the end of it. If that isn't what happened, someone should be losing their job. The minute you think diagnosing network problems is a reason to ignore security to a degree that your users need to be notified you're incompetent.
- icedchai 7y agoIf they used the info to hack accounts or other malicious activity, then yes, they should lose their job. If they just left a log or capture file around somewhere, nope. Everyone makes mistakes.
- redleggedfrog 7y ago"If they just left a log or capture file around somewhere, nope. Everyone makes mistakes." I guess that's where you and I differ. There are certain mistakes that are tolerable. But ones that result in mass emailing to your clients that they need to change their passwords are not. I'd never heard of robinhood.com before, but now I have, and my impression is I'd never trust them because they have staff that makes mistakes with my secure information. Think of how robinhood.com's employees feel about this. Management, and the people who are now watching the fallout rob the company of revenue. Telling me that the person who made that mistake is gone might make me think differently. The minute you start taking users name and passwords you're in the big boy world and need to treat it as such. Mistakes can be fatal, not just for your job, but your company.
- driverdan 7y agoThat's no excuse, especially for a financial services company like Robinhood. They have a duty to have process and people in place to prevent this exact type of issue.
- islon 7y agoBut isn't it bad practice to send plain text passwords to the backend? Shouldn't you encrypt them in the client already?
- eximius 7y agoThis is why there is client side hashing. You still hash+salt on the server side, but the client side hashing prevents the worst kind of exposure.