12 ms·
This doesn't mean they were stored in a database. "a readable format within our internal systems" could be log files if they didn't scrub passwords when logging
by bdcravens 7y ago
This doesn't mean they were stored in a database. "a readable format within our internal systems" could be log files if they didn't scrub passwords when logging requests.
- xvolter 7y agoThis is what I thought had happened. They've supported MFA for awhile, so simple enough to reset a password and make sure MFA is turned on. I prefer when companies are proactive.
- trianglesphere 7y agoI agree that it's likely a log file. I remember hearing about Twitter doing this (I also just found out that github may have done this as well). https://www.bleepingcomputer.com/news/security/twitter-admits-recording-plaintext-passwords-in-internal-logs-just-like-github/ https://www.bleepingcomputer.com/news/security/twitter-admit...
- stevezsa8 7y agoI worked at a place where on incorrect login, we recorded your attempt (with whatever was in the user/password field) into the logs. The logs went to data warehouse too. That was a fun day in the office :P
- sofaofthedamned 7y agoExactly. They've got Kibana or similar setup with uri traces, tonnes of companies have been caught up with this same issue.
- sroussey 7y agoNot palantir, their logging system defaults to not writing unknown values.
- stcredzero 7y agoI wonder if there couldn't be some form of automated tooling which creates dummy accounts in your system and looks for leaks of this type?
- ezekg 7y agoThat's a really interesting idea. Generate unique tokens e.g. for detecting leaked passwords and alert upon any text match within database, logs, etc.
- stcredzero 7y agoIt wouldn't just be for passwords. It would be for any kind of confidential data.
- popotamonga 7y agoSet an one time url as a password. Someone accesses it bam.
- slenk 7y agoAzure has this built into some of their DLP...at least when looking at emails
- dmix 7y agoThe detector would have to run everywhere, not just on your web application. It'd have to check linux/nginx/apache/etc logs, metric systems transit pipes and storage, etc, etc. I guess you could install it as a Linux system which searches through all plaintext traversing the network AND filesystems of the entire infrastructure. Which might be a good business idea if you could pull it off.
- losvedir 7y agoMy first guess, too. I flagged the story for the misleading headline, but alternatively an admin could change it to "Robinhood security incident" or something to that effect.
- markdown 7y ago> I flagged the story for the misleading headline, but alternatively an admin could change it to "Robinhood security incident" or something to that effect. Why is the headline misleading? I mean, they say in black and white that they stored passwords in a readable format. The headline is completely factual. If my password gets compromised, it's irrelevant where Robinhood stored it, just that they did so in plain text. That it was stored in text files, in a database, or handwritten on pink post-it notes doesn't make much difference.
- deleted 7y ago[deleted]
- Deimorz 7y agoYep, probably the same thing that happened with Facebook recently: https://newsroom.fb.com/news/2019/03/keeping-passwords-secure/ https://newsroom.fb.com/news/2019/03/keeping-passwords-secur...
- ecnahc515 7y agoThis has happened to basically every large web app company. Turn on debug logging in the app which logs HTTP request headers and likely doesn't strip out sensitive information. Easy mistake, and hopefully it wasn't like this since the beginning, and was maybe only found to be an issue for a subset of Robinhood's logs or something.
- yread 7y agoplain text passwords in HTTP request headers? Wouldn't that be usually in POST body which almost never gets logged?
- dharmab 7y agoHeaders often include session tokens, though
- jrockway 7y agoWhile careless and problematic, that is less problematic than logging passwords. The user probably reuses their password, so as soon as some developer can just look at logs and get a bunch of email/passwords, they probably now have access to other systems that they shouldn't. That said, there are more than just logs to worry about. Is your service-to-service traffic encrypted? Then passwords are in the tcpdumps that you took to analyze a strange networking bug (you then copied the password to your workstation to look at the trace in wireshark). Do your programs segfault and core dump? Then passwords are probably in that core dump. Do you check the bounds of every memory read and write operation? Then passwords are probably in random variables (see: Heartbleed/Cloudbleed). Sessions, though, are less scary. Sure, if you steal someone's cookie internally, you can impersonate that user on your own services. Nobody wants that, because it probably bypasses all the internal auditing systems; the auditing system can't tell that request apart from a legitimate user request. But, cookies can be revoked and have a shorter lifetime than a passwords, so it's not quite as bad. Ultimately, it's all about limiting risk. If you have a database full of passwords, then someone compromising that database today gets all passwords ever. If you have a disk full of logs full of passwords, someone gets all the passwords that were used to log in within that log server's retention time period. If someone hacks in and starts tcpdumping your internal network and it's not encrypted, they only get passwords from users that log in while they're running tcpdump. If you encrypt network traffic, then someone only gets passwords that were in memory when your binary crashed. Nothing is perfect, but you can do more to add more security. Perfection is impossible, but not logging passwords is one step closer to perfection.
- mnem 7y agoI'm not an authentication system programmer, so this may be a silly question, but why do clients still send passwords to a server? Doesn't it make more sense to hash the username and password together with some sort of nonce/salt that's sent to the server for validation?
- dwheeler 7y ago> why do clients still send passwords to a server? Doesn't it make more sense to hash the username and password together with some sort of nonce/salt that's sent to the server for validation? You must ALWAYS hash the password (as a salted iterated cryptographic hash) on the server. You can in ADDITION also hash the password on the client end, but by itself that's not enough. The issue, as usual, comes down to "what is the attack you're trying to thwart"? If you hash on the client and not the server, then when (not if) the attacker manages to download the hashed password set, the attacker can create a modified client & send those hashes directly. If you don't hash on the server, it's exactly the same as storing passwords as clear text, because you're storing the data that an attacker can directly use to log in. You can also hash the password on the client IN ADDITION to the server. In that case, you're hiding from the server the actual password you type in. That's an improvement if you're sharing a single password across many services; in this case the attack you're trying to thwart is to prevent an attacker from actively capturing the password & trying to reuse the password on other systems. However, a much better idea is to not share a single password anyway, so it's not such a great thing.
- forty 7y agoUsers reuse passwords. The main reason of not having your passwords in clear text is to protect them from that. At least when (not if as you noticed) your site is hacked (the same day this attacker download your password database probably), it has no other impact for your users that whatever can happen on your site (the attacker might be able to use the hashed passwords to log on your site but at this point you probably have bigger problems, given that someone is able to download your password database, and you are probably going to reset all those passwords anyway).
- rdtsc 7y agoLogs or audit trace is the first thing I thought of as well. Have seen that happen multiple times. And yeah, logs are stored somewhere so the headline still makes sense (logs are "stored" on disk). Not that it makes it a good thing, but it's a bit of a different story than what is implied.
- pbreit 7y agoI would agree. Although technically if they were accidentally in a log (the likely scenario) that does mean they were being stored.
- dmix 7y agoThe distinction still matters and I give Robinhood credit for admitting it and immediately notifying their users. Security is a long term investment and it's easy to mess up. Taking responsibility and being transparent when you don't is the only way you're going to get it right. There are some reverse proxies that are being sold as a way to mirror HTTP data to metric/analytics systems, which would bypass built-in filtering mechanisms which automatically remove passwords from logs (as Rails does). Metric/big data systems are an easy place for sensitive data leaks to happen. So your web developers could be getting security 100% right but some higher level systems integration with an analytics or ops software (requested by marketing or BI guys or whatever) could mess it up.
- noobermin 7y agoPerhaps it would be helpful to somehow modify the title to reflect this. It doesn't seem at all that Robinhood is being negligent here.
- foobiekr 7y agoThe version I got said the following: >>> When you set a password for your Robinhood account, we use an industry-standard process that prevents anyone at our company from reading it. Additionally, our policy is to not store usernames and passwords for any bank accounts that you link to your Robinhood account. On Monday night, we discovered that some user credentials were unintentionally stored in a readable format within our internal systems. We wanted to let you know that your Robinhood password and linked bank account credentials may have been included. <<< What level of incompetence is required to have accidentally logged bank account credentials? This is not simple logging of login, oops, this is a very serious breach of basic security.
- deleted 7y ago[deleted]
- HenryBemis 7y ago+1 on that. I remember on a bank's IVR I was auditing, required "_random 2 out of your 4 characters of a PIN_" (not your card's but dedicated to a call center support - that gave you full access to your accounts though) was being recorded/logged together with any phone key press (e.g. your customer ID). It was possible (if obtaining the logs of a few of a customer's calls) to reconstruct the PIN (e.g. call1#1 you press 1, 3, call#2 you press 1,2, call#3 you press 2,4). Those logs were scrubbed/dropped once a year, so for a frequent IVR user it was easy to reconstruct your IVR PIN. The folder containing the logs was a Windows shared folder with Everyone/Everyone (share/modify). You should have seen the COO's face as I was describing how easily this could have been abused (likelihood), and how it could open a floodgate of serious problems for the bank (impact).
- amingilani 7y agoAnd this is why I still use an old framework like Ruby on Rails in 2019. Highly opinionated, and old, but rock solid and not easy to screw up.
- vectorEQ 7y agogood point. it would still be silly for 'industry standard methods' to log passwords, in plain text - even for debugging purposes. But it could be result of some bug or error which dumped a load of information ,or perhaps even crashdumps of processes. they contain memory ,which could contain passwords if for instance, the login handling service crashed. It would be useful if companies were a little more clear about how these things happen, it'd be a good learning point for readers with similar setups in their environments. since the issue was resolved, it could be 'responsible disclosure'.