15 ms·
Robinhood Stored Passwords in Plaintext
Just received this email from Robinhood (https://robinhood.com/):
"When you set a password for your Robinhood account, we use an industry-standard process that prevents anyone at our company from reading it. On Monday night, we discovered that some user credentials were stored in a readable format within our internal systems. We wanted to let you know that your Robinhood password may have been included.
We resolved this issue, and after thorough review, found no evidence that this information was accessed by anyone outside of our response team. Out of an abundance of caution, we still recommend that you change your Robinhood password.
We take matters like this seriously. Earning and maintaining your trust is our top priority, and we’re committed to protecting your information. Let us know if you have any questions–we’re here to help.
Sincerely,
The Robinhood Team"
If you've used Robinhood in the past, it's a good idea to check your emails!
- bdcravens 7y agoThis doesn't mean they were stored in a database. "a readable format within our internal systems" could be log files if they didn't scrub passwords when logging requests.
- xvolter 7y agoThis is what I thought had happened. They've supported MFA for awhile, so simple enough to reset a password and make sure MFA is turned on. I prefer when companies are proactive.
- trianglesphere 7y agoI agree that it's likely a log file. I remember hearing about Twitter doing this (I also just found out that github may have done this as well). https://www.bleepingcomputer.com/news/security/twitter-admits-recording-plaintext-passwords-in-internal-logs-just-like-github/ https://www.bleepingcomputer.com/news/security/twitter-admit...
- stevezsa8 7y agoI worked at a place where on incorrect login, we recorded your attempt (with whatever was in the user/password field) into the logs. The logs went to data warehouse too. That was a fun day in the office :P
- sofaofthedamned 7y agoExactly. They've got Kibana or similar setup with uri traces, tonnes of companies have been caught up with this same issue.
- sroussey 7y agoNot palantir, their logging system defaults to not writing unknown values.
- stcredzero 7y agoI wonder if there couldn't be some form of automated tooling which creates dummy accounts in your system and looks for leaks of this type?
- ezekg 7y agoThat's a really interesting idea. Generate unique tokens e.g. for detecting leaked passwords and alert upon any text match within database, logs, etc.
- stcredzero 7y agoIt wouldn't just be for passwords. It would be for any kind of confidential data.
- popotamonga 7y agoSet an one time url as a password. Someone accesses it bam.
- slenk 7y agoAzure has this built into some of their DLP...at least when looking at emails
- dmix 7y agoThe detector would have to run everywhere, not just on your web application. It'd have to check linux/nginx/apache/etc logs, metric systems transit pipes and storage, etc, etc. I guess you could install it as a Linux system which searches through all plaintext traversing the network AND filesystems of the entire infrastructure. Which might be a good business idea if you could pull it off.
- losvedir 7y agoMy first guess, too. I flagged the story for the misleading headline, but alternatively an admin could change it to "Robinhood security incident" or something to that effect.
- markdown 7y ago> I flagged the story for the misleading headline, but alternatively an admin could change it to "Robinhood security incident" or something to that effect. Why is the headline misleading? I mean, they say in black and white that they stored passwords in a readable format. The headline is completely factual. If my password gets compromised, it's irrelevant where Robinhood stored it, just that they did so in plain text. That it was stored in text files, in a database, or handwritten on pink post-it notes doesn't make much difference.
- deleted 7y ago[deleted]
- Deimorz 7y agoYep, probably the same thing that happened with Facebook recently: https://newsroom.fb.com/news/2019/03/keeping-passwords-secure/ https://newsroom.fb.com/news/2019/03/keeping-passwords-secur...
- ecnahc515 7y agoThis has happened to basically every large web app company. Turn on debug logging in the app which logs HTTP request headers and likely doesn't strip out sensitive information. Easy mistake, and hopefully it wasn't like this since the beginning, and was maybe only found to be an issue for a subset of Robinhood's logs or something.
- yread 7y agoplain text passwords in HTTP request headers? Wouldn't that be usually in POST body which almost never gets logged?
- dharmab 7y agoHeaders often include session tokens, though
- jrockway 7y agoWhile careless and problematic, that is less problematic than logging passwords. The user probably reuses their password, so as soon as some developer can just look at logs and get a bunch of email/passwords, they probably now have access to other systems that they shouldn't. That said, there are more than just logs to worry about. Is your service-to-service traffic encrypted? Then passwords are in the tcpdumps that you took to analyze a strange networking bug (you then copied the password to your workstation to look at the trace in wireshark). Do your programs segfault and core dump? Then passwords are probably in that core dump. Do you check the bounds of every memory read and write operation? Then passwords are probably in random variables (see: Heartbleed/Cloudbleed). Sessions, though, are less scary. Sure, if you steal someone's cookie internally, you can impersonate that user on your own services. Nobody wants that, because it probably bypasses all the internal auditing systems; the auditing system can't tell that request apart from a legitimate user request. But, cookies can be revoked and have a shorter lifetime than a passwords, so it's not quite as bad. Ultimately, it's all about limiting risk. If you have a database full of passwords, then someone compromising that database today gets all passwords ever. If you have a disk full of logs full of passwords, someone gets all the passwords that were used to log in within that log server's retention time period. If someone hacks in and starts tcpdumping your internal network and it's not encrypted, they only get passwords from users that log in while they're running tcpdump. If you encrypt network traffic, then someone only gets passwords that were in memory when your binary crashed. Nothing is perfect, but you can do more to add more security. Perfection is impossible, but not logging passwords is one step closer to perfection.
- mnem 7y agoI'm not an authentication system programmer, so this may be a silly question, but why do clients still send passwords to a server? Doesn't it make more sense to hash the username and password together with some sort of nonce/salt that's sent to the server for validation?
- dwheeler 7y ago> why do clients still send passwords to a server? Doesn't it make more sense to hash the username and password together with some sort of nonce/salt that's sent to the server for validation? You must ALWAYS hash the password (as a salted iterated cryptographic hash) on the server. You can in ADDITION also hash the password on the client end, but by itself that's not enough. The issue, as usual, comes down to "what is the attack you're trying to thwart"? If you hash on the client and not the server, then when (not if) the attacker manages to download the hashed password set, the attacker can create a modified client & send those hashes directly. If you don't hash on the server, it's exactly the same as storing passwords as clear text, because you're storing the data that an attacker can directly use to log in. You can also hash the password on the client IN ADDITION to the server. In that case, you're hiding from the server the actual password you type in. That's an improvement if you're sharing a single password across many services; in this case the attack you're trying to thwart is to prevent an attacker from actively capturing the password & trying to reuse the password on other systems. However, a much better idea is to not share a single password anyway, so it's not such a great thing.
- forty 7y agoUsers reuse passwords. The main reason of not having your passwords in clear text is to protect them from that. At least when (not if as you noticed) your site is hacked (the same day this attacker download your password database probably), it has no other impact for your users that whatever can happen on your site (the attacker might be able to use the hashed passwords to log on your site but at this point you probably have bigger problems, given that someone is able to download your password database, and you are probably going to reset all those passwords anyway).
- rdtsc 7y agoLogs or audit trace is the first thing I thought of as well. Have seen that happen multiple times. And yeah, logs are stored somewhere so the headline still makes sense (logs are "stored" on disk). Not that it makes it a good thing, but it's a bit of a different story than what is implied.
- pbreit 7y agoI would agree. Although technically if they were accidentally in a log (the likely scenario) that does mean they were being stored.
- dmix 7y agoThe distinction still matters and I give Robinhood credit for admitting it and immediately notifying their users. Security is a long term investment and it's easy to mess up. Taking responsibility and being transparent when you don't is the only way you're going to get it right. There are some reverse proxies that are being sold as a way to mirror HTTP data to metric/analytics systems, which would bypass built-in filtering mechanisms which automatically remove passwords from logs (as Rails does). Metric/big data systems are an easy place for sensitive data leaks to happen. So your web developers could be getting security 100% right but some higher level systems integration with an analytics or ops software (requested by marketing or BI guys or whatever) could mess it up.
- noobermin 7y agoPerhaps it would be helpful to somehow modify the title to reflect this. It doesn't seem at all that Robinhood is being negligent here.
- foobiekr 7y agoThe version I got said the following: >>> When you set a password for your Robinhood account, we use an industry-standard process that prevents anyone at our company from reading it. Additionally, our policy is to not store usernames and passwords for any bank accounts that you link to your Robinhood account. On Monday night, we discovered that some user credentials were unintentionally stored in a readable format within our internal systems. We wanted to let you know that your Robinhood password and linked bank account credentials may have been included. <<< What level of incompetence is required to have accidentally logged bank account credentials? This is not simple logging of login, oops, this is a very serious breach of basic security.
- deleted 7y ago[deleted]
- HenryBemis 7y ago+1 on that. I remember on a bank's IVR I was auditing, required "_random 2 out of your 4 characters of a PIN_" (not your card's but dedicated to a call center support - that gave you full access to your accounts though) was being recorded/logged together with any phone key press (e.g. your customer ID). It was possible (if obtaining the logs of a few of a customer's calls) to reconstruct the PIN (e.g. call1#1 you press 1, 3, call#2 you press 1,2, call#3 you press 2,4). Those logs were scrubbed/dropped once a year, so for a frequent IVR user it was easy to reconstruct your IVR PIN. The folder containing the logs was a Windows shared folder with Everyone/Everyone (share/modify). You should have seen the COO's face as I was describing how easily this could have been abused (likelihood), and how it could open a floodgate of serious problems for the bank (impact).
- amingilani 7y agoAnd this is why I still use an old framework like Ruby on Rails in 2019. Highly opinionated, and old, but rock solid and not easy to screw up.
- vectorEQ 7y agogood point. it would still be silly for 'industry standard methods' to log passwords, in plain text - even for debugging purposes. But it could be result of some bug or error which dumped a load of information ,or perhaps even crashdumps of processes. they contain memory ,which could contain passwords if for instance, the login handling service crashed. It would be useful if companies were a little more clear about how these things happen, it'd be a good learning point for readers with similar setups in their environments. since the issue was resolved, it could be 'responsible disclosure'.
- jchw 7y agoThis is yet another reminder: Do Not Reuse Passwords. There’s good password management even for mobile operating systems these days. I’m using Bitwarden on iOS, and it integrates well both with native apps and webpages. Also, use two factor authentication, of course. If you are going the U2F route, I highly recommend having a permanent key for each computer and a bluetooth+NFC key for on-the-go. It’s a worthwhile investment. My biggest problem today is that it is tedious to generate and save new passwords on mobile, which is increasingly where I do so... but security always comes with some costs, I suppose.
- tmikaeld 7y agoThe convenience has come a long way on most password managers though, I started with lastpass when it was released and it would loose a few generated passwords now and then and yubikey would fail. I've been using bitwarden for a few years and I'm more than satisfied now, it just works as expected.
- hunter2_ 7y agoLose a few? Is that like when a password stored by a browser no longer autofills because the website changed the field IDs or the subdomain or whatever else -- not lost but also not being used without digging it up?
- AdmiralAsshat 7y agoI can also attest to LastPass once "losing" a password. My memory of the event is hazy, but it went something like: 1) Logged into website with LastPass 2) Had to enter a second piece of identifying info (like a PIN). 3) Once entered, LastPass thought the PIN was a new password and prompted me, "Would you like to update the login to [url] with the new password?" 4) I clicked "No". 5) Some kind of network hiccup or communication failure happened, the pop-up kinda hung there for a bit. 6) I opened the vault and checked the password for that site. LP had apparently not honored my request and replaced the password and, more disturbingly, the password history was missing. Luckily I had my phone on me, and evidently it hadn't synced with LP's servers in the last few minutes, so I was able to pull out the old version of the password from the cached version of the fault and manually re-enter it on my computer to get the password fixed. I've never been able to reproduce the issue after that (although I'm not keen to try). But it did happen. LP also used to be pretty terrible at automatically saving passwords you entered into a form, even if you explicitly used the random password that LP generated. You really had to copy+paste it into notepad just as a backup until you could successfully confirm that it made it to the vault.
- jacquesm 7y agoSimilar things have happened to Facebook and Google recently, the two companies that people keep repeating have the best security teams in the world. As long as humans are involved in programming - something that will likely be the case for the foreseeable future - these things will happen. But props to them for owning up about it and promptly mailing users pro-actively.
- hunter2_ 7y agoGenuine question: how would the non-involvement of humans make such issues unlikely? Doesn't the machine learning process involve trying (which, in the case of programming, would probably mean deploying to some users) what is quite certainly a combination of successful and unsuccessful things, to see what sticks?
- jacquesm 7y agoI would assume, perhaps in error, that when such a system is created that it could be fed enough rules about what not to do that such oversights would not happen. The typical error is 'programmer debugging something will serialize a struct to a log file without realizing that there are critical fields in the struct'.
- theshadowmonkey 7y agoIsnt it so convenient that they announce this after they close their funding round.
- wesammikhail 7y agoMy thoughts exactly. How the hell do financial applications not take security more seriously? I just don´t understand. It isn´t that hard to make security a top priority. It isn´t even that expensive in comparison to the price they pay for issues like these, yet it seems that time after time, fast growth and dumping shares onto new VCs or public market investors takes priority over all else...
- hangonhn 7y agoBecause the punishment for such lapses aren't punished financially. The companies aren't generally held liable for damages resulting from such leaks. I'm not agreeing with the status quo but that's how it's been.
- zrm 7y ago> How the hell do financial applications not take security more seriously? This is what taking security more seriously looks like. The lazy company doesn't even bother to look for problems like this, never finds them, and then an attacker eventually gains access to the plaintext passwords and compromises their customers. The shortsighted company finds the problem and fixes it silently, even though they should really notify users to change their passwords to mitigate the possibility that the plaintext passwords were already compromised. The company that takes security more seriously does own up to it despite the PR hit.
- Shatnerz 7y agoYeah, at least they notified customers. I found a similar issue issue at a financial services company (money lending) where I previously worked as a junior dev. A dev accidentally added a log statement to debug something that made it to production. To make matters worse, the logs were also sent off to 3rd party log aggregator that we used and all devs had access to. The company refused to do anything. No emails sent, not even a forced password reset. The dev who made the mistake responded with "This is not a real concern. I am disappointed we spent so much time working on this." I brought it up with the CTO who essentially did nothing. Then I brought it up with CEO who came to our standup where the responsible dev than said something along the lines of "we don't serve any heads of state, so it doesn't really matter." CEO did nothing. I emailed the general counsel who told me no one else brought it up with him. I think I gave notice 2 weeks later. The general counsel apparently left within a year (not sure if related).
- ecnahc515 7y agoSo they probably have a request logger that logs request headers, and they accidently were logging credentials from those headers is probably what happened. This has hit basically every large web service at some point. Crazy this never came up in an earlier internal security audit, but not surprising it occurred in the first place.
- huntermeyer 7y agoFiltering those parameters should be a fundamental practice.
- codezero 7y agoIf you ever run a service, make sure you're not storing nginx logs with the contents of POST requests. A few years later you'll realize you "stored passwords in plaintext" I assume this is what happened here, but maybe Robinhood will elaborate at some point.
- papito 7y agoMan, I would spend time masking sensitive data in a shop with no traffic, but someone like Robinhood or Facebook can get away with it. They don't sweat the small stuff, do they?
- gilbertmpanga12 7y agoLooks like storing in text is now a trend will do this in rest of my projects.
- shawnjanas 7y agoProbably server logs / http request handler console log
- Havoc 7y agoI smell weapons grade bullshit. >On Monday night, we discovered Because one casually does security audits on a Monday night and then releases a "nothing has come to our attention" statement? The one is proactive (on a monday night?) the other speaks of a response to an external actor. Which is it? The mere fact that a release like this happened suggests some sort of legal/SEC/accounting requirement was triggered. i.e. Something happened.
- luhn 7y agoI think it's plausible. As other commenters have noted, the leak was most likely in application logs. So on Monday night, SRE gets paged about random issue, pulls up the logs and starts debugging, ends up stumbling across a user's password.
- snek 7y agoI got an email this morning that my hulu had been logged into... my hulu and my robinhood did in fact share a password. I have no evidence that these are connected, but better safe than sorry. (And I do use 1password now, I just didn't back when I used robinhood and hulu).
- Scaevolus 7y agoHave you checked HaveIBeenPwned? for your email? If you shared a password between Hulu and Robinhood in your pre-lastpass days, you probably used it on yet another site that was hacked.
- beatgammit 7y agoThe problem is that it says I've been pwned, but it's completely unhelpful because it doesn't say which service(s) it was. I use different passwords for most sites (randomly generated via password manager), though a few unimportant ones share a common password. I use Bitwarden, which has a way of checking password dumps, but only for individual passwords. I have over a hundred entries, so checking that frequently is quite time consuming. I've decided to just switch my email on as many sites as possible using a few aliases to make put them into buckets, but there has to be an easier way...
- ebg13 7y ago> The problem is that it says I've been pwned, but it's completely unhelpful because it doesn't say which service(s) it was This is the most frustrating thing about hibp. I have hundreds of passwords on hundreds of services. "we found your email in a dump" doesn't help me if I don't know which server you found it for.
- mehrdadn 7y agoNot surprised, considering their security practices... see https://news.ycombinator.com/item?id=15679099 https://news.ycombinator.com/item?id=15679099
- hartator 7y agoI think it was some kind of logs. Probably request logs.
- blattinum 7y agothanks for the heads up. I got the same email earlier.
- 2_listerine_pls 7y agoIt's hard to believe a financial services company would store passwords in plain text out of stupidity. Could this be a legal strategy to avoid responsibility in some scenario?
- all_blue_chucks 7y agoThe fact that they prompt you for your literal bank password when funding your account says everything you need to know about how they look at security.
- asdff 7y agoThat's true of nearly every app I've used that links to your bank account (mint and venmo off the top of my head). Super archaic and obtuse and I solely blame the banks for not building a better method.
- sroussey 7y agoThe EU is forcing banks to work together and with FinTech. Have a look at PSD2 and the open banking regulations.
- niij 7y agoThey could just ask for routing/account #. Yes it allows for unauthenticated push & pulls, but when they ask for username/password they're just scraping the routing/account number from your account and doing a normal ach transfer anyway. Except now that they're logged in they can verify your account has X funds, see transaction history, and be yet another security breach waiting to happen. Transferwise switched to requiring this and it's a miserable feature. Lost me as a customer because of it.
- all_blue_chucks 7y agoACH transfers do NOT require you to give your password to anyone, and they have been around forever. Never give your bank password to anyone other than your bank.
- astura 7y agoBut ACH transfers require you to give the information needed to print out checks against your bank account (routing number and account number).
- astura 7y ago
- huxflux 7y ago"take matters like this seriously", not really.
- foobiekr 7y agoPlaintext bank account credentials. Wow.
- dumbeldore 7y agoThe title is so misleading....
- pgl 7y ago> "we ... recommend that you change your Robinhood password" This should really be a forced reset. Not finding evidence that it was accessed isn't proof that it wasn't.
- paulie_a 7y ago"found no evidence that this information was accessed by anyone outside of our response team. Out of an abundance of caution, we still recommend that you change your Robinhood password." That is complete bullshiting pr spin and I'm greatly concerned about my account. Gee wiz I wonder if anyone on the response team could have done something. Or a dev with nefarious purposes like guy that rigged the lottery multiple times.
- garryc1304 7y agoGarry cole GREETINGS EVERYONE, are you looking for a LEGIT and Trustworthy HACKERS with 100% Guarantee and you want to get your job done urgently withing one Hour or you are face with delay and unnecessary excuses and error on your job?. Then Worry no more because easyhackingguru@gmail.com are the Best Bet in any hacking Services. They are ready to render and attend to your job with swift response and No delay at all. Their services are outlined as follows: . LONG TIME LOAN GIVING . PROFESSIONAL in SCHOOL GRADE changing . WHATSAPP Hack . FACBOOK hack . PROFESSIONAL in any BANK ACCOUNTS TRANSFER . TWITTERS hack . EMAIL,YAHOOMAIL and HOTMAIL ACCOUNTS hack . WEBSITE CRASHED hack . SERVER CRASHED hack . SALES OF SPYWARE and KEYLOGGER SOFTWARE . RETRIVAL OF LOST FILES and DOCUMENTS . ERASE and EXPUNGE of CRIMINAL RECORDS . DATABASE hack . SALES of ATM CARDS in WHITE . SKYPE hack . PAYPAL hack . DROPBOX ACCOUNT hack and Lots more.......... CONTACT: their services at easyhackingguru@gmail.com and you will be glad you did whatsApp contact- +1(216)525-9704