10 ms·
Notifying administrators about unhashed password storage
- segmondy 8y agoIn addition, as we were troubleshooting new G Suite customer sign-up flows, we discovered that starting in January 2019 we had inadvertently stored a subset of unhashed passwords in our secure encrypted infrastructure. These passwords were stored for a maximum of 14 days. Is this another way of saying they stored cleartext passwords? Interesting they choose to use "uncashed" instead of "cleartext"
- maxton 8y agoThey are probably not saying "cleartext" because the data was still encrypted, just not hashed.
- U3OSB 8y agoStrange... Didn't realize G Suite and Cloud Platform were well coupled, but it appears they must be. I recently opened two of what I though were new 'consumer' Google accounts to be dedicated to managing Google Cloud Platform resources. On both accounts I'v also setup Identity & Access Management with an organization and additional sub-user-accounts. Both of those parent account received this email warning about five hours ago... Subject: [ACTION REQUIRED] Important Information about your G Suite Account Google Customer Alert Dear G Suite Administrator, We are writing to inform you that between January 13, 2019 and May 9, 2019, an internal system that logged account signup information for diagnostic purposes, inadvertently stored one of your user account passwords in our encrypted systems in an unhashed format. This impacted the user account password provided during the initial account signup process. The log information was retained for 14 days following the signup process, and then was deleted according to our normal retention policies. We have reviewed the login information for the account and have found no evidence that the unhashed password was misused. The following is the user account impacted in your domain(s): [@] Google Planned Action: for your security, starting tomorrow Wednesday, May 22, 2019 PT we will force a password change unless it has already been changed prior to that time. Our password update methodology is as follows: • We will terminate the impacted user’s session and prompt the user to change their password at their next login. • In addition, starting Wednesday, May 29, 2019 PT we will reset the password for the user if they have not yet selected a new password or have not had a password reset. This user will need to follow your organization’s password recovery process. However, Super Admins will not be impacted. For information on password recovery options please refer to the following [Help Center Article]. For further questions please contact [Google Support] and reference issue number ###. Sincerely, The G Suite Team
- solotronics 8y agowould this violate GDPR?
- FDSGSG 8y agono
- brcha 8y agoAs long as data is private, I don't think so. Although, it is pointless to discuss that since keeping passwords in plaintext is, well, :)
- mijoharas 8y agoWow, TIL that strong passwords are actually a part of GDPR (I was aware of the policies around deleting data, being clear about the data that you're storing, but not about passwords). That's actually great that industry standards are being codified with actual deterrents for failing to strongly store secure passwords. > If passwords are being used, they should not be accessible and should be securely stored to avoid them being visible to unauthorized individuals. Some sort of standard such as encryption or an equivalent should be used to protect the passwords. quote from some random blog [0] [0] https://www.compliancejunction.com/gdpr-password-requirements/ https://www.compliancejunction.com/gdpr-password-requirement...
- the_pwner224 8y ago> That's actually great that industry standards are being codified with actual deterrents for failing to strongly store secure passwords. What would be better is for us folks to do this to ourselves first. If self regulation works well, the government will never need to step in. The GDPR seems to be pretty good, but it's not unimaginable that some assholes in office who are looking to get reelected respond to a high-profile data breach by making a shitty set of regulations to govern all software developers in the country for all time. W.R.T. enforcing compliance, having a licensing board that oversees this would provide incentive to developers to not cut corners, and would also provide a way to resist management that is intent on releasing a shitty product quickly in the hope of making a quick buck before it all crashes down / padding their resume.
- regnerba 8y agoThey sure liked repeating "secure encrypted infrastructure". I was a little confused about the timeline of this as they mentioned implementing some functionality back in 2005. If I understand this right the specific issue of storing some (unsure as to what percentage or quantity) users passwords unhashed: - started in January of this year - is now fixed - were stored for up to 14 days in this state - would only effect new years signing up (based on it being part of the signup flow) Do I understand that right?
- teraflop 8y agoMy interpretation was that unhashed passwords were stored for all users whose passwords were manually set by an administrator, from 2005 until some time very recently.
- owaislone 8y agoLooks like two issues were discovered. 1. All passwords set by admins for other accounts since 2005 were impacted. 2. Also all passwords starting Jan 2019 were also stored un-hashed for 14 days.
- jkaplowitz 8y agoI have no inside info on this incident and haven't worked for Google in over 4 years, but the description of the 14-day 2019-only issue sounds like it would have affected only a subset of users whose data they included in a sample for troubleshooting purposes. The sample rate they use is probably far below 100%. (Once again: this is speculation based on general firsthand knowledge about how Google operates, but I know nothing specific to this incident.)
- owaislone 8y agoYes, of course. Not all but a different subset of passwords.
- Kalium 8y ago> They sure liked repeating "secure encrypted infrastructure". Their audience is people who know those words and know that they want them, but might not know what they mean. Using technically accurate details when addressing people who don't understand the vocabulary is generally a recipe for misery.
- rodmena 8y agoThank you Google (AKA fk you Facebook).
- discreditable 8y agoMy domain of 800+ users had one unhashed user. It was an account I created for testing something then suspended over two years ago. It never completed the first logon. I have many newer users that have not completed a first logon, yet their passwords are (apparently) hashed. They said users created through admin console are affected. The might explain my nearly zero count. I've used GAM to automate creation for a very long time. We are a high school school so we churn about 1/4 of our user count every year.
- whalesalad 8y agoI had the same thing happen, only one email. An account for my sister, actually. Turns out she has never logged-in either.
- chmars 8y agoSame here: One user!
- miohtama 8y agoI doubt Google would store passwords unhashed in a database - after all you check against the hashed record. This sounds like good old "accidentally logged sensitive data" issue.
- Wowfunhappy 8y agoThis happens to include a nice two-paragraph summary of how password hashing works, that I think a non-technical person could easily understand. It's a shame they used the term "string", but it reads fine in context. I should save for future use... ----- > If you have a Google account, Google’s core sign-in system is designed not to know your password. How, then, can we verify your password when you sign in to your Google account again? The answer lies in a bit of cryptography: when you set your password, instead of remembering the exact characters of the password, we scramble it with a “hash function”, so it becomes something like “72i32hedgqw23328”, and that’s what we store with your username. Both are then also encrypted before being saved to disk. The next time you try to sign in, we again scramble your password the same way. If it matches the stored string then you must have typed the correct password, so your sign-in can proceed. > The effectiveness of the hash function lies in its one-way nature: it is simple to scramble your password, but nearly impossible to unscramble it. So, if someone should obtain the scrambled password, they won’t be able to recover your real password. The downside of password hashing is that if you forget your password, we cannot show you what it was; there’s nothing we can do other than reset it to a temporary password (valid one time only) and then require you to pick a new one.
- kbirkeland 8y agoIMO they would have ideally used "key derivation function" instead of "hash function". It could lead those who know enough to be dangerous to think that safely storing passwords is a simple `sha512($password)` away.
- cdmckay 8y agoYou’re kidding, right? It’s supposed to be a simple explanation.
- hammock 8y agoDoesn't really explain that the "scrambling" is deterministic which is an important feature. In colloquial use "scramble" means to randomize, for example "scramble the code to your gym locker." When you scramble your locker combo, is really is random. When you scramble your password with a hash function, it may look random, but it's really not. Also just for fun, here's a creative stab at my own explanation. As every dog knows, each dog has a unique smell. If you have smelled the dog before, then you can use its smell to recognize who it is, but without that past experience, nothing in the smell itself tells you who it might be. The hash function creates the smell from your password, which is the dog. A website does't save a photo of your dog, just the smell. So if a hacker breaks in, they might smell your password but they can't see it. And when you login to the website, it makes sure your password smells right before it lets you in.
- judge2020 8y agothe email I got said it was primarily accounts with one-time use and generated passwords were affected. I'm really confused as to why this isn't in the blog. > due to legacy functionality that enabled customer Domain Admins to view passwords, some of your users’ passwords were stored in our encrypted systems in an unhashed format. This primarily impacted system generated or admin generated passwords intended for one-time use. Sure enough, the accounts listed in the email as affected were never signed in and still needed a password reset. The email: https://i.judge.sh/Twilight/iRcF0e7D.png https://i.judge.sh/Twilight/iRcF0e7D.png
- BIair 8y agoSame, the address in the notification email had never been signed in.
- topmonk 8y agoIt's a shame that password hashing isn't standardized. There should be a standard password box in html in which you can just call a function to pull out a salted hash of a password. The attributes of the box could include options for password restrictions such as a minimum of 8 chars, etc. The fact that developers still need to write custom hashing algorithms in javascript is kind of ridiculous . edit: nevermind, I just realized this wouldn't work
- j16sdiz 8y agoThis issue only affect those account resetted by admin.
- manch23 8y agoIts my understanding that the hashing should not be done client side, as that would expose the secret salt that is being used and reduce security?
- eropple 8y agoPassword salts aren't secret. They are stored alongside a password hash because they're necessary to reconstruct the hash from the plaintext. If you look at a bcrypt string, for example, the salt's encoded in it. The problem is instead that now that the client-side hash is actually the password. (edit: some people use a "pepper" at the application level and apply it to all passwords, which might be kind of what you're thinking of? Buy you don't need to do this with modern key derivation algorithms. You can if you want, it just doesn't really matter much.)
- amalcon 8y agoThat's not why (the salt does need to be secret secret), but it is true that the hashing should not be done client side. It's more because it doesn't actually accomplish a whole lot. The hash basically just becomes the password, so you'd need to hash it again on the server to get the same level of security.
- eropple 8y agoHow would this work when you have to store the hash on a server? A hypothetical client-side hasher doing this means that an attacker would just...send the hash, acquired through nefarious means. Just as they'd send the password today. Server-side, that hash would be the password and that itself would need to be hashed before storage. I don't know why you're talking about "custom hashing algorithms", in JavaScript or elsewhere; you pull a bcrypt/scrypt/Argon2 library and use it.
- tptacek 8y agoThe ordinary response to notifications like these is, to put it bluntly and accurately, a rage-mob. But here you have a corrective, a case where that doesn't make sense. Because Google operates what is considered, by an overwhelming majority of expert opinion, one of the 3 best security teams in the industry, likely exceeding in so me ways the elite of some major world governments. And they can't reliably promise, at least not in 2019, never to accidentally durably log passwords. If they can't, who else can? What are we to do with this new data point? The issue here is meaningful, and it's useful to have a reminder that accidentally retaining plaintext passwords is a hazard of building customer identity features. But I think it's at least equally useful to get the level set on what engineering at scale can reasonably promise today. correction: in the world -> in the industry
- xSHAme 8y agoFor one of the world's best security teams, "rumor" has it that their password hashing scheme is very far from best practices...
- cheeze 8y agoWhat do you mean?
- Godel_unicode 8y agoThat data point is meaningless by itself, everyone makes mistakes. Most of those people have nothing like the scale for making mistakes that Google does
- brunoTbear 8y agoWho else would you put in that league?
- JamesBarney 8y agoHe's mentioned before that Google, Microsoft, and Apple are in a league of their own with regards to security.
- 8y ago
- bredren 8y agoJust had a thread yesterday where people raged at the very idea companies and organizations still use poor hashing algos. Here we have google storing in plaintext by accident.
- Godel_unicode 8y agoIt can simultaneously be true that nobody should do this and that even the best people can make mistakes.
- asdfasgasdgasdg 8y agoI am reminded of the discourse on hypocrisy in The Diamond Age. That we fail does not mean we are insincere in our aspirations to do good.
- awinter-py 8y agoI mean hedgqw is my mother's maiden name and 23328 was my first pet, this isn't a very strong hash
- tomglynch 8y agoI've always been intrigued as to why passwords are not hashed and salted client side before sending (and then hashed again on the server side). The usual argument against it is that SSL is better - but I don't see why it shouldn't be standard to do both. It would prevent issues like this (Malicious admins or compromised server) from ever occurring. Assuming the salt is unique to the website it also solves the problem of users using the same password on many different websites - this would no longer be an issue. It would also mean as an end user we can have a look at our login POST request and confirm our password is not sent and therefore cannot be stored in plaintext. EDIT: I see the other discussion and understand the hash now becomes the password so if a hacker obtains the hash they now can login. I'm advocating for it to be hashed server side also, protecting the user from cases just like this one.
- olalonde 8y agoThis is essentially the same as using a password manager since your browser (and all your devices) would have to know about your username/salt pairs.
- tomglynch 8y agoYeah very good comparison. And password managers are by all means the best way to go to ensure privacy. So I don't see why people argue against client side hashing.
- GoblinSlayer 8y agoI think they only say it's not perfect.