15 ms·
Docker Hub Hacked – 190k accounts, GitHub tokens revoked, builds disabled
Received this email a few minutes ago:
"On Thursday, April 25th, 2019, we discovered unauthorized access to a single Hub database storing a subset of non-financial user data. Upon discovery, we acted quickly to intervene and secure the site.
We want to update you on what we've learned from our ongoing investigation, including which Hub accounts are impacted, and what actions users should take.
Here is what we’ve learned:
During a brief period of unauthorized access to a Docker Hub database, sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of these users, as well as Github and Bitbucket tokens for Docker autobuilds.
Actions to Take:
- We are asking users to change their password on Docker Hub and any other accounts that shared this password.
- For users with autobuilds that may have been impacted, we have revoked GitHub tokens and access keys, and ask that you reconnect to your repositories and check security logs to see if any unexpected actions have taken place.
- You may view security actions on your GitHub or BitBucket accounts to see if any unexpected access has occurred over the past 24 hours -see https://help.github.com/en/articles/reviewing-your-security-log and https://bitbucket.org/blog/new-audit-logs-give-you-the-who-what-when-and-where
- This may affect your ongoing builds from our Automated build service. You may need to unlink and then relink your Github and Bitbucket source provider as described in https://docs.docker.com/docker-hub/builds/link-source/
We are enhancing our overall security processes and reviewing our policies. Additional monitoring tools are now in place.
Our investigation is still ongoing, and we will share more information as it becomes available.
Thank you,
Kent Lamb
Director of Docker Support
info@docker.com"
- lugg 7y agoIf you got an email you should: - Change your password on https://hub.docker.com https://hub.docker.com - Check https://github.com/settings/security https://github.com/settings/security - Reconnect oauth for Automated Builds - Roll over effected passwords and API keys stored in private repos / containers Quick take: - Password hashes - Github tokens - Bitbucket tokens - Your Automated Builds might need new tokens Checking my github logs - It looks like they've known about this for at least a full 24 hours. Most people aren't going to have this looked at until Monday which kind of sucks. Hopefully there is more of a postmortem coming. Is anyone from github able to comment on this as well? There doesn't seem to be a way for us to tell if a repo was read by these keys over that time period.
- judge2020 7y agoYep - my "deleted by associated Oauth application" event was triggered 2019-04-25 20:12:25 -0400
- typpo 7y agoYesterday at 9pm PT my private Github repo produced this notification: The following SSH key was added to the foo/bar repository by myorg-dockerhub-user: Docker Cloud Build 39:31:51:be:d6:00:c4:ef:c7:74:c2:16:66:33:93:06 If you believe this key was added in error, you can remove the key and disable access... I wonder if this is related? Dockerhub integration and its keys were still present on Github. In any case, I've revoked everything until the impact becomes clearer.
- reidrac 7y agoAt the moment I can't change the password. It fails with "Failed to save password" error, no more information. EDIT: it finally worked, 4th attempt, and very slowly. Looks like something isn't working 100% as it should EDIT 2: aaaand I can't login now with the new password. A password reset did work, but it looks like their password database is under some stress at the moment.
- franee 7y agoSame can't change password
- sodosopa 7y agoI could as of 10 minutes ago
- hn_throwaway_99 7y agoMy guess is their auth system is/was under a ton of load. Specifically to make the password database more secure, the generation of password hashes is very computationally intensive by design (e.g. that's the whole point of something like bcrypt vs. sha1) Password systems really shouldn't be designed to handle a 10x or 100x load without some slowdown. If they could handle that, it means their password DB probably isn't as hardened as it should be.
- pavanagrawal123 7y agoI can't find an announcement of this anywhere besides HN? Will Docker be publishing info via official mediums?
- lugg 7y agoI assume they will. I only just got the email and it looks like only a small subset of accounts are affected. Or at least that's what that PR spin is supposed to make you think.
- pavanagrawal123 7y agoI see. I originally thought this was the announcement, as that is what the post indicated.
- lugg 7y agoYea sorry about that I was more focused on figuring out what needed to be done today and who needed waking up so I just dumped the email. I hope this doesn't hurt docker too badly. I really like the hub / auto build service.
- ohyeshedid 7y agoYou aren't the one hurting Docker, they've done that themselves. You put the word out there, so thank you for thinking of everyone else out there.
- RyJones 7y agoI got email at work.
- Leo_Verto 7y agoDocker Hub being hacked was basically just a question of time. With how much of the internet blindly pulls images from it, the potential gain from hijacking just one high-profile one would be monumental.
- pavanagrawal123 7y agoThis raises the question of whether any high profile images were targetted by the infiltrators?
- anaphor 7y agoThat doesn't mean there aren't plenty of things they could have done to make this more secure. The fact that you can just `docker login` with the same credentials that allow access to your entire registry is pretty poor security design IMO.
- tylerl 7y agoI was originally going to argue with it being "just a matter of time" -- there is such a thing as good security practices. It's certainly not "just a matter of time" before Microsoft or Google see such compromises. I'm pretty confident that these companies have their sh*t in order. But no, not Docker. You're totally right; with as important as their registry is to well funded attackers, and as startup-y and "agile" as they are, and as godawful as the security practices are that underlie their tools and standards... they hadn't a chance. They still don't. There is no reason to expect them to get better.
- lowpro 7y agoFun fact, there was a universal XSS vulnerability on google (including search, support, accounts, cloud, etc) found just last week [0]. I'd say it's always just a matter of time. That doesn't mean they don't have everything in order, but securing everything as much as possible is half the battle. The other half is a solid response when things do happen, which we will now see in how Docker handles this situation. [0] https://twitter.com/WHHackersBR/status/1118393568656334850 https://twitter.com/WHHackersBR/status/1118393568656334850
- aerovistae 7y agoWould have made this a bit clearer to note in the post that this is an email you received, and that you are not Kent Lamb using Hacker News as a medium to distribute Docker announcements, which is what this looks like.
- lugg 7y agoGood point, it does look wrong. Updated.
- unixbhaskar 7y agoheck ..pathetic
- thosakwe 7y agoWell, this is pretty disappointing. Docker doesn’t let you install it without an account, so I registered and used it for maybe a day in all. And poof, there goes my account data. I’m just hoping that I was using a password manager by then. Any word as to the cause of this? Was something important stored in plaintext, etc.?
- rhizome 7y agoThey say "accessed database," so I'm thinking SQLi.
- Operyl 7y agoSQLi that managed to access only a single shard though? Hm.
- lugg 7y agoIt sounds more like a developer environment got exposed with prod data on it. This going by the way it's worded "single hub database with a subset of non financial data"
- ohyeshedid 7y agoYeah, I got that vibe too.
- koolba 7y ago> Well, this is pretty disappointing. Docker doesn’t let you install it without an account, so I registered and used it for maybe a day in all. And poof, there goes my account data. Eh? Doesn’t let you use what without an account? Anyone can pull images anonymously. An account is only for publishing.
- dlor 7y agoInstalling Docker for Mac/Windows has required users to login for awhile now.
- starpilot 7y ago0wned.
- Operyl 7y agohttps://status.docker.com https://status.docker.com still not a mention. Wonder how long until it is.
- marcus_holmes 7y agothat's not good...
- ahmedalsudani 7y agoThat's the wrong place to track a hack. The status page is concerned with uptime, not security.
- Operyl 7y agoI disagree, destroying a ton of keys breaks stuff.
- dvdgsng 7y agoThey added it.
- VirtualAirwaves 7y agoI'm glad I don't use Docker.
- craftoman 7y agoImagine the impact if NPM got hacked instead of Docker Hub. People would go crazy, run the streets like monkeys and yelling why NPM is untrustworthy must be boycotted. Last time one user got hacked and they blamed NPM for letting it happened. Everyone went crazy...
- ohyeshedid 7y agoThat's because npm has a history of screwing the pooch.
- tannhaeuser 7y agoWhat did they do specifically? Not saying npm is beyond criticism, but we shouldn't just accept vague and unsubstantiated claims here.
- ohyeshedid 7y agoThere's a few previous issues, just use the site search here for npm and have a look.
- leowoo91 7y agoNPM already freaks out many people.
- quickthrower2 7y agoI secretly love NPM. If your open source project’s first code section is “npm i ...” I’m happy.
- manigandham 7y agoBoth situations are bad, and people are upset over Docker Hub. It just happens to be Friday night so it's not getting as much attention. NPM is bad because the Javascript ecosystem is fast-moving with loose builds that have thousands of dependencies that are all bundled and run insider consumer's browsers.
- skilled 7y agoSo, would it have been possible that the perpetrators knew about the keys and had built a way to scan them all beforehand? Or is this more likely to be an attempt at farming passwords?
- deleted 7y ago[deleted]
- choward 7y agoI wonder if that will encourage them to finally resolve this issue: https://github.com/docker/docker.github.io/issues/6910 https://github.com/docker/docker.github.io/issues/6910
- dbnoch 7y agoOr fix this 4 year old issue where you cant use 2FA for accounts https://github.com/docker/hub-feedback/issues/358 https://github.com/docker/hub-feedback/issues/358 (Side note: this obviously wouldn't have prevented the current attack)
- WindowsFon4life 7y agoThis!
- deleted 7y ago[deleted]
- morpheuskafka 7y agoWhat permissions did the leaked tokens have? If they had write access, then leaked personal data is the least of anyone's worries. The real concern is how close the hackers came to infiltrating the image source for virtually every modern microservices system. If you could put a malicious image in say alpine:latest for even a minute, there's no telling how many compromised images would have been built using the base in that time.
- lugg 7y agoPretty sure (don't quote me) those are read only and repo specific but that could contain all sorts of juicy info depending how lax you are with security of configs in private repos. Even then just read access to code often allows enough info for leveraging/escalating privilege.
- buzer 7y agoWhen you connect your Github account to Docker Hub, that will give DH full access to all repos (https://i.imgur.com/4jJWrez.png https://i.imgur.com/4jJWrez.png). I'm not even sure if Github's permission model supports adding only read access to private repositories. I'm not 100% sure if Docker hub uses deploy keys for repos it has access to thru the integration, but at least previously there was an option to manually add one to repository if it couldn't access it otherwise.
- rkrzr 7y ago> I'm not even sure if Github's permission model supports adding only read access to private repositories. Their newer GH apps permission model allows fine-grained access to only specific repos (and also only read access e.g.). However their older Oauth flow only allows full access to everything. And 99% of GH integrations still seem to use the older authentication method. This is also something that many CI providers suffer from. There are only few that already support GH apps.
- martinlofgren 7y agoI tried to give an user read-only access to a private repository on GitHub a few weeks ago, and from what I could tell it isn't possible.
- Gonzih 7y agoIt saddens me that docker hub is still lacking FIDO or any 2FA support.
- gigatexal 7y agoDoes this lessen the relevance that docker has these days?
- viraptor 7y agoDid Docker become any less useful for you due to this, or provides less value? Unlikely.
- gigatexal 7y agoI’m thinking twice about using docker hub. And the main usecase is k8s. So docker is just an implementation detail its relevancy is waning imo
- friedrichg 7y agoDocker hub is a centralized service. What we are seeing is the result of having a huge centralized service: if it gets compromised, then many dependencies are compromised. Some organizations took the risk of running docker taking images directly from docker hub. They were relaying the security of the images to them. Some organizations are going to panic now and host their own registry. Which they need to protect as well. But in general it will create a better decentralized ecosystem. I think this is good for the docker community in general.
- gigatexal 7y agoWe run our own registry that just mirrors images that we want to use and keeps them up to date. It’s not a silver bullet but it works.
- turtlebait 7y agoI'd worry about mirroring the images because of cases like this, you'd want some sort of triage process before it gets into your environment.
- blcknight 7y agoI did not get any email but my github is showing dozens of failed login attempts over the last 3 days.
- diNgUrAndI 7y agoWhat are dockerhub's alternatives? No 2FA. That is bad.
- viraptor 7y agoYour own repo, AWS ECR, whatever GCP's version is called, and many others.
- buzer 7y agoThere are actually very few alternatives for the autobuild part. The only alternative that I'm aware of is Quay, others require you to roll out your own build & push process.
- viraptor 7y agoIt's not that hard to roll your own (I'm doing that). It's not trivial, but if you need autobuild rather than just tags, it's not a huge time investment either. Some systems have all the necessary stuff exposed as plugins too (for example buildkite)
- oszione 7y agoGCP's Cloud Build is also a simple option.
- techntoke 7y agoAutobuilding is really just a free GitLab pipeline.
- dawnerd 7y agoI use drone.io self hosted to build all my images. They then get pushed to a self-hosted hub.
- nurettin 7y agoon-site nexus is good.
- geezerjay 7y ago
- viraptor 7y agoThat's a nice summary. One thing I'm curious about is: > Data includes usernames and hashed passwords How are they hashed? And specifically, can we expect them to be already cracked?
- ghusbands 7y agoYes, in particular we need to know algorithm, work factor and salting details to know whether or not the passwords may be compromised.
- trulyrandom 7y agoJust assume that it's compromised and generate a new one. There is no point in wasting time trying to estimate how long it might take someone to crack it.
- viraptor 7y agoIt matters at lower extreme. If it was something trivial and people shared the password with another account, then they may be already compromised. If it was hard and salted per-user, they still have to change it, but the chance of compromise on other services is significantly lower. It may also explain some suspicious behaviour / source of compromise in the past (we know when the issue was uncovered, not when the first dump was taken)
- Perceptes 7y agoNot a huge surprise. Here's another security issue with Docker Hub they've let sit for 4 years with no action: https://github.com/docker/hub-feedback/issues/590 https://github.com/docker/hub-feedback/issues/590 (which is apparently a dupe of https://github.com/docker/hub-feedback/issues/260 https://github.com/docker/hub-feedback/issues/260).
- villgax 7y agohash+salt please
- jaequery 7y agoIf the passwords are hashed, just what are the likelihood of your passwords being decrypted? I’d also imagine it is a one way hash since that’s typically the norm so I don’t even know how it can get decrypted.
- TheDong 7y agoHashes are not decrypted, they are bruteforced. > I imagine it is a one way hash All hashes are one way. If it's lossless and can be reverted, it's a compression algorithm or isomorphism or encryption or cipher or any of a number of other things, but not a hash. > I don’t even know how it can get decrypted. It is not decrypted, but brute forced. For example, even if you can't algorithmically figure out what the input to md5sum is that gives you '1f3870be274f6c49b3e31a0c6728957f', you could apply md5sum to every word in the dictionary in a matter of seconds and find out that 'apple', when md5summed, has that output. You would then have one possible password for that hash (though technically there are infinitely many inputs that have that output). The only way we can know how computationally difficult it is to brute force the password hashes is if we know the following: 1) the hash algorithm used (and other inputs like cost factor) and 2) the entropy of the salt used. Those two together lets us calculate the amount of computation needed to try one brute force "guess". Individual password's difficulty to be brute forced can then be calculated from their entropy (e.g. 'apple' has less entroy than '2SEZb'), to determine the average number of inputs needed to be tried, multipled by the cost of each attempt. Given that difficulty, you can then estimate how long an attacker will take to find your password by estimating how much computational power they have at their disposal. In general, if you randomly generate 10+ character passwords and docker used best practices, the answer is that any attacker will not get your password in under a thousand years, and if you use a password which has been leaked before or is a dictionary word (or simple variation), it can be found on the order of minutes to days.
- techntoke 7y agoHopefully they are salted with a unique ID because of the are using a md5sum only then you're screwed with rainbow tables.
- hestefisk 7y agoI think this is a very ugly incident for Docker.
- bamboozled 7y agoYou're probably busy, but you might want to update the splash page on Docker https://hub.docker.com https://hub.docker.com to notify users of the incident ?
- sambe 7y agoWhy can’t these emails just come out and say it: “your account was affected”. It’s always implicit. Also, why rely on users to change their passwords? Is there a security log I can check?
- gruturo 7y agoShould they change your password for you? How do they communicate it securely then? Over unencrypted email, whose password may or may not be the same of your just-compromised docker account?
- supakeen 7y agoThey could invalidate the passwords making you use a 'forgot password' link to enter a new password instead of keeping the old compromised ones :)
- zoobab 7y agoError 500 when I try to login: "Sorry, it's not you. It's us, but we are working on it!"
- rad_gruchalski 7y agoTheir hub website is pretty bad. I tried changing the password and the website came back with an error: Failed to save password. Interesting, so I tried again. This time it said: Current password is incorrect. I thought, maybe I need to log out and try if the new password works. I clicked on Log Out link, the website has refreshed and I was still logged in.
- bvm 7y agoyeh that happened to me when i rotated the password on our master docker hub (or cloud or whatever it is today) account prior to all of this.
- strictfp 7y agoSame here.
- tnolet 7y agosame issue.
- pacifika 7y agoPassword reset works
- bproven 7y agoYep - same here. :( It changes it, but reports error...
- rnotaro 7y agoOfficial Article from Docker (Same Text as the email): https://success.docker.com/article/docker-hub-user-notification https://success.docker.com/article/docker-hub-user-notificat...
- fock 7y agosuccess.docker.com!
- aneutron 7y agoFrom the same company that tried to force people to login before downloading Docker CE.
- saurabhnanda 7y agoJust wondering, genuinely out of curiosity - how does one get to this 5% number? If the attacker had access to the DB s/he had access to 100% user data right? Or did the get access to a partition of the user data? How is this even possible? Some very old backup that had only 5% of earliest users? Some log file which had plain-text creds of approx 5% users? Or did they discover the attack as it was happening and kicked-out the attacker in the middle of a data download (only 5% complete)?
- croh 7y agosame feelings here. On what basis they are predicting 5% ?
- torvald 7y agoTheir data can be sharded whereas only a part of their databases got compromised. Or it could be a cache layer that got compromised. Or a partial user dump intended for something else that somehow ended up in the wrong hands. I guess there could be a lot of reasonable explanations.
- chungleong 7y agoA differential backup file would be my guess.
- ghusbands 7y agoKnowing the hash algorithm, work factor and salting details would be helpful in knowing whether or not passwords may be compromised. This should be standard information given in a breach, rather than just whether passwords were hashed. Though, as they say that passwords need changing, we can safely assume that their salting, hashing and work factor were insufficient and not following best practice. Just like the lack of 2FA.
- efficax 7y agoEh, if hashes leaked I would still suggest changing passwords no matter the crypto practices involved. If you change the password, the hash is useless. If you don't, it's sill an attack vector, even if a technically impractical one (for now)
- Kudos 7y agoWhy am I being asked to change my password? Why haven't they just invalidated it for me already? I'm astounded I was still able to login with my existing password.
- gtirloni 7y agoIt looks like they have sent emails to everyone, not just the 5% affected.
- logophobia 7y agoI haven't received an e-mail, I've got multiple docker-hub accounts.
- tedmiston 7y agoI haven't received one yet either.
- krferriter 7y agoNeither have I. I manage over 30 images on dockerhub. Maybe this means they are certain my data was not in the data that was leaked but I'm not sure how they'd be certain of that. They did just post the notice in a banner at the top of https://hub.docker.com https://hub.docker.com https://success.docker.com/article/docker-hub-user-notification https://success.docker.com/article/docker-hub-user-notificat...
- GordonS 7y agoHmm, my GitHub account is showing failed logins starting from 2 days ago, with none for the remaining period that GitHub shows - no email from Docker yet, but I wonder if this is related?
- wtdata 7y agoWhat does this means for users? I was using watchtower to auto update the images in my system. One of them was autoupdated after the failure. Can this be used to upload containers with security exploits in order to gain access to machines (i.e. does it give write access to the containers)?
- lumjjb 7y agoAnother reason to have Encrypted Container Images :) https://github.com/opencontainers/image-spec/issues/747 https://github.com/opencontainers/image-spec/issues/747
- aphextron 7y agoIs this affecting CircleCI? As far as I know their images pull from Docker Hub.
- mgalgs 7y agoWell this is fun, I'm now unable to logout. I have a feeling there's more to this incident than Docker is currently disclosing...
- billconan 7y agoI’m curious, how can a database be accessed without authorization? If authorization is enabled? Also how unauthorized access can be discovered? Say I use Mongodb and enabled authorization. Will I be fine then? How to discover unauthorized access?
- sirclueless 7y agoAuthorization has a common English definition too. If, for example, an employee's credentials were compromised, anyone who wasn't that employee who accessed the database would be considered "without authorization". And checking the access logs for any use of that employee's credentials would give you some idea of what data was accessed. Enabling authorization on your mongodb is good, but it absolutely won't stop all forms of unauthorized access. They may gain access to your server itself, or gain some credentials to your MongoDB database some other way (for example, if someone carelessly ships them as part of your software, or includes them in a github commit, or something like that). In the worst case, if someone notifies you of a configuration problem or some software bug that allows anonymous access to your database or the ability to remove logs, you may have to assume the entire database was compromised since the existence of that configuration issue or software bug.
- jite 7y agoI've seen some failed attempts to log on to my GitHub account from 'Quito, Provincia de Pichincha, Ecuador' (which is quite far from where I live, as I live in Sweden...). Not sure this is related at all, but they started appear after this leak was announced... Luckily I use both 2fa and random password for github, would suck to loose that account ;)
- nudpiedo 7y agoWhy is this publicly posted here and not just in the platform and directly contacting people affected? So big became the HN influence and marker? It could have been just linked ️
- maxhedrome 7y agohttps://github.com/docker/docker.github.io/issues/6910#issuecomment-487297149 https://github.com/docker/docker.github.io/issues/6910#issue...
- begueradj 7y agoThat is a great news. Happy security.
- ankushnarula 7y agoDocker has revoked GitHub and BitBucket access tokens tokens at least as of 27 Apr 2019 18:41:36 UTC http://archive.fo/bKGKq http://archive.fo/bKGKq
- madhuakula 7y agoThis step by step checklist might help you "what should I do" to review your accounts. https://blog.madhuakula.com/some-tips-to-review-docker-hub-hack-of-190k-accounts-addcd602aade https://blog.madhuakula.com/some-tips-to-review-docker-hub-h...
- pyman 7y agoLet me get this right, Docker now forces users to register in order to download their client and they don't secure our data? Insane!
- arjamizo 7y agoyou guys should partner with github to disable those token which have leaked