12 ms·
NPM security update: Attack campaign using stolen OAuth tokens
- kseifried 4y agoI've been thinking about this a lot, how do we do Security at the SaaS level. Some thoughts: https://github.com/cloudsecurityalliance/CSA-IT-Operations/tree/main/tactics#readme https://github.com/cloudsecurityalliance/CSA-IT-Operations/t... https://github.com/cloudsecurityalliance/CSA-IT-Operations/blob/main/strategy/README.md https://github.com/cloudsecurityalliance/CSA-IT-Operations/b...
- fomine3 4y agoReminder: npm is now owned by GitHub
- classified 4y agoOn one hand it's good to have an organization with lots of engineering resources behind these, but I have to wonder whether that much stuff in one place (Microsoft in this case) won't bite us down the line.
- arzel 4y agoWhat were* the passwords hashed with?
- yardstick 4y agoFrom https://github.blog/2022-05-26-npm-security-update-oauth-tokens/ https://github.blog/2022-05-26-npm-security-update-oauth-tok... “The password hashes in this archived data were generated using PBKDF2 or salted SHA1 algorithms previously used by the npm registry. These weak hashing algorithms have not been used to store npm user passwords since the npm registry began using bcrypt in 2017. ”
- alexghr 4y agoThis is probably a good time to remind people to check their authorized OAuth applications on Github[1] and make sure that any unused apps have their access revoked. [1]: https://github.com/settings/applications https://github.com/settings/applications
- tommoor 4y ago> Using their initial foothold of OAuth user tokens for GitHub.com, the actor was able to exfiltrate a set of private npm repositories, some of which included secrets such as AWS access keys. > Using one of these AWS access keys, the actor was able to gain access to npm’s AWS infrastructure. How many individual best practices were not followed to result in this nightmare? Sigh.
- dlsa 4y agoAll of them to at least some extent and several we will soon have names for because they are still emerging.
- richardfey 4y agoYou say "attack campaign", I say bad habits catching up with secrets scanners *and* someone noticing it. Black hats might have been exploiting this already in the past.
- jabiko 4y agohttps://github.blog/2022-05-26-npm-security-update-oauth-tokens/ https://github.blog/2022-05-26-npm-security-update-oauth-tok... > Using their initial foothold of OAuth user tokens for GitHub.com, the actor was able to exfiltrate a set of private npm repositories, some of which included secrets such as AWS access keys. So NPM was storing AWS secrets in their (private) git repos. IMHO that was an accident waiting to happen.
- goodpoint 4y agoThis is why we should use packages from well-known Linux distributions instead of npm/pip/cargo etc. I know the available libraries are a fraction of the ecosystem, but very often it's a good enough fraction if you are willing to be flexible in your choices.
- maxloh 4y agoYou cannot download React from apt/yum AFAIK.
- goodpoint 4y agohttps://packages.debian.org/source/bullseye/i386/node-react https://packages.debian.org/source/bullseye/i386/node-react This one? EDIT: it's a bit old, probably due to the usual mess of dependency management in JS. Still perfectly usable tho. Having seen that, I would stay away from react.
- Gigachad 4y agoThat package is wildly out of date. I’d estimate the vast majority of react apps are running on a much newer version. And no one wants to have their react upgrade tied to a Debian upgrade. Forcing these things to happen at the same time would be a nightmare.
- noodlesUK 4y agoSerious question: what’s the difference? Do Linux distros actually audit packages very much? Supply chain attacks in Linux distros are pretty scary, as you can basically expect them to run install scripts as root.
- goodpoint 4y ago> Do Linux distros actually audit packages very much? It varies widely depending on the distro. Debian is the the most strict on review process but also on screening the volunteers doing the work. Yet, most popular distros has been very effective at blocking a large number of supply chain attacks over the last 20 years.
- philsnow 4y ago> Unrelated to the OAuth token attack, we also recently internally discovered the storage of plaintext credentials in GitHub’s internal logging system for npm services This isn't the first time GitHub has found logging of plaintext credentials [0]; it's not a good look, for a company with the resources that GitHub has, to have to disclose it again almost exactly 4 years later. [0] https://www.zdnet.com/article/github-says-bug-exposed-account-passwords/ https://www.zdnet.com/article/github-says-bug-exposed-accoun...
- tinus_hn 4y agoThis happens all the time in a lot of companies, you just don’t hear about it.
- rmbyrro 4y agoNever seen plain text credentials being logged in systems I've looked at. PII, sure, even login usernames or email, also sure, but not credentials (plain text passwords, tokens, etc).
- bastardoperator 4y agoLOL, literally entire products designed to prevent and report on these issues. Just because you haven't seen it doesn't mean it isn't happening daily unfortunately.
- deleted 4y ago[deleted]
- mschuster91 4y agoSay someone implemented oauth or whatever fancy auth mechanism in a dockerized software and had the problem that the environment variable containing the secrets were not passed through the layers to the application server, and so the developer (as they didn't have SSH access to the machine or the container) was forced to build in a lot of logging to discover where in the path the secrets got lost. Then the developer forgot to remove all logging incantations.
- cpach 4y agoMore comments over here: https://news.ycombinator.com/item?id=31526649 https://news.ycombinator.com/item?id=31526649
- alexghr 4y agoI'll quote my comment from the duplicate thread because I think it's important people audit their authorized applications on Github > This is probably a good time to remind people to check their authorized OAuth applications on Github[1] and make sure that any unused apps have their access revoked. > > [1]: https://github.com/settings/applications
- deleted 4y ago[deleted]
- pvg 4y agohttps://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=author%3Adang%20copy-paste%20comments&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- deleted 4y ago[deleted]
- dodgerdan 4y agoWill this be met with a shrug from the JS community? Or is this the come to Jesus moment for the JS supply chain?
- rmbyrro 4y agoWas this incident facilitated by something inherent in the JS ecosystem? I have the impression it wasn't. The JS ecosystem sucks, but anyway, not particularly their fault in this case.
- VoidWhisperer 4y agoFor once, I don't think this highlights an issue exclusively specific to JS.. this could've happened to any package system that Github owned when the attacker was able to pivot after accessing the private repos.
- tuxie_ 4y agoWhat would you expect the JS community do after this? What would you do?
- stolenmerch 4y agoAs a member of the JS community: shrug. I revoked my OAuth apps on Github, changed my passwords. The tarballs are unaffected. Not worried.
- throwaway290 4y agoI hope this is self-deprecating humor, but for anyone that takes it at face value: The implication of a successful attack on NPM, with huge unvetted dependency graphs currently in fashion, would be that any of the thousands dependencies of a modern small JavaScript app could suddenly include malicious code that runs your dev machine or your production systems. (That's why the key part of the announcement is "GitHub is currently confident that the actor did not modify any published packages in the registry or publish any new versions to existing packages".)
- ratww 4y agoWhat do you expect the community to do? Stop using thousands of packages? Start vetting packages, as if security was important? There are already dozens of us saying it is possible to not have too many dependencies, and vet packages before installing. But every time we open our mouths we are treated as if we just escaped some sort of insane asylum. At a place I worked in the past we used to have a 40-line microservice using plain-node without any dependencies. That was by design. One junior dev took it upon himself, in their spare time, to convert the whole thing to use some js MVC framework, complete with a full-blown build process, transpilers, and all the nine yards. There was a big discussion in the PR and a lot of juniors complained that we should migrate because they "didn't learn plain node.js in college". We can't have nice things anymore.
- glenngillen 4y agoDid I miss the previous GitHub announcements about this amidst all the noise about how badly Heroku handled their part of this problem? Or have GitHub been sitting on the specific facts they had a database, email, and hash passwords leaked for over a month now?
- tln 4y agoTFA starts with "On April 15, we published a blog[0] detailing an attack campaign ..." So, there was a blog post on April 15, at least. [0] https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/ https://github.blog/2022-04-15-security-alert-stolen-oauth-u...
- glenngillen 4y agoYeah, that’s the only one I recalled seeing. And nothing at the time of a database leak with emails and hashed passwords, nor a backup that included access to private repos.
- ralph84 4y agoI take it you’ve never been involved in a breach investigation. Figuring out what an attacker had access to and whether they exploited that access isn’t trivial, especially for a heavily used service like npm. To say they were “sitting on” information while probably tens if not hundreds of engineers assisted in making sure the investigation was complete and accurate is uncharitable.
- glenngillen 4y agoHow presumptuous. I’ve worked with all of the teams involved in this specific incident over the past 12 years. I’ve helped lead incident response for much larger breaches. I know what’s involved. And the moment you’re aware customers have had their passwords exposed (even if hashed) you come clean on it and empower them to take mitigating action so that they’re not unduly exposed longer than they need to be. You probably even consider forcing password resets for everyone. As Heroku did, in its own hamfisted and still too slow way.
- jeremymcanally 4y ago> Using their initial foothold of OAuth user tokens for GitHub.com, the actor was able to exfiltrate a set of private npm repositories, some of which included secrets such as AWS access keys. Keep those keys out of source control, folks. There are a lot of options for secrets management these days, and making it harder for attackers to totally own you if they only manage to crack one piece of your infrastructure is key to limiting damage from this sort of attack.
- dgb23 4y agoThis is also a thing that will not get past some interviewers.
- urda 4y agoA rule of thumb is: if the key is pushed into a git remote you should consider it compromised and roll a new key.
- celticninja 4y agoWhat if they key is in a git-crypted file? I get what you are saying about an open file, but surely best practice is to use encrypted files to store secrets that are needed e.g for deployment
- urda 4y agoAn encrypted file stored along with the git repo (without the decryption key) has a different attack surface. My original comment was more targeted towards users storing their keys in plaintext.
- joshstrange 4y agoI agree fully with you but I don't think secret management is as easy/cheap as some people pretend. On AWS, for example, each secret you store is $0.40 + $0.05 per 10,000 API calls and that can add up if you only have 1 api key/password/etc per "Secret" (for an individual at least, I hate bleeding off $10+/mo to store tiny bits of strings). Then, once you have the secret stored, you need setup roles/policies to be able to retrieve them. I have this setup pretty well in my code now but getting there wasn't simple or easy from my perspective and keeping the list of secrets your IAM user can access up to date can be a pain as well. I'm working in a lambda environment so my options might be more limited but I'm interested to see how other people are solving this issue (maybe specifically for small/side projects). As it stands my lambdas all get a role applied to them that gives them access to the secrets but something not AWS-specific would need a "bootstrap secret" to be injected before the code could call out to the third-party to get the other secrets. For Lambdas I suppose I could inject that "bootstrap secret" in via environmental variables but now I've got a new issue to deal with. Injecting at build time via something like GitHub Actions Secrets is an option I guess. All that to say, while I agree secrets should never be in source, in practice it's not super easy (I'd love to be proven wrong, maybe I'm not doing it right).
- junon 4y agoTop 10 maintainer here, got a few emails this morning about it. Meh. Shit happens. If we all Pikachu face every time an exploit happens we are lying to ourselves. We'll never reach perfect security. It's a pipe dream. What matters more is the disclosure and response. I'm not a huge advocate of npm personally, but I respect their response to this thus far. From what I gather (the email was a bit long-winded) nothing vastly detrimental occurred, they automatically invalidated passwords and going to publish again next time will require a couple minutes tops of extra work. I'll take it. Let's all stop acting like products need to be perfectly and eternally secure. That's not how threat modelling works, any security professional knows that's impossible, and it's unfair to expect that from anyone, including big corporations. Npm has done a lot of relevant and good work toward their security efforts over the years, in some cases going a bit far even in my own opinion. The comments I've seen so far have been a bit unfair.
- capableweb 4y agoIf this was a clever hack, I'd agree with you, shit happens. But for gods sake, having secrets hardcoded in VCS?? You seem to understand threat modelling. What's the threat towards one of the biggest and most used package registry? It's not like npm Inc just started running the registry. They have been doing this for years. To let such a beginner mistake risk the supply chain of basically the entire JS ecosystem is not only sloppy, it's completely unprofessional. What can we do in the short-term? I'm not sure, but I hope smarter people than me comes up with some solutions ASAP before a compromise like this starts actually impact developers using npm.
- 0daystock 4y agoIt is long past time for channel-binding of bearer credentials. It's unconscionable we as engineering professionals allow the total sum of security controls of a system to constantly be reduced to a string humans pass around.
- judge2020 4y agoOIDC-based authorization between trusted parties seems to be picking up steam[0], with the main issue being that setting it up isn't as easy as "paste shared secret". 0: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services https://docs.github.com/en/actions/deployment/security-harde...
- bvrmn 4y agoIt's an interesting spin, we learned to store hashed passwords. But there is virtually zero advise to store hashed tokens. I can't remember any corresponding point in OWASP guides. It's funny how old username:password pair is more secure than modern tech.
- swang 4y agoWhat were they using to hash passwords? sha256? I actually am curious since they didn't mention it in their email.
- msbarnett 4y ago> The password hashes in this archived data were generated using PBKDF2 or salted SHA1 algorithms previously used by the npm registry. These weak hashing algorithms have not been used to store npm user passwords since the npm registry began using bcrypt in 2017. Which is so frustrating. When you upgrade your hashing algorithm, always always _always_ immediately remediate the weak hash mess by hashing your weak hashes with your new stronger hash, and turn the login check into bcrypt(sha1(user-entered-password)) == stored-bcrypted-sha1 you can then upgrade them to a straight bcrypt if the check succeeds, but keeping the weak hashes on disk indefinitely until the user logs in (if ever!) is such a risk.
- wepple 4y agoI’ve never heard this suggested before. Such a simple elegant solution I’m kinda embarrassed as a security engineer.
- xbar 4y ago"On April 12, GitHub Security began an investigation that uncovered evidence that an attacker abused stolen OAuth user tokens issued to two third-party OAuth integrators..." Too vague to be useful. Stolen from whom? Simultaneously from two different third parties? The whole thing began April 12? Based on what signal and from whom? Here's what this vagueness tells me: GitHub OAuth token management for integrators is badly operated and poorly monitored by the entity issuing tokens. My conclusion is that Microsoft (GitHub), SalesForce (Heroku), and Idera (TA Associate, Travis CI) have not learned the SolarWinds lessons of Supply Chain Security yet. Perhaps this will be the incident that helps. Some of the posts in this thread are a little too c'est la vie for my tastes.
- msbarnett 4y ago> > "On April 12, GitHub Security began an investigation that uncovered evidence that an attacker abused stolen OAuth user tokens issued to two third-party OAuth integrators..." > Too vague to be useful. Stolen from whom? Heroku and Travis CI. That information is literally in the end of the sentence you cut off: "On April 15, we published a blog detailing an attack campaign utilizing stolen OAuth user tokens issued to two third-party GitHub.com integrators, Heroku and Travis CI."
- celticninja 4y agoIt could also be read as a total of 4 third party integrators. The comma between integrators and Heroku could indicate a part of a list. It's not as clearly written as it could be that is for sure.
- msbarnett 4y agoIt would really only make sense for someone to write it that way if Heroku and Travis CI were not "third-party OAuth integrators" (which they are). At some point we're just playing "Yes but if we assume the author did not speak English natively and made this particular kind of odd error, then you could read it as meaning...". Occam's Razor applies – the simplest possible reading is almost certainly the correct one.
- kodah 4y ago> Salesforce-owned Heroku noted that some of its private repos were accessed on April 9 before slamming the brakes on GitHub integration. That integration was restored earlier this week, according to the company's status page. This is important for this forum, imo. Heroku was cast a lot of shade during this.
- deleted 4y ago[deleted]
- born-jre 4y agoon separate note, why don't registry like npm take a raw repo and build in some kind of reproducible environment (like nixos/or even standardized docker ) then user uploading pre built package is beyond me. i think fdroid does sth like that. i think it will help with a lot with different supply chain attack. (not this one though). native extensions (c++) will be pain though, but with the advent of wasm maybe native extensions are not that great idea now huh ¯\_(ツ)_/¯
- Sohcahtoa82 4y agoHow did this happen? You'd think a company like GitHub would know better than to allow passwords to land in log files.
- klodolph 4y agoWhat does it mean for a company to "know better" than to do something? It means some combination of the following: people don't make the mistake in the first place, the mistake is caught in code review, the mistake is caught in audits, the mistake is caught by automated tooling...
- blip54321 4y agoThis isn't uncommon. For example, when MIT/Harvard took over edX from the original researcher who built it, they found they didn't know how to build software. The new team introduced this same issue: passwords in log files. It was fixed much later. You can look over the git logs. I'm not sure this is so much of a "how good this happen," but a "thank you for being transparent." Most organizations cover this sort of thing up, like MIT/Harvard (and now 2U). Good of github to announce this openly!
- lol1lol 4y agoIsn't a decent auth library boilerplate at this point?
- deleted 4y ago[deleted]
- willcipriano 4y agoI've never had this problem but I thought of a partial solution. Say you have you unit tests and they are using the same auth and logging mechanisms as prod. Create a user with a password like "ThisStringIsAPassword1234" and run the unit tests having them output logs to the disk. Then see if the logs contain that value. Anybody ever do something like that? How effective it would be probably depends on unit test coverage. You could also probably just do the same thing in prod with a dummy user.
- danpalmer 4y agoI agree it should be this simple, but I'd bet they have tests like this and unfortunately it's never quite this simple in a production system. I've always wanted to apply strong type systems to this problem – wrapping sensitive data in types that do not have the ability to be printed to logs would theoretically allow you to know after type-checking that passwords can't be output to logs. However again I think this is wishful thinking as a password needs to be sent somewhere at some point, and that creates places where issues can occur.
- rvnx 4y agoWell it's their choice to ask the user to send a plain-text password. There are alternatives to avoid this, on the same model of SSH key authentication where the secret stays on the client-side. Nothing prevents them from using a password to derive a private key using PBKDF2 in client-side and answer to a specific challenge.
- willcipriano 4y agoThat's kind of how I've always solved it at the smaller companies I've worked at. I'm on Hacker News all day and know all about these kinds of footguns so I write all the auth code myself and then provide a couple of helper functions/methods for the team to use that are unable to do anything too silly. Creating an object that throws an exception when you try to get it as a string or print it though is something I'll have to try next time.
- er4hn 4y agoThat is one thing that RFC8959 is intended to solve. If you see "secret-token:" in any logs after running tests you flag that as a problem and fail the test.
- bearjaws 4y agoSomething I cannot emphasize enough for startups is to not use your own auth no matter how convenient you might think it is. Github was built before the era of many "Authentication as a service" providers, but this is another example of why you don't roll your own.
- sergiomattei 4y agoI don’t understand. Just about everyone rolls their own auth. This is awful advice. I’d never trust my authentication needs to external providers who can hold my user information hostage.
- bearjaws 4y ago... AWS offers managed authentication service (Cognito), for many websites your data is literally already in AWS. They have zero reason to hold your usernames and emails hostage... The same can be said for GCP (Identity Platform) and Azure.
- joemi 4y agoRolling your own has some benefits, though. Such as not being as big a target as a provider that services many sites, and being more in control of the auth data, and not being reliant on a service that's beyond your control to not suddenly stop working or shut down.
- bearjaws 4y agoFor sure some auth providers will be compromised at some point in the future, but there are literally thousands (tens of thousands?) of examples of self managed identity management being compromised. How many Auth0 or AWS Cognito service compromises have there been? To your point about availability, AWS, GCP and Azure all have managed authentication services, that are fundamental to their platforms. I highly doubt they are going anywhere. I have yet to have an Cognito outage in three years.
- latchkey 4y agoI consider my passwords public knowledge at this point. My NPM is secured with 2FA, which should be the minimum default requirement for any service.
- eximius 4y agoAnd this is one reason why client side hashing is a good idea (in addition to other procedures). Even if you screw up, the impact is so much less severe.
- 01acheru 4y agoIf you do client side hashing then the hash becomes the password, it only helps with password reuse on other services. From your service perspective the security issue doesn't change much.
- SAI_Peregrinus 4y agosaPAKEs are one possible solution, but none are currently standardized.
- eximius 4y agoI can already impersonate my users. But yes, client side hashing is just one part of a security story, but it sure does make log leaks less damaging.
- chaps 4y agoI saw this at a.. scary-large.. company I was at in 2014ish. If a client changed their password, it would get logged into a log file, plaintext. I asked a coworker why they did this, and he said it was to tell the client their password. Hm. They did actually patch it before I got there though.. but they didn't get rid of the years-old log files with the passwords. Found them while trying to find the root password (unsuccessfully) for a host that we couldn't reboot. The ones I tested still worked. I wouldn't be surprised if something similar happened here. Old log files in backups and such.
- srathi 4y agoShameless plug: I created a Golang utility to scrub passwords from a deeply nested struct, before logging, at Nutanix some time back [0][1]. We also run an automated test to try out all operations with a known password, and then ensuring that it is not present in any of the log files. [0] https://github.com/ssrathi/go-scrub https://github.com/ssrathi/go-scrub [1] https://www.nutanix.dev/2022/04/22/golang-the-art-of-reflection/ https://www.nutanix.dev/2022/04/22/golang-the-art-of-reflect...
- WallyFunk 4y agoDoes anyone know if Socket[0] can help here? [0] https://socket.dev/ https://socket.dev/