10 ms·
Attack campaign involving stolen OAuth tokens issued to third-party integrators
- heyoni 5y agoThis wasn’t clear to me at first but it looks like these might be heroku specific repos and not salesforce ones.
- paxys 5y agoPrivate repos of Heroku customers, to be more specific. Including "Salesforce" in the submission title only causes confusion.
- heyoni 5y agoI was confused when I wrote the title because they referred to Heroku as Salesforce and I’m not used to that.
- greysteil 5y agoThere's more detail in this blog post from GitHub: https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/ https://github.blog/2022-04-15-security-alert-stolen-oauth-u...
- r-s 5y agoIts not fully clear to me, entire private repos been confirmed stolen?
- minaguib 5y agoMy reading is if you allowed Salesforce/Heroku or Travis CI access to your GitHub account/organization, then there is a chance that that grant was leaked to a malicious third party - and they may have accessed your github account and downloaded private material. GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.
- hardwaresofton 5y ago> Known-affected OAuth applications as of April 15, 2022: > Heroku Dashboard (ID: 145909) > Heroku Dashboard (ID: 628778) > Heroku Dashboard – Preview (ID: 313468) > Heroku Dashboard – Classic (ID: 363831) > Travis CI (ID: 9216) Developers at Heroku and Travis are going to have a rough Friday
- darkhorn 5y agoOAuth properties are like jet plane cockpit buttons. For computer science students there should be one semester OAuth course. And in big companies there should be OAuth Expert. That is how much knowledge OAuth requires. If you don't agree with me try using Keycloak. It has EVERYTHING so you don't need to implement.
- jchw 5y ago> Based on subsequent analysis, we believe this API key was obtained by the attacker when they downloaded a set of private npm repositories using a stolen OAuth token from one of the two affected third-party OAuth applications described above. Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub? Seems like a bad idea, though I’m sure it’s hard to get off that practice if you’ve been on it for a long time.
- nightpool 5y agospecifically the attack Github describes seems to go: ??? ↓ Heroku/Travis OAuth token ↓ Private GitHub repo ↓ S3 token for NPM prod infra
- DoctorOW 5y agoIs it possible they used Github Actions and the attacker was able to access the secrets?
- Aeolun 5y ago> Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub? Large organisation, hundreds of developers. Kind of likely that something is visible somewhere. Still a bit surprising it’s the prod access keys for npm…
- jacobr 5y agoGitHub’s own Application Security product has secret scanning, allowing you an overview of secrets exposed across your entire organization.
- formerheroku 5y agoNeither Heroku nor GitHub are addressing the key part of this - if Heroku's Dashboard apps were compromised, and thus access was given to connected GitHub repos for download - were the secrets in the Heroku config vars for the app also visible? That is the nightmare scenario for the affected app's owners.
- prosim 5y agoHeroku/SalesForce updated their status on https://status.heroku.com/incidents/2413 https://status.heroku.com/incidents/2413: "Salesforce continues to investigate this incident in coordination with GitHub and our retained third-party breach vendor. Once we identify how the threat actor gained access to customers' OAuth tokens, we will immediately take appropriate actions." Sounds like they simply don't know yet how the actor got access and what else was exposed.
- fdr 5y agoLooks like it's addressed in an update to https://status.heroku.com/incidents/2413 https://status.heroku.com/incidents/2413 The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts While the situation can get worse, this token alone may not be enough...
- throwaway__thr 5y agoThe token isn’t nearly as concerning as what it implies: this stolen token would have been stored in heroku’s DB. However it got out likely has broader implications.
- mjg59 5y agoJust going to say that this is the kind of scenario I was discussing in https://mjg59.dreamwidth.org/59353.html https://mjg59.dreamwidth.org/59353.html . RFC 8705 actually provides a mechanism for dealing with this, so I guess I'm going to be trying to implement that for our infrastructure in the near future.
- ec109685 5y ago“A lot of services will verify the user, and then issue an oauth token that'll expire some time around the heat death of the universe” mTLS is definitely an excellent step towards solving this.
- pm90 5y agoConvincing orgs to use mtls can be difficult although perhaps incidents like this one will generate more interest.
- CaliforniaKarl 5y agoFor people looking for the HN discussion on the above-linked blog post: https://news.ycombinator.com/item?id=30916192 https://news.ycombinator.com/item?id=30916192
- jrochkind1 5y agoI still don't totally understand, can anyone read between the lines to explain what's going on here? The attackers stole what sort of OAuth tokens with what sort of access?
- tomatowurst 5y agoI could be wrong but you know when you sign up for services you have option of signing in with github or other oAuth providers? I think they used this to gain access to various private repos, npms, cloud credentials. So it seems like if your company had some super secret project running on github, heroku, AWS, they had full access to it. This is my rough understanding just skimming so could be wrong.
- jrochkind1 5y agoI guess here's what I'm wondering: did they somehow get multiple Oauth keys, one per account compromised on github -- from both Heroku and Travis, just coincidentally both heroku and travis exposed their customers keys somehow? Or did they get one Oauth key each from Heroku and Travis that exposed all of their customers data on github? Are there keys that would do this, set up for what purpose?
- Sytten 5y agoIf one thing can be gotten from that incident is forcing GitHub to create finer OAuth scopes and per repository. I am always uneasy when an application even thrust worthy like sentry asks access to ALL my private repositories even when they dont need it. I feel that in 2022 GitHub is way too critical to many organizations and they should really work on stricter rules. Even google requires special and costly audits from oauth applications to access users critical data like drive and emails.
- bswinnerton 5y agohttps://docs.github.com/en/developers/apps/getting-started-with-apps/differences-between-github-apps-and-oauth-apps#requesting-permission-levels-for-resources https://docs.github.com/en/developers/apps/getting-started-w...
- judge2020 5y agoAll the oauth prompts I’ve gotten in the past 3 years or so seem to have the granular per-repo control you’re looking for.
- media-trivial 5y agoOnly "app installations" from the GitHub marketplace can be limited to specific repositories. For OAuth apps this is not possible sadly (everything that uses GitHub to login or connect accounts, e.g. Travis CI, Terraform Registry, Heroku).
- greggman3 5y agoI wrote about this a couple of years ago https://games.greggman.com/game/github-permission-problem/ https://games.greggman.com/game/github-permission-problem/ Github has (had) some of the worst wording in the permission prompts as well as too many 3rd party apps asking for blanket permission to do anything and everything in all your repos. It might as well be "hey, give us root on your servers and our new service will do cool things for you!" And even worse is the 1000s and 1000s of repos that have signed up for those services including all kinds of popular libraries. They've given the keys to hack the libraries at will to 10s or 100s of random companies that integrate with github. It's insane.
- no-dr-onboard 5y agoAnyone else noticing an uptick in OAuth bug/authentication integrations for CICD/dev ops platforms?
- jarym 5y agoIf github themselves hadn’t been targeted then would their security team have even detected this? That is scary to me!
- clearf 5y agoHow _lucky_ that github itself was the subject of the attack in npm. Unless I'm missing something, this attack could have gone unnoticed for a long time (it would be hard for someone to connect a random breach in their infrastructure to an oauth intrusion affecting two of their service providers).
- nomilk 5y agoAm I right to say GitHub knew about this on the 12 April [1], but Heroku customers are only finding out now, 4 days (!!!!) later? Heroku boasts: > we will notify affected customers by email without undue delay. 4 days. Really. "without undue delay". [1] https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/ https://github.blog/2022-04-15-security-alert-stolen-oauth-u...
- sofixa 5y ago> Following immediate investigation, we disclosed our findings to Heroku and Travis-CI on April 13 and 14 13-14, so "only" 1-3 days later, depending on what GitHub mean and timezones involved.
- Aeolun 5y agoSo this means they also got access to all the private npm package code? Compromising all the organisations using travis or heroku is one thing. But all npm packages quite another.
- rsanheim 5y agoYikes. This is bad. On a friday night no less. Some questions that come to mind: * Would encrypted secrets for GitHub Actions be exposed to these tokens? * How to best audit for unauthorized access to exposed repos? * Were Heroku encrypted secrets exposed?
- judge2020 5y agoNothing can pull the value of secrets from GitHub actions without either running a compromised/modified runner, or the workflow itself encrypting/printing/shipping that actions secret off to somewhere. If you’re really worried, use oidc GitHub actions where possible: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services https://docs.github.com/en/actions/deployment/security-harde...
- rsanheim 5y agoGood to know - thanks!
- rndgermandude 5y ago>On a friday night no less. Not just any Friday, but Good Friday, which happens to be a legal holiday in many countries, including here in Germany. And since Easter Monday is one too, a lot of people will go on vacation on this "long" weekend (maybe even taking a couple of days off before or after to make it even longer)... Flights and hotel rooms booked, and all that.
- nomilk 5y agoCan anyone confirm or rule out whether config vars for heroku apps have been exposed? I can't see anything mentioned on the incident page nor config vars page https://status.heroku.com/incidents/2413 https://status.heroku.com/incidents/2413 https://devcenter.heroku.com/articles/config-vars https://devcenter.heroku.com/articles/config-vars
- samcheng 5y agoI don't think they know yet. It's pretty plain that source code exfiltration occurred, though. It's not clear to me how exactly to confirm that the exfiltration happened to your account or not. From the other thread: > GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.
- nomilk 5y agoProbably for a lot of us, a source code leak is not the end of the world, but leaked secrets (i.e. config vars) should be addressed immediately. Rotating credentials will take time. E.g. 20 heroku apps with 30 secrets per app could realistically take several hours (even days) to fully rotate creds.
- lumberjack24 5y agoA few days before this attack campaign started, I wrote a guide to help security and engineering teams prioritize and remediate thousands of secrets-in-code incidents. Hope it can help some of the organizations dealing with this right now! https://blog.gitguardian.com/a-practical-guide-to-prioritize-and-remediate-thousands-of-secrets-leaks-incidents https://blog.gitguardian.com/a-practical-guide-to-prioritize...
- media-trivial 5y agoI was confused because I got an email from Salesforce to my personal email. (We use Salesforce at work.) Email looked phishy (e.g. unsecured (non-https) links). Apparently they acquired Heroku in 2010.
- vemv 5y agoFor all the complexity of OAuth, an OAuth token is essentially a free pass for $stuff. Perhaps we could start validating that a token not only is valid, but also is digitally, freshly signed with a known certificate? I'd say there's a huge difference between: * I stole a token * I stole the token AND am able to keep signing said token with my victim's public certificate A token can be stolen from many places and is comparatively easy to obtain, while full intrusion into an org's infrastructure is less likely, and even if it happens, eventually will be remediated (and the related certificate will be revoked). Edit: looks like https://tools.ietf.org/id/draft-ietf-oauth-mtls-09.html https://tools.ietf.org/id/draft-ietf-oauth-mtls-09.html would fit the bill.
- taf2 5y agoI like this idea makes tons of sense… not only is the token valid but it’s from a trusted source… it is not going to prevent future attacks but it will add one more lock for an attacker to break before gaining access that is a good thing…
- remram 5y agomtls is problematic because it is not proxy friendly (needs special support in the frontmost proxy). AWS and Mastodon use HTTP request signing which seems interesting.
- bbtcyyzv 5y agoI think HTTP request signing will be a non-starter for an interoperable standard like oauth because of the difficulty of canonicalization without an authoritative library. AWS can pull that off because they have an SDK that will canonicalize requests for you, and it's tested in lockstep with the server accepting it. Historically, XML (and JSON to a lesser extent) have been security minefields for this exact reason.
- varanauskas 5y agoAFAIK according the draft HTTP request signing rfc [1] defines explicit instructions for normalising, serialising and signing HTTP headers (not body). However, you can use an HTTP Digest header for example, that is a hash of the plaintext body to authenticate the body [1] https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-message-signatures https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-mes...
- londons_explore 5y agoThis is why it's important to have 'bait' to know if secrets are stolen or attackers have access to something they shouldn't. For example, my site, serverthiefbait.com might help...
- anon3949494 5y agoWe're a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don't show this, you can only see this information in the traffic graphs (/graphs/traffic). If you're seeing cloning of repos that you haven't touched in a while, you've likely been compromised.
- Arctic_ 5y agoJust wanted to check this for my own repos but it’s only available for GitHub Pro users.
- deleted 5y ago[deleted]