22 ms·

Heroku Security Notification

• deleted 5y ago
  [deleted]
• ozgune 5y ago
  Related HN article: https://news.ycombinator.com/item?id=31046791 https://news.ycombinator.com/item?id=31046791
• deleted 5y ago
  [deleted]
• roddylindsay 5y ago
  Ah yes, the Friday night security incident drop.
  • jayzalowitz 5y ago
    Trash day
  • deleted 5y ago
    [deleted]
  • quickthrower2 5y ago
    The Easter egg.
• TheSpiciestDev 5y ago
  I do remember hooking up Heroku to Github for auto-deployments and thinking to myself something along the lines of, "why does Heroku need ALL of this access?" It'd be great if Github could allow read/write permission grants on a per-repo basis. Maybe they do already!.. in which case I'd much rather have and setup that granular detail than have a token that goes across all my public/private repos... Edit: I do see in my Github's integration page that the Heroku connection was used within the past week... but it doesn't show how exactly it was used. Until Github can provide specific details, is it safe to assume that all repos, public and private, could have been cloned?
  • pineconewarrior 5y ago
    > It'd be great if Github could allow read/write permission grants on a per-repo basis. Maybe they do already! They totally do. Shopify's Github integration works this way, and it is fantastic!
    • bswinnerton 5y ago
      These are known as GitHub apps and were designed to address this problem. https://docs.github.com/en/developers/apps/getting-started-with-apps/differences-between-github-apps-and-oauth-apps#requesting-permission-levels-for-resources https://docs.github.com/en/developers/apps/getting-started-w...
    • TheSpiciestDev 5y ago
      Thanks and I'm not surprised, it's a pretty intuitive feature! So really the only thing Heroku gets from these all-inclusive tokens is something to drive their type-a-head input on their integration page, right? Totally not worth it, I'd rather use Github's prompt.
• frays 5y ago
  > Additionally, we recommend disconnecting Heroku from your GitHub repositories. I have respect for the Heroku/Salesforce Security team for willing to ask users to perform this action. Many companies would be too worried about losing customers or having users not reconnect it afterwards. My thoughts are with the team working on responding to this incident on Easter Friday.
  • ryantgtg 5y ago
    From what I can tell from the update from a couple minutes ago, this recommendation wasn’t deemed to be enough. Now they’re revoking all tokens.
  • czbond 5y ago
    For anyone wondering where this is....it is in your project -> "Deploy" -> "Deployment method" -> "Connect to Github".
    • forgingahead 5y ago
      For some reason for the single app I had used this for, there was no "Disconnect" button - so I just did the nuke option of deleting the connection right from the main settings page: https://dashboard.heroku.com/account/applications#third-party-applications https://dashboard.heroku.com/account/applications#third-part... For checking your Github audit logs, you can go directly here (replace ORG_NAME with your own): https://github.com/organizations/<ORG_NAME>/settings/audit-log https://github.com/organizations/<ORG_NAME>/settings/audit-l... Or from: Organization > Settings > Archive > Logs > Audit Log I hope we get some more clarity on the extent of this incident soon. We'll rotate our keys anyway but I really hope the attackers did not have access to the ENV vars that are commonly set on Heroku directly.
      • pselbert 5y ago
        For many accounts the audit log isn’t going to show some crucial information—git activity. Clone events are only available through the REST API for Enterprise customers.
• samcheng 5y ago
  Does anyone know what to look for in the github audit logs, exactly?
  • nomilk 5y ago
    Try here: https://github.com/settings/security-log https://github.com/settings/security-log
    • captn3m0 5y ago
      Don’t think these logs mention the oauth application being used to access at all.
    • samcheng 5y ago
      I used here: https://github.com/organizations/<ORG_NAME>/settings/audit-log https://github.com/organizations/<ORG_NAME>/settings/audit-l... ... but the real question is what would malicious activity look like, exactly?
      • yeskia 5y ago
        I see a heap of "downloaded a zip of repository" but I suspect that's Heroku CI or other CI tool running.
        • domh 5y ago
          Yeah I see lots of the same. They seem to correspond with Heroku deploys? Anyone know if that happens when a valid heroku deploy occurs? Do they download a zip? I've reached out to Heroku support to ask.
          • andrelaszlo 5y ago
            Please let us know if you get any info from them! :)
          • heartbreak 5y ago
• nu11ptr 5y ago
  If I want to revoke all Heroku's access to Github, is it the "Heroku Dashboard" I'm looking for under "Authrorized Oauth apps"? I revoked that one, but not sure if that is everything.
• duxup 5y ago
  Interesting I had an OLD project associated with a Gmail email address that would send emails in some cases. Yesterday I got a notification that someone tried logging into that Gmail account. The password was hard coded in the code…
  • r-s 5y ago
    I too have seen something very similar to this. I am extremely nervous all my private repos got compromised. Watching this very closely.
    • duxup 5y ago
      I’m assuming they have been. This email was a one off that outside this code exists nowhere else.
• mepiethree 5y ago
  Ughh here I was at 1:30AM after a hard week, checking the news one last time before going to bed, and see this. I hate this dang industry and regret ever becoming a tech lead. I know the Heroku engineers have it worse and all, but just venting.
  • ketzo 5y ago
    That’s rough. Stay strong, friend.
  • aaronbrethorst 5y ago
    Probably just Heroku engineer, singular.
    • glenngillen 5y ago
      What makes you think that?
  • notimetorelax 5y ago
    News on HN are usually not relevant to my work and I actively avoid reading anything work related on my off time. I’d find some other activity to relax and wind down if reading HN has an impact on you. Burnout is no joke - you need yourself rested for yourself and your team. Take care of yourself.
  • rosndo 5y ago
    But why do you care? Not your company, not your problem.
• samcheng 5y ago
  This is a chance to reiterate best practices: Credentials and other secrets, like API keys, should never be hard-coded in the source code repo. Use some sort of secrets management or configuration for that kind of stuff.
  • ryanSrich 5y ago
    In this case, can we confirm that Heroku environment variables were not accessed? Because if they were, even not storing secrets in the source code wouldn’t have prevented a breach. If Heroku could confirm environment variables were safe I’d have a much better sleep tonight.
    • ameliaquining 5y ago
      The announcement claims that, as far as they know, the attacker can't access your Heroku account, only your GitHub repo (because they only got the GitHub-issued token). So this would imply that your environment variables are safe. That being said, it sounds like they don't yet know how the attacker managed to do what they did, which means they can't rule out the possibility that more data was stolen than is currently known. But it sounds like probably not.
    • drusepth 5y ago
      >The compromised tokens could provide the threat actor access to customer GitHub repos, but not customer Heroku accounts. With the access to customer OAuth tokens, the threat actor may have read and write access to customer GitHub repositories connected to Heroku. Based on the above information, my assumption is that the attacker gained access to code repos hosted on GitHub, but not access to live dynos or the Heroku dashboard (that happily shows ENV vars). We'll see if this information updates as the investigation progresses, though. However, given 1) write access to a github repo, and 2) auto-deployments from github to production dynos (if enabled), an attacker could exfiltrate env vars (among many other nasty things). However, this would trigger events in your app's activity log (new commits + deploys) and should be quick to verify that it didn't happen.
  • maccard 5y ago
• kadoban 5y ago
  The attacker got _write_ access to all of these repositories as well? That's extremely worrying. I hope github or someone will be able to track down if any code changes were made.
  • quickthrower2 5y ago
    A forced push could introduce vulnerability into code in a years old commit that is hard to detect. I guess it will look dodgy when your next push is rejected.
    • couchand 5y ago
      It would not at all be "hard to detect", the SHAs of it and all subsequent commits would change, it would appear in the GitHub audit log, etc etc
      • nijave 5y ago
        Honestly should be pretty easy to just compare remote to someone's recent local copy and just spot check any new commits. Definitely a time sink, though
        • kadoban 5y ago
          Yeah it's not the hardest thing to check for, the real problem is that you have to check and you have to know what you're seeing. Someone who misses the memo on this one, isn't so good at git, and/or is on a smaller project could really be bad news.
• decidertm 5y ago
  Heroku users using GitHub should start rotating any secrets stored in their repos (people still do), if the OAuth has been compromised it means your repositories could have been cloned. Non-enterprise GitHub users have no detailed audit history to see if their repos were cloned/zipped.
• andrelaszlo 5y ago
  I just started a Discord server: https://discord.gg/K9ecetqn https://discord.gg/K9ecetqn Please join if you're impacted by this incident, or interested in these topics in general. I'd like to discuss mitigations around this and similar incidents with other HN:ers: - Knowledge sharing: resources, how-tos, tips - Discussing prevention, mitigation, etc - Moral support and venting If there's already such a forum (I assume there is), please send me an invite :)
• buf 5y ago
  This is as good a time as any to ask about Render. I've been eyeing it recently and I'm thinking about launching my next project with it. Does anyone have any takeaways from using Render vs Heroku?
  • aaronbrethorst 5y ago
    I’ve been using Render since January, after switching from Heroku for a project of mine with about 10,000 direct users at present. The project is a Rails app that imports hundreds of thousands of rows of data from XLSX files and spits out PDFs on demand from those imported spreadsheets. I was worried that Heroku would end up costing me a small fortune as demand scaled. Plus the platform seemed to have stagnated. I contemplated switching to AWS, but didn’t want to deal with the extra hassle of it. By chance, I saw someone mention Render on here, checked it out, and couldn’t be happier. It’s a bit harder to get up and running with Render than Heroku, but orders of magnitude easier than with AWS. And once you’re operational, it’s a cinch. And way, way, way cheaper.
    • akpa1 5y ago
      Render is really not a very good name from a SEO point of view. Took me three or four searches to find it.
      • aaronbrethorst 5y ago
        Google: "render hosting"
  • leetrout 5y ago
    Id also recommend a look at Porter.run
  • fourstar 5y ago
    It’s alright but things are still flaky given they are smaller and newer. It hung on spinning up a postgres instance. Builds are slower as well.
• fxtentacle 5y ago
  To me, this reads like someone got read access to all the source code on Heroku, so if you have any credentials hardcoded in there, now's the time to change them, too. I wonder if the hackers were kids who got bored around Easter holiday - meaning Heroku's security is shit - or if Heroku deliberately waited to announce this during Easter holiday to minimize the attention it gets - meaning they are as deceitful as all proper megacorps. I haven't been able to trust their status page to accurately reflect what works and what doesn't for a long time. The only reliable signal is when their status page goes offline ;)
  • drusepth 5y ago
    Based on their incident report, it sounds more like someone gained access to the OAuth tokens used to authenticate with linked GitHub repos, which would actually be both read _and_ write access to all source code of repos linked to Heroku apps. RE kids or misdirection: as with all things, it's probably somewhere in the middle -- a somewhat-sophisticated attacker and a slow, evolving investigation unfortunately coinciding with a holiday weekend. They say the report was received 3 days ago (on April 13) from GitHub after they noticed suspicious activity on April 9.
• deleted 5y ago
  [deleted]
• fxtentacle 5y ago
  FYI looks like the stolen Heroku OAuth tokens are already being used: https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/ https://github.blog/2022-04-15-security-alert-stolen-oauth-u...
  • jsmeaton 5y ago
    Both heroku and travisci were affected? I wonder if and what the common vector would be for two major providers to lose their OAuth tokens.
    • andrelaszlo 5y ago
      - Heroku might be using Travis? - They were compromise independently but the hackers decided to start leveraging them at the same time to get the most out of them? - The obvious common denominator: Github? It doesn't make a lot of sense though.
• samwillis 5y ago
  Disappointed that the GitHub security log doesn’t show access for personal accounts. Would be rather nice if they temporarily made that available for a short period of time so we can see if any of our repositories have been cloned/downloaded.
  • macintux 5y ago
    I've been disappointed with GitHub's auditing capabilities. I noticed that some members of my team were committing changes to our enterprise repositories with poorly-configured clients, so it was impossible to tell who made the commit or performed the push. I understand that git itself doesn't have any protections against this, but GitHub knows who pushed it; why isn't that metadata available?
    • gtirloni 5y ago
      GitHub Enterprise has push logs. https://docs.github.com/en/enterprise-server@3.2/admin/monitoring-activity-in-your-enterprise/exploring-user-activity/viewing-push-logs https://docs.github.com/en/enterprise-server@3.2/admin/monit...
      • macintux 5y ago
        Do you know whether that's available via GitHub proper? I'd like the user identity for commits to be accessible through the API, but last I checked it wasn't. (Updated for clarification: user identity is available via the API, but only the identity embedded in git itself, not the GitHub authenticated user.)
• debarshri 5y ago
  I might be naive but how does a threat actor get access to encrypted bunch of keys.
  • worker_thread 5y ago
    an employe gone rogue?
• sandstrom 5y ago
  This is a much bigger problem than Heroku. There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access). - Productboard - Bugsnag - Sentry - Skylight - Percy - CodeTree - Databox There are heaps of others, these are just some on top the of my mind. A ticking supply chain attack waiting to happen, since these companies make themselves into alluring hacking targets. Most of them need access only to issues (a few need read access to code or recent commits, almost none need write). Solution: - Let customers give granular access (only issues, only read to source code, etc) when the integration is setup. This is possible with Github's APIs. - Try to use push instead of pull where possible, i.e. provide a CLI tool to use with Github actions or use Github's webhooks.
  • the_mitsuhiko 5y ago
    > There are countless SaaS applications asking for full-repo access to Github (all the source code, with write access). Sentry does not request write access to source code. It requests read/write access to issues and read access to source code. You can also see this on the documentation for the GitHub enterprise integration which lists the exact permissions required: https://docs.sentry.io/product/integrations/source-code-mgmt/github/ https://docs.sentry.io/product/integrations/source-code-mgmt...
    • sandstrom 5y ago
      Thanks for clarifying. It would still be better if I didn't have to give read access to source code, but could still use the issues integration. But I agree it's not as a bad as write access to source code.
      • michaelbuckbee 5y ago
        Most error monitoring services want this as they map errors to commits.
        • dharmab 5y ago
• uallo 5y ago
  Instead of connecting to a Github repo, you can use https://github.com/heroku/heroku-builds https://github.com/heroku/heroku-builds. It allows to create a build locally and then deploy it to Heroku. From what I've read so far, this approach has not been impacted and it should still be possible to do deployments like that.
  • Exuma 5y ago
    What about simply using the heroku CLI which is the default? I've used that for every heroku project, I didn't even think to connect to github. I suppose the main benefit is that it's automated.
• jrochkind1 5y ago
  I have always wondered if putting confidential info in even private git repo was a good idea, although it seems to be a common practice? I feel like that question has been answered, for me anyway.
  • nijave 5y ago
    Definitely not a good idea. Source code gets sprinkled around a lot of locations with varying security - it's stored locally (so any developer machine access could compromise secrets) - it's stored on the build/CI machines (that might run untrusted code in the form of dependencies--possibly from unrelated repos) - it can end up in build artifacts
• anon3949494 5y ago
  We're a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don't show this, you can only see this information in the traffic graphs (/graphs/traffic). If you're seeing cloning of repos that you haven't touched in a while, you've likely been compromised.
  • from_endor 5y ago
    Might be a hint if you really have not touched them at all in years. In other cases this might also stem from CI tools like GH actions.
  • Ozzie_osman 5y ago
    Are you on an enterprise plan or a personal one? I know personal plans have limited logs, but I'd hope the enterprise ones would show clones (in reality, they both probably should). Kind of defeats the purpose of an audit log if it doesn't.
    • anon3949494 5y ago
      We're on the Team plan
  • mariusz331 5y ago
    For anyone not on a pro plan: I believe you can upgrade and still see the past two weeks of data. I cloned a few of my private repos last night to see how that affects my security logs and no logs appeared. I later upgraded to pro and visited /<username>/<repo>/graphs/traffic and can see the clone counts from before I upgraded. I also can see visitor counts from about a week ago. These clones still don't appear in the security logs though.
    • ezekg 5y ago
      I tried upgrading my org to a Team to check traffic, but the upgrade seemingly did nothing. I do have sponsorships, so maybe it’s waiting until my next billing cycle? Can’t figure it out.
• surjithctly 5y ago
  Heroku Support is sh*t. Thanks. I can't access my account now.
• Ozzie_osman 5y ago
  I'm not a security expert, but if you're reading this and wondering what to do, a good start could be to just assume your repo was accessed, and so to run a tool like gitleaks against your repo. If it detects anything sensitive, I'd see about revoking/deleting those secrets right away. In general, it's good practice not to check anything sensitive into source code for precisely this reason (if your code is compromised you don't want your secrets to be as well). So it'd also be good practice to add something like gitleaks into your CI/CD pipeline for the future.
  • ryandrake 5y ago
    It's such a cryptic call to action. When people reading an incident disclosure end up wondering what to do, you're not finished writing. I haven't logged into or even used Heroku in close to a decade. Am I affected? I just toyed around with a hobby project on Heroku those years ago and didn't release anything. Am I affected? If that project didn't connect Heroku with GitHub, am I affected? It's been so long, I don't even know how to use Heroku anymore or even access my repos there. Yet, I got this message in my E-mail so that would imply that maybe I'm affected and need to do... something? Who knows, the E-mail doesn't say what to do either. Sounds like the best course of action for someone like me would be to just to log in, find any button that says "delete everything" and use it. So confusing.
    • heartbreak 5y ago
      > Sounds like the best course of action for someone like me would be to just to log in, find any button that says "delete everything" and use it. So confusing. Yet that isn’t even close to the best course of action. If your GitHub account was compromised and you had secrets in private repos, deleting everything in your Heroku account isn’t going to do anything to help.
  • lumberjack24 4y ago
    A few days before this attack campaign started, I wrote a guide to help security and engineering teams prioritize and remediate thousands of secrets-in-code incidents. Hope it can help some of the organizations dealing with this right now! https://blog.gitguardian.com/a-practical-guide-to-prioritize-and-remediate-thousands-of-secrets-leaks-incidents/ https://blog.gitguardian.com/a-practical-guide-to-prioritize...
• dangoor 5y ago
  See also GitHub's post about this: https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/ https://github.blog/2022-04-15-security-alert-stolen-oauth-u... Travis-CI was also compromised here and that may actually affect more people than the Heroku side of this.
  • dangoor 5y ago
    Discussion about GitHub's post is here: https://news.ycombinator.com/item?id=31046791 https://news.ycombinator.com/item?id=31046791
• mgomez 5y ago
  The only thing in my security log over at GitHub that I'm not familiar with is a handful of "repo.change_merge_setting...Blocked a merge setting on the ${my_repo} repository" entries. Googling that provides little information. Can anyone explain what that means? Should I be worried?
  • macNchz 5y ago
    That log entry happens when you change the settings on a repo for what kind of merges are allowed for PRs (merge commit, squash, rebase). It’s three checkboxes under “Pull Requests” in the settings for a repo. I can’t speak to any reason a threat actor would have wanted to change those settings though.
    • mgomez 5y ago
      I see, thanks. I don't ever remember touching these settings before.