6 ms·

Plans to Re-enable the GitHub Integration

ChrisArchitect 4y ago
957 hours. Pretty crazy. Can't think of another 'outage' with that kind of length on it in awhile or ever.
- firebaze 4y ago
  The remaining engineers had to deal with 900 hours of replying to recruiter spam. Please excuse them. /s
- OJFord 4y ago
  Yeah, just last week I was thinking 'this is pretty amateur hour' as I spent almost a whole morning bringing us back up.. (We are.. a tiny fraction of Heroku, in terms of anything you like - it's excusable IMO that it was an untested procedure not smooth etc., small team with MVPs to ship.) In 957h I would think you can start to think about bringing on a specialist on contract (or implement the new GH App based still-future version mentioned) if the permanent team can't figure it out / don't have capacity! It's not good for reputation, surely, I have to imagine it was considered low priority rather than something they actively tried but failed to fix for so long, but I don't think that's a good look, even if metrics show it's little-used or only by free tier or whatever.
- kodah 4y ago
  Atlassian had a similar one.
- rurban 4y ago
  Only the github connect was broken, automatic deploy. Just push to heroku also, when you push to github. I just re-enabled the Github connect, re-enabled Automatic Deploy, and removed the `git push heroku` line. 1min.
  - nicoburns 4y ago
    Sure, but that means deploying was broken for a large number of people, and that they had to spend a non-trivial amount of time setting up an alternative deployment method. For us it wasn't too bad as we already had external CI pipelines (but it still took me most of a day to sort it out). But if you were using heroku for CI then it would have been a much bigger issue.
tflinton 4y ago
I'm actually impressed that Heroku despite so much backlash refused to enable it until they were certain it was secure. Even if it took forever and no doubt probably lost them significant customers. My armchair guess is whatever method someone used to gain access more than likely took an architectural change to fix.
- joeconway 4y ago
  My anecdotal understanding is that it has been GitHub who has been apprehensive to allow Heroku to reenable and not something Heroku could be lauded for
- SkyPuncher 4y ago
  The way they handled this was atrocious. It token them a month to identify and recommend secret rotation - after having communicated they didn't feel it was necessary. That can't happen from a PaaS provider.
- __alexs 4y ago
  It's not entirely clear that they do know it's secure given they don't seem to know how the attacker gained access in the first place.
wlll 4y ago
I'd still love to get a response to the comment I made on my submission (https://news.ycombinator.com/item?id=31450100 https://news.ycombinator.com/item?id=31450100) > I'd love to hear from someone at GitHub (anonymously or not) what they've done to be satisfied with action Heroku have taken that would allow the integration to be turned back on. My confidence in Heroku to give me accurate information on this is low. As far as I can tell from Heroku's communications they: - Have no idea how the attacker gained access - Have no idea if the attacker still has access If they do know these things then I've not seen them say so.
- abathur 4y ago
  It's nitpick, but I'll note that it follows that you wouldn't know if the attacker has access if you don't know how they gained it.
  - danmur 4y ago
    They could use the access again without revealing how they got it. Double nitpick!
jrochkind1 4y ago
> Currently, when you authenticate with GitHub using OAuth, we request repo scope… As GitHub OAuth integration is designed, it provides us with greater access than we need to get the integration working. > In an effort to improve the security model of the integration, we are exploring additional enhancements in partnership with GitHub… Github permissions possibilities continually confuse me, but integrations are always asking for more github permissions than I really want to give them, more than it seems like they should need for the integration; I'm never clear in an individual case if this is because they are doing it wrong, or because github doesn't offer granular enough permissions. Some vendors with integrations in the past, when I've complained, have _claimed_ it's because github does not offer any more granular permission that includes what they need. This announcement still leaves it unclear which it was in this case. I wonder if the fallout of this thing will result in github fixing whatever it is about their permissions system that is leading to integrations asking for and getting more permissions than should be required? I have seen most blame over this kerfuffle focused on heroku, but I suspect github's too blunt integration permissions could use some ire, which might help motivate Microsoft/github to improve things.
- josegonzalez 4y ago
  Having integrated with Github before - for providing OAuth and pulling private repositories - I will say that they've never really had fine-grained permissions. The scopes are here[1] and from what I can tell, I can't ask for private repo access to a _specific_ repository for a given OAuth token. Maybe this is different for a Github App, but just quickly browsing through their docs, I don't think this is the case either. - [1] https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-apps
  - vorador 4y ago
    They actually give you fine-grained permissions, down to the single repo access level – but only if you build a Github app. OAuth app don't offer that unfortunately and I assume are considered a bit "legacy".
  - glenngillen 4y ago
multiplegeorges 4y ago
This really requires the publication of a proper post-mortem. As a customer, I'd like to know how the attacker got in and how Heroku can guarantee they don't still have access through those means.
deleted 4y ago
[deleted]
dieselgate 4y ago
Oh is this why the GitHub linking with Heroku has been broken? It was just randomly not working for me and didn’t look into it..
spondyl 4y ago
As someone who works at a company that used Heroku (I shut down our relatively dormant account) and Travis CI, this was a fun exercise. This is not exactly related to the topic but in the course of this wider fiasco, we actually uncovered a bug in Github's audit logging. The Travis CI app was removed at an org level by a Travis employee in the Middle East and to my knowledge, this wasn't publicised in advance so at first glance, it seemed kind of concerned. Anyway, that org level event didn't actually propogate up to the enterprise/umbrella level. That is, you can have an umbrella consisting of multiple Github orgs and the audit logs are supposed to roll up into the umbrella audit log. Anyway, we got confirmation a couple of days ago that it should be fixed now but worth a note if you used Github audit logs to respond to the Heroku incident or the Travis CI one
yowlingcat 4y ago
We were already moving off of Heroku anyways and this just accelerated it. Sad to see what was once a giant (really two giants, if you consider the Heroku + GH integration) fall. I wonder what the real root cause is, organizationally? Is it corporate apathy as Heroku was swallowed up by Salesforce, and GitHub by Microsoft, or something else? I wish I had a bird's eye view inside the org. For now though, I guess all I can do is move the workload to AWS.
- nicoburns 4y ago
  It's a Heroku issue. Salesforce has basically gutted the team, and put a freeze on new development (source: previous HN threads). Github is just fine.
bborud 4y ago
I think the most valuable thing to come of this debacle is the firm understanding that Heroku is on its deathbed and shouldn't be used.
- aabhay 4y ago
  Absolutely. I moved several projects from Heroku on to more modern alternatives and was surprised that many were actually just totally free. For example, I had some Vue apps that could be rendered as static sites and hosted on Cloudflare Pages at no cost.
- barkerja 4y ago
  Unless you live in the Salesforce world. Heroku has a couple of really nice sfdx products that make life much easier. Heroku Connect has been a godsend for our company.
cdubzzz 4y ago
Is there a service that provides something like Heroku’s PR deployments in a free tier (for open source development)? While Heroku was out I took a look at Fly.io and Render.com but neither one quite fits the same use case. Fly does not have automated PR builds and Render does have them but they all use the same database instance and you can’t have more than one active free database (so multiple PRs affecting the database are not possible). I actually switched to Render before realizing that limitation. It was easy to do and works well but now that Heroku has their integration working again I’m thinking I’m just going to switch back…
- Dangeranger 4y ago
  Nobody that I’m aware of offers this as a turn-key solution at the same level that Heroku does. It’s a major selling point. Setting up a well organized PR to preview environment deployment pipeline, that can be automated by CI passing is time consuming and full of yak shaving opportunities. Just clicking “enable” for a small team is a huge win.
- tyingq 4y ago
  There's this for fly.io: https://github.com/superfly/fly-pr-review-apps https://github.com/superfly/fly-pr-review-apps
  - cdubzzz 4y ago
    Thanks. I’ll check that out. I liked Fly better overall than Render but I bailed on it pretty quickly when I didn’t see a PR environment option in the settings.
- tealpod 4y ago
  https://railway.app https://railway.app is another good alternative.
  - cdubzzz 4y ago
    Thanks. I’ll give that a look. I think my needs may fit in that $5/month credit…

957 hours. Pretty crazy. Can't think of another 'outage' with that kind of length on it in awhile or ever.

firebaze 4y ago
The remaining engineers had to deal with 900 hours of replying to recruiter spam. Please excuse them. /s
OJFord 4y ago
Yeah, just last week I was thinking 'this is pretty amateur hour' as I spent almost a whole morning bringing us back up.. (We are.. a tiny fraction of Heroku, in terms of anything you like - it's excusable IMO that it was an untested procedure not smooth etc., small team with MVPs to ship.) In 957h I would think you can start to think about bringing on a specialist on contract (or implement the new GH App based still-future version mentioned) if the permanent team can't figure it out / don't have capacity! It's not good for reputation, surely, I have to imagine it was considered low priority rather than something they actively tried but failed to fix for so long, but I don't think that's a good look, even if metrics show it's little-used or only by free tier or whatever.
kodah 4y ago
Atlassian had a similar one.
rurban 4y ago
Only the github connect was broken, automatic deploy. Just push to heroku also, when you push to github. I just re-enabled the Github connect, re-enabled Automatic Deploy, and removed the `git push heroku` line. 1min.
- nicoburns 4y ago
  Sure, but that means deploying was broken for a large number of people, and that they had to spend a non-trivial amount of time setting up an alternative deployment method. For us it wasn't too bad as we already had external CI pipelines (but it still took me most of a day to sort it out). But if you were using heroku for CI then it would have been a much bigger issue.

I'm actually impressed that Heroku despite so much backlash refused to enable it until they were certain it was secure. Even if it took forever and no doubt probably lost them significant customers. My armchair guess is whatever method someone used to gain access more than likely took an architectural change to fix.

joeconway 4y ago
My anecdotal understanding is that it has been GitHub who has been apprehensive to allow Heroku to reenable and not something Heroku could be lauded for
SkyPuncher 4y ago
The way they handled this was atrocious. It token them a month to identify and recommend secret rotation - after having communicated they didn't feel it was necessary. That can't happen from a PaaS provider.
__alexs 4y ago
It's not entirely clear that they do know it's secure given they don't seem to know how the attacker gained access in the first place.

wlll 4y ago

I'd still love to get a response to the comment I made on my submission (https://news.ycombinator.com/item?id=31450100 https://news.ycombinator.com/item?id=31450100) > I'd love to hear from someone at GitHub (anonymously or not) what they've done to be satisfied with action Heroku have taken that would allow the integration to be turned back on. My confidence in Heroku to give me accurate information on this is low. As far as I can tell from Heroku's communications they: - Have no idea how the attacker gained access - Have no idea if the attacker still has access If they do know these things then I've not seen them say so.

abathur 4y ago
It's nitpick, but I'll note that it follows that you wouldn't know if the attacker has access if you don't know how they gained it.
- danmur 4y ago
  They could use the access again without revealing how they got it. Double nitpick!

jrochkind1 4y ago

> Currently, when you authenticate with GitHub using OAuth, we request repo scope… As GitHub OAuth integration is designed, it provides us with greater access than we need to get the integration working. > In an effort to improve the security model of the integration, we are exploring additional enhancements in partnership with GitHub… Github permissions possibilities continually confuse me, but integrations are always asking for more github permissions than I really want to give them, more than it seems like they should need for the integration; I'm never clear in an individual case if this is because they are doing it wrong, or because github doesn't offer granular enough permissions. Some vendors with integrations in the past, when I've complained, have _claimed_ it's because github does not offer any more granular permission that includes what they need. This announcement still leaves it unclear which it was in this case. I wonder if the fallout of this thing will result in github fixing whatever it is about their permissions system that is leading to integrations asking for and getting more permissions than should be required? I have seen most blame over this kerfuffle focused on heroku, but I suspect github's too blunt integration permissions could use some ire, which might help motivate Microsoft/github to improve things.

josegonzalez 4y ago
Having integrated with Github before - for providing OAuth and pulling private repositories - I will say that they've never really had fine-grained permissions. The scopes are here[1] and from what I can tell, I can't ask for private repo access to a _specific_ repository for a given OAuth token. Maybe this is different for a Github App, but just quickly browsing through their docs, I don't think this is the case either. - [1] https://docs.github.com/en/developers/apps/building-oauth-apps/scopes-for-oauth-apps
- vorador 4y ago
  They actually give you fine-grained permissions, down to the single repo access level – but only if you build a Github app. OAuth app don't offer that unfortunately and I assume are considered a bit "legacy".
- glenngillen 4y ago

multiplegeorges 4y ago

This really requires the publication of a proper post-mortem. As a customer, I'd like to know how the attacker got in and how Heroku can guarantee they don't still have access through those means.

deleted 4y ago

[deleted]

dieselgate 4y ago

Oh is this why the GitHub linking with Heroku has been broken? It was just randomly not working for me and didn’t look into it..

spondyl 4y ago

As someone who works at a company that used Heroku (I shut down our relatively dormant account) and Travis CI, this was a fun exercise. This is not exactly related to the topic but in the course of this wider fiasco, we actually uncovered a bug in Github's audit logging. The Travis CI app was removed at an org level by a Travis employee in the Middle East and to my knowledge, this wasn't publicised in advance so at first glance, it seemed kind of concerned. Anyway, that org level event didn't actually propogate up to the enterprise/umbrella level. That is, you can have an umbrella consisting of multiple Github orgs and the audit logs are supposed to roll up into the umbrella audit log. Anyway, we got confirmation a couple of days ago that it should be fixed now but worth a note if you used Github audit logs to respond to the Heroku incident or the Travis CI one

yowlingcat 4y ago

We were already moving off of Heroku anyways and this just accelerated it. Sad to see what was once a giant (really two giants, if you consider the Heroku + GH integration) fall. I wonder what the real root cause is, organizationally? Is it corporate apathy as Heroku was swallowed up by Salesforce, and GitHub by Microsoft, or something else? I wish I had a bird's eye view inside the org. For now though, I guess all I can do is move the workload to AWS.

nicoburns 4y ago
It's a Heroku issue. Salesforce has basically gutted the team, and put a freeze on new development (source: previous HN threads). Github is just fine.

bborud 4y ago

I think the most valuable thing to come of this debacle is the firm understanding that Heroku is on its deathbed and shouldn't be used.

aabhay 4y ago
Absolutely. I moved several projects from Heroku on to more modern alternatives and was surprised that many were actually just totally free. For example, I had some Vue apps that could be rendered as static sites and hosted on Cloudflare Pages at no cost.
barkerja 4y ago
Unless you live in the Salesforce world. Heroku has a couple of really nice sfdx products that make life much easier. Heroku Connect has been a godsend for our company.

cdubzzz 4y ago

Is there a service that provides something like Heroku’s PR deployments in a free tier (for open source development)? While Heroku was out I took a look at Fly.io and Render.com but neither one quite fits the same use case. Fly does not have automated PR builds and Render does have them but they all use the same database instance and you can’t have more than one active free database (so multiple PRs affecting the database are not possible). I actually switched to Render before realizing that limitation. It was easy to do and works well but now that Heroku has their integration working again I’m thinking I’m just going to switch back…

Dangeranger 4y ago
Nobody that I’m aware of offers this as a turn-key solution at the same level that Heroku does. It’s a major selling point. Setting up a well organized PR to preview environment deployment pipeline, that can be automated by CI passing is time consuming and full of yak shaving opportunities. Just clicking “enable” for a small team is a huge win.
tyingq 4y ago
There's this for fly.io: https://github.com/superfly/fly-pr-review-apps https://github.com/superfly/fly-pr-review-apps
- cdubzzz 4y ago
  Thanks. I’ll check that out. I liked Fly better overall than Render but I bailed on it pretty quickly when I didn’t see a PR environment option in the settings.
tealpod 4y ago
https://railway.app https://railway.app is another good alternative.
- cdubzzz 4y ago
  Thanks. I’ll give that a look. I think my needs may fit in that $5/month credit…