7 ms·

Incident with GitHub Packages and GitHub Pages

83 points by angrygoat 4y ago

angrygoat 4y ago
It seems wider than the status page says – all of my actions are failing, and I can't load repository pages. I can still push code though.
rpadovani 4y ago
Definitely wider than they say. I cannot access e.g. https://github.com/kubernetes-sigs/external-dns https://github.com/kubernetes-sigs/external-dns, which should be just the homepage of a project.
- nigma1337 4y ago
  had the exact same thing with https://github.com/actions-runner-controller/actions-runner-controller https://github.com/actions-runner-controller/actions-runner-... Seems to be up again now.
rvz 4y ago
Again? Last time a serious incident happened was just 11 days ago [0] with reports of actions going down again. Like I said, I expect them to go down once a month guaranteed. But this is like 6 times in one month. Now I expect something in GitHub to go down twice in one month. Not looking good at 'going all in' and 'centralizing everything to GitHub' as I predicted [0]. [0] https://news.ycombinator.com/item?id=31023941 https://news.ycombinator.com/item?id=31023941
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
perkonix 4y ago
Attempting to install packages via 'brew' (macOS) currently fail which has quite an impact I guess.
drimphgol 4y ago
Github packages have been down for almost 1.5 hours here. Sad.
intunderflow 4y ago
We're trending close to 1 incident every 72 hours with GitHub at work now, absolutely nuts. Lots of talk about moving stuff on-prem.
- trinovantes 4y ago
  Sounds like it's a bad time to be on-call at ~~GitHub~~ Microsoft
- sofixa 4y ago
  > Lots of talk about moving stuff on-prem. That can be complicated in itself, but GitLab's operational overhead is minimal ( both their omnibus and Helm deployment methods are highly automated) and IMHO it would be hard to have availability lower than Github's is at the moment ( last few months).
- ramraj07 4y ago
  We run our CI Cd on GitHub and all our code is there and I’ve never felt there outages really affect things much ever (except the one time actions was down proper for a few hours). If anything it’s more stable than our ragged old Jenkins crap, or our airflow clusters. I’ll happily trade them to GitHub if they can take care of it. Of course we did make sure we can still deploy our code if GitHub was down but that sounds like a good redundancy practice anyway so why not?
smartbit 4y ago
Status now is degraded (yellow), previously it was red.
uuyi 4y ago
GitHub has ruined my lunch or dinner time at least once a week or so for the last three months in some capacity. I am absolutely fucked off with it now. Centralising on this infra was a stupid technical decision.
- chrisseaton 4y ago
  If it goes down I would have thought you'd get a longer lunch or dinner break!
  - uuyi 4y ago
    No I have to field a thousand questions about why nothing is working from just about every part of the organisation. Then spend several hours after it has recovered assigning blame back to the engineers when they do something stupid rather than blaming it on the GitHub outage.
    - mxuribe 4y ago
      It reminds me of a time many years ago - before SaaS days - that a co-worker was similarly disrupted due to unreliability of a vendor system...and his anger towards internal senior staff for (what many obviously) considered to be a poor decision to go with said vendor. So my buddy was tempted to automate email forwards to a partiocular senior leader...so they somewhat feel the pain of all the messages being sent on a partoicular topic...plus my buddy acvited his email's out of office message to both assure folks that he was aware of the issue and to contact the senior leader most responsible for the poor vendor decision...again, to share the paion. Ultimately he didn't do that direct of a blame for fear of getting fired...but he did implement the out of office in a general fashion at least to lessen/avoid him having to answer questions about the numerous outages, etc. He didn't name any names...but some savvy folks in the org who understood the management hierarchy eventually figured things out and knew to start pionging that senior leader...and my friend didn't get fired...So there could be some mildly automated ways to lessen disruptions to your life. ;-)
      - 4y ago
th0th 4y ago
Felt sorry for the on-call folks at GitHub. What a great way to start the week :/
prepend 4y ago
So is there a general idea for why so many outages? I’ve used GitHub for 10+ years and don’t remember these outages historically. I’m sure there’s more usage now, but usage has been high for a long time. My initial guess is Microsoft monkeying with the tech stack to use inferior solutions from their own stack rather than proper architecture design and evaluation.
- zthrowaway 4y ago
  Microsoft and moving things to Azure is my guess.
  - rurban 4y ago
    nope, MySQL fuckery. https://github.blog/2022-03-23-an-update-on-recent-service-disruptions/ https://github.blog/2022-03-23-an-update-on-recent-service-d...
  - jrochkind1 4y ago
    Is there public (or other) info that they are moving things to Azure, or you're just guessing that part?
- cdubzzz 4y ago
  A slightly more generous interpretation — GitHub was pretty much stagnant (for better or worse) until Microsoft took over and started adding new features and making improvements are a pretty considerable rate. This has led to more complexity and bugs. I at least have found the recent improvements and additions pretty great — GitHub’s newish PR review UX is way ahead of BitBucket (where my employer currently hosts our codebases) for example.
  - ollien 4y ago
    My employer just moved from Bitbucket (on-prem) to Github (cloud). I have to say, as much as I hated the Bitbucket review system, there's a bunch of things we run into with Github that just are surprisingly lacking compared to Bitbucket. - In Bitbucket, you can reply to _any_ comment as a thread, rather than a quote-reply, making conversations easier to follow. Github only lets you do this if you're replying to a comment on a file. - Though this has been added very recently as a beta addition, the lack of a file tree made large reviews in our monorepo hard to follow. - If someone force-pushes, the UI doesn't tell me what commits were removed; just the new branch-head. - Our devops team has had to make it so we can re-trigger our CI (or add additional pipelines) via comment-commands, while in BitBucket we were able to add custom buttons to the PR menu. This means there's a lot of noise of comments that just say "/retrigger-ci" or some such. - This one is not Github's fault per-say, but because we used to add our users via LDAP, we used to just be able to type <first-initial + last name> to add a user to a PR, but now because everyone has bespoke usernames, it's a lot more annoying to add people because I now have to also remember their username. - In our repo, we work with lots of release branches for maintenance (committing to `develop` is not rare, but quite frequently we're adding features to old branches), and Github's PR view does not show the target branch at a glance, which can be annoying if someone has multiple PRs to cherry-pick a commit across branches and you don't know which PR you're clicking on. - Being a cloud solution, we can't add pre-receive hooks. We previously used to reject commits at push-time that didn't follow the format we use for time/issue tracking. Now, we have to enforce this in CI which means rebasing/force pushing to fix it. While pre-commit hooks can help, this depends on every developer keeping their hooks up to date, which is a challenge. That said, Github's "review" system, rather than limiting you to only individual comments, is great for email noise.
tomasreimers 4y ago
The history of GitHub statuses on HN is getting scarier and scarier: https://news.ycombinator.com/from?site=githubstatus.com https://news.ycombinator.com/from?site=githubstatus.com
- bfz 4y ago
  Graphical form: https://imgur.com/6rnFVsl https://imgur.com/6rnFVsl
  - anecd0te 4y ago
    What are the axes? Is that number of posts per date or total number of posts?
    - bfz 4y ago
      cumulative count of posts having matching URL over time

83 points by angrygoat 4y ago

It seems wider than the status page says – all of my actions are failing, and I can't load repository pages. I can still push code though.

rpadovani 4y ago

Definitely wider than they say. I cannot access e.g. https://github.com/kubernetes-sigs/external-dns https://github.com/kubernetes-sigs/external-dns, which should be just the homepage of a project.

nigma1337 4y ago
had the exact same thing with https://github.com/actions-runner-controller/actions-runner-controller https://github.com/actions-runner-controller/actions-runner-... Seems to be up again now.

rvz 4y ago

Again? Last time a serious incident happened was just 11 days ago [0] with reports of actions going down again. Like I said, I expect them to go down once a month guaranteed. But this is like 6 times in one month. Now I expect something in GitHub to go down twice in one month. Not looking good at 'going all in' and 'centralizing everything to GitHub' as I predicted [0]. [0] https://news.ycombinator.com/item?id=31023941 https://news.ycombinator.com/item?id=31023941

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

perkonix 4y ago

Attempting to install packages via 'brew' (macOS) currently fail which has quite an impact I guess.

drimphgol 4y ago

Github packages have been down for almost 1.5 hours here. Sad.

intunderflow 4y ago

We're trending close to 1 incident every 72 hours with GitHub at work now, absolutely nuts. Lots of talk about moving stuff on-prem.

trinovantes 4y ago
Sounds like it's a bad time to be on-call at ~~GitHub~~ Microsoft
sofixa 4y ago
> Lots of talk about moving stuff on-prem. That can be complicated in itself, but GitLab's operational overhead is minimal ( both their omnibus and Helm deployment methods are highly automated) and IMHO it would be hard to have availability lower than Github's is at the moment ( last few months).
ramraj07 4y ago
We run our CI Cd on GitHub and all our code is there and I’ve never felt there outages really affect things much ever (except the one time actions was down proper for a few hours). If anything it’s more stable than our ragged old Jenkins crap, or our airflow clusters. I’ll happily trade them to GitHub if they can take care of it. Of course we did make sure we can still deploy our code if GitHub was down but that sounds like a good redundancy practice anyway so why not?

smartbit 4y ago

Status now is degraded (yellow), previously it was red.

uuyi 4y ago

GitHub has ruined my lunch or dinner time at least once a week or so for the last three months in some capacity. I am absolutely fucked off with it now. Centralising on this infra was a stupid technical decision.

chrisseaton 4y ago
If it goes down I would have thought you'd get a longer lunch or dinner break!
- uuyi 4y ago
  No I have to field a thousand questions about why nothing is working from just about every part of the organisation. Then spend several hours after it has recovered assigning blame back to the engineers when they do something stupid rather than blaming it on the GitHub outage.
  - mxuribe 4y ago
    It reminds me of a time many years ago - before SaaS days - that a co-worker was similarly disrupted due to unreliability of a vendor system...and his anger towards internal senior staff for (what many obviously) considered to be a poor decision to go with said vendor. So my buddy was tempted to automate email forwards to a partiocular senior leader...so they somewhat feel the pain of all the messages being sent on a partoicular topic...plus my buddy acvited his email's out of office message to both assure folks that he was aware of the issue and to contact the senior leader most responsible for the poor vendor decision...again, to share the paion. Ultimately he didn't do that direct of a blame for fear of getting fired...but he did implement the out of office in a general fashion at least to lessen/avoid him having to answer questions about the numerous outages, etc. He didn't name any names...but some savvy folks in the org who understood the management hierarchy eventually figured things out and knew to start pionging that senior leader...and my friend didn't get fired...So there could be some mildly automated ways to lessen disruptions to your life. ;-)
    - 4y ago

th0th 4y ago

Felt sorry for the on-call folks at GitHub. What a great way to start the week :/

prepend 4y ago

So is there a general idea for why so many outages? I’ve used GitHub for 10+ years and don’t remember these outages historically. I’m sure there’s more usage now, but usage has been high for a long time. My initial guess is Microsoft monkeying with the tech stack to use inferior solutions from their own stack rather than proper architecture design and evaluation.

zthrowaway 4y ago
Microsoft and moving things to Azure is my guess.
- rurban 4y ago
  nope, MySQL fuckery. https://github.blog/2022-03-23-an-update-on-recent-service-disruptions/ https://github.blog/2022-03-23-an-update-on-recent-service-d...
- jrochkind1 4y ago
  Is there public (or other) info that they are moving things to Azure, or you're just guessing that part?
cdubzzz 4y ago
A slightly more generous interpretation — GitHub was pretty much stagnant (for better or worse) until Microsoft took over and started adding new features and making improvements are a pretty considerable rate. This has led to more complexity and bugs. I at least have found the recent improvements and additions pretty great — GitHub’s newish PR review UX is way ahead of BitBucket (where my employer currently hosts our codebases) for example.
- ollien 4y ago
  My employer just moved from Bitbucket (on-prem) to Github (cloud). I have to say, as much as I hated the Bitbucket review system, there's a bunch of things we run into with Github that just are surprisingly lacking compared to Bitbucket. - In Bitbucket, you can reply to _any_ comment as a thread, rather than a quote-reply, making conversations easier to follow. Github only lets you do this if you're replying to a comment on a file. - Though this has been added very recently as a beta addition, the lack of a file tree made large reviews in our monorepo hard to follow. - If someone force-pushes, the UI doesn't tell me what commits were removed; just the new branch-head. - Our devops team has had to make it so we can re-trigger our CI (or add additional pipelines) via comment-commands, while in BitBucket we were able to add custom buttons to the PR menu. This means there's a lot of noise of comments that just say "/retrigger-ci" or some such. - This one is not Github's fault per-say, but because we used to add our users via LDAP, we used to just be able to type <first-initial + last name> to add a user to a PR, but now because everyone has bespoke usernames, it's a lot more annoying to add people because I now have to also remember their username. - In our repo, we work with lots of release branches for maintenance (committing to `develop` is not rare, but quite frequently we're adding features to old branches), and Github's PR view does not show the target branch at a glance, which can be annoying if someone has multiple PRs to cherry-pick a commit across branches and you don't know which PR you're clicking on. - Being a cloud solution, we can't add pre-receive hooks. We previously used to reject commits at push-time that didn't follow the format we use for time/issue tracking. Now, we have to enforce this in CI which means rebasing/force pushing to fix it. While pre-commit hooks can help, this depends on every developer keeping their hooks up to date, which is a challenge. That said, Github's "review" system, rather than limiting you to only individual comments, is great for email noise.

tomasreimers 4y ago

The history of GitHub statuses on HN is getting scarier and scarier: https://news.ycombinator.com/from?site=githubstatus.com https://news.ycombinator.com/from?site=githubstatus.com

bfz 4y ago
Graphical form: https://imgur.com/6rnFVsl https://imgur.com/6rnFVsl
- anecd0te 4y ago
  What are the axes? Is that number of posts per date or total number of posts?
  - bfz 4y ago
    cumulative count of posts having matching URL over time