5 ms·

~10 downtime per month. What is happening at GitHub?

tylfin 6y ago
The last time this happened, it was after they shipped the phone app. I wonder if they have a big feature underway or are just migrating more infrastructure to Azure? EDIT: Either way, some postmortems would be appreciated before more customers have to look for a backup solution...
- capableweb 6y ago
  If you're working on a serious project, hosting it mainly on GitHub via Git and don't already have a backup solution in place, I'm afraid you're late. But better late than never! Make sure you can always deploy when less reliable services are down, and GitHub has always been one of those. Git makes it incredibly easy as well, as long as you have your CI/CD externalized already.
  - tylfin 6y ago
    Yeah, this is very good advice. I think if revenue or product quality is tied to a VCS, having an active-active or active-passive setup is the way to go. Fortunately, I'm on an on-prem product so that investment hasn't seemed worth it yet. This doesn't mean we don't escrow our code, but rather than try to rebuild from source, I just take a short coffee break and wait for the impacted service to come back up :)
- dessant 6y ago
  I'm still awaiting the promised post-mortem [1] of a retracted blog post [2][3] which has annnounced the deprecation of the GitHub Developer Program. > There must be quite a story behind this - will you be putting up a post-mortem ? (Post mortems of business "outages" are usually more instructional) > Yes. We will. Please stay tuned. [1] https://news.ycombinator.com/item?id=21718171 https://news.ycombinator.com/item?id=21718171 [2] https://news.ycombinator.com/item?id=21718083 https://news.ycombinator.com/item?id=21718083 [3] https://web.archive.org/web/20191205225751/https://developer.github.com/changes/2019-12-05-gdp-deprecation/ https://web.archive.org/web/20191205225751/https://developer...
TruthWillHurt 6y ago
Microsoft.
rvz 6y ago
All I know is that it doesn't seem like a wise choice to be locked into GitHub features or even use their tools with these frequent downtime episodes. If a large open-source organisation was to rely on say, GitHub Actions for example, well you'll probably see more and more of "GitHub down" posts and they'll be unable to push that critical patch or run that cloud CI on GitHub, and some maybe considering solutions like this [0]. Every time this happens, you'll be completely locked in and ending up contacting / complaining to the CEO of GitHub for support via Twitter. No thanks and no deal I'm afraid. [0] https://news.ycombinator.com/item?id=23849565 https://news.ycombinator.com/item?id=23849565
- bob1029 6y ago
  Good deal for us, actually. I used to get upset about this, but the alternative really does suck more. You can either deal with the occasional non-productivity from a SAAS offering (which for GH has never lasted more than ~half a work day), or you can spin up all your own stack on-prem and generate 10 additional full-time problems in the quest to solve this one periodic issue. The trick is to never put yourself in a position where your tools absolutely must work immediately or you lose a customer. Why make a promise on delivering a piece of software until its already in hand? Also, if github goes down and I really wanted to get an issue comment in, I can just open a text editor and keep a note around in my local repo until everything is back up. I can even do some crazy things, like share my local branches with other developers over side channels until things get back to normal in the centralized system. Wasn't this kinda the entire point of talking developers into moving to the git model? Would be fun to rewind the clock and use these takes as an argument for sticking with TFS, et. al.
  - yarcob 6y ago
    > I can even do some crazy things, like share my local branches with other developers over side channels Haha, right, totally crazy, you could use git just like it was designed to be used! Who would do that!! (Joking aside, despite Githubs frequent outages, I don't recall the git service itself ever being affected)
    - 6y ago
hulitu 6y ago
> ~10 downtime per month. What is happening at GitHub? They belong to Microsoft. Reliability was never a feature of Microsoft products.
Droobfest 6y ago
Maybe they're porting over the Azure DevOps code which also has 'degraded performance' issues about every other day.
Florin_Andrei 6y ago
> What is happening at GitHub? They were acquired by Microsoft a while ago, and now the chicken are coming home to roost. It's pretty much on par with my Azure experience.
- robbyt 6y ago
  Me too. The only people who think Azure is viable, haven't actually used it.
  - seniorThrowaway 6y ago
    I recently left a project using AWS for one using Azure. I thought the AWS API's were inconsistent and janky but they look great compared to Azure. Azure is also extremely slow to perform actions in my experience and the documentation is very heavily tilted towards being sales funnels. I do like the keyvault service and the idea of resource groups. The whole tenant / subscription / roles / user mess of permissions not so much, but I expected that from Microsoft.
    - Florin_Andrei 6y ago
      The AWS API is just fragmented. Too many teams that probably didn't communicate very closely. That being said, you can sort of follow the logic - or the multiple logics. But that's really the only negative. I'm trying to find a single term to describe all of Azure, and I'm having difficulty with it. Sophomoric? Like a place where the leaders are a bunch of B-class players, who lead all sorts of C-, D- and all the way down to Z-class characters. And hopefully you'll never need support with Azure, especially urgent support, because it's atrocious.
      - hiyou102 6y ago
        What specific issues are you having? What do you mean by "B-class players, who lead all sorts of C-, D- and all the way down to Z-class characters"? I assume in this ranking you are an A+ player yourself, which is very impressive.
bredren 6y ago
This is a big problem with GitHub Actions. It took a community drumbeat and persistence from an enterprise customer to get the status message to even show a problem last month. [1] It does suck and I do think there must be some political infighting going on that the service is having so many disruptions. There’s no excuse for something this important to not only have so much unplanned downtime, but no resources to connect with the community by offering post mortems or other reasonable interactions. That said, I’m still all in on GA. It’s amazing and the coupling with repos is great. It continues to be subtly refined. So I just hope whoever is holding this product back gets out of the way. [1] https://github.community/t/random-unknown-blob-error-when-pushing-to-docker-github-package-repository/16725/36?u=banagale https://github.community/t/random-unknown-blob-error-when-pu...
dustinmoris 6y ago
The Azure DevOps team has taken over. Things will only go downhill from here.
- eurasiantiger 6y ago
  The orcs are raiding the woods around Isengard. When Sauron promised your people stability, did you think he was going to do it for free? Welcome to AzureHub.
- jbm 6y ago
  You said it. I'm consulting for a company that uses Azure DevOps and I cannot believe how much harder it is to use than Github Enterprise for getting things done. The documentation is also strictly worse, and localization is just not there at all. -edit- I assume someone who works on Azure DevOps might be looking, so a few small specific things so you don't think I'm just a hater. - It is hard to use markdown consistently, and it is particularly painful when doing any project management work on DevOps (which I assume is one of its strong points) - The lack of a Japanese menu really sucks for something that is supposedly aimed at enterprises. Having to explain both native and English language vocabulary terms is double plus ungood (again, in an enterprise product put out by a company that has usually done excellent business in Japan) - It's baffling that I can't make changes to the template when you give me a template option. I assume it is a permissions issue, but really?
danielheath 6y ago
I personally know multiple senior ex-githubbers who quit over the ICE contracts; it seems plausible to me that they’ve lost key expertise and can’t safely integrate with their older systems.
hiyou102 6y ago
In the last availability report they mentioned MySQL issues: https://github.blog/2021-02-02-github-availability-report-january-2021/ https://github.blog/2021-02-02-github-availability-report-ja... Based off these reports most of their recent issues have been them hitting scale limits for their MySQL configurations and not having sufficient monitoring.

The last time this happened, it was after they shipped the phone app. I wonder if they have a big feature underway or are just migrating more infrastructure to Azure? EDIT: Either way, some postmortems would be appreciated before more customers have to look for a backup solution...

capableweb 6y ago
If you're working on a serious project, hosting it mainly on GitHub via Git and don't already have a backup solution in place, I'm afraid you're late. But better late than never! Make sure you can always deploy when less reliable services are down, and GitHub has always been one of those. Git makes it incredibly easy as well, as long as you have your CI/CD externalized already.
- tylfin 6y ago
  Yeah, this is very good advice. I think if revenue or product quality is tied to a VCS, having an active-active or active-passive setup is the way to go. Fortunately, I'm on an on-prem product so that investment hasn't seemed worth it yet. This doesn't mean we don't escrow our code, but rather than try to rebuild from source, I just take a short coffee break and wait for the impacted service to come back up :)
dessant 6y ago
I'm still awaiting the promised post-mortem [1] of a retracted blog post [2][3] which has annnounced the deprecation of the GitHub Developer Program. > There must be quite a story behind this - will you be putting up a post-mortem ? (Post mortems of business "outages" are usually more instructional) > Yes. We will. Please stay tuned. [1] https://news.ycombinator.com/item?id=21718171 https://news.ycombinator.com/item?id=21718171 [2] https://news.ycombinator.com/item?id=21718083 https://news.ycombinator.com/item?id=21718083 [3] https://web.archive.org/web/20191205225751/https://developer.github.com/changes/2019-12-05-gdp-deprecation/ https://web.archive.org/web/20191205225751/https://developer...

TruthWillHurt 6y ago

Microsoft.

rvz 6y ago

All I know is that it doesn't seem like a wise choice to be locked into GitHub features or even use their tools with these frequent downtime episodes. If a large open-source organisation was to rely on say, GitHub Actions for example, well you'll probably see more and more of "GitHub down" posts and they'll be unable to push that critical patch or run that cloud CI on GitHub, and some maybe considering solutions like this [0]. Every time this happens, you'll be completely locked in and ending up contacting / complaining to the CEO of GitHub for support via Twitter. No thanks and no deal I'm afraid. [0] https://news.ycombinator.com/item?id=23849565 https://news.ycombinator.com/item?id=23849565

bob1029 6y ago
Good deal for us, actually. I used to get upset about this, but the alternative really does suck more. You can either deal with the occasional non-productivity from a SAAS offering (which for GH has never lasted more than ~half a work day), or you can spin up all your own stack on-prem and generate 10 additional full-time problems in the quest to solve this one periodic issue. The trick is to never put yourself in a position where your tools absolutely must work immediately or you lose a customer. Why make a promise on delivering a piece of software until its already in hand? Also, if github goes down and I really wanted to get an issue comment in, I can just open a text editor and keep a note around in my local repo until everything is back up. I can even do some crazy things, like share my local branches with other developers over side channels until things get back to normal in the centralized system. Wasn't this kinda the entire point of talking developers into moving to the git model? Would be fun to rewind the clock and use these takes as an argument for sticking with TFS, et. al.
- yarcob 6y ago
  > I can even do some crazy things, like share my local branches with other developers over side channels Haha, right, totally crazy, you could use git just like it was designed to be used! Who would do that!! (Joking aside, despite Githubs frequent outages, I don't recall the git service itself ever being affected)
  - 6y ago

hulitu 6y ago

> ~10 downtime per month. What is happening at GitHub? They belong to Microsoft. Reliability was never a feature of Microsoft products.

Droobfest 6y ago

Maybe they're porting over the Azure DevOps code which also has 'degraded performance' issues about every other day.

Florin_Andrei 6y ago

> What is happening at GitHub? They were acquired by Microsoft a while ago, and now the chicken are coming home to roost. It's pretty much on par with my Azure experience.

robbyt 6y ago
Me too. The only people who think Azure is viable, haven't actually used it.
- seniorThrowaway 6y ago
  I recently left a project using AWS for one using Azure. I thought the AWS API's were inconsistent and janky but they look great compared to Azure. Azure is also extremely slow to perform actions in my experience and the documentation is very heavily tilted towards being sales funnels. I do like the keyvault service and the idea of resource groups. The whole tenant / subscription / roles / user mess of permissions not so much, but I expected that from Microsoft.
  - Florin_Andrei 6y ago
    The AWS API is just fragmented. Too many teams that probably didn't communicate very closely. That being said, you can sort of follow the logic - or the multiple logics. But that's really the only negative. I'm trying to find a single term to describe all of Azure, and I'm having difficulty with it. Sophomoric? Like a place where the leaders are a bunch of B-class players, who lead all sorts of C-, D- and all the way down to Z-class characters. And hopefully you'll never need support with Azure, especially urgent support, because it's atrocious.
    - hiyou102 6y ago
      What specific issues are you having? What do you mean by "B-class players, who lead all sorts of C-, D- and all the way down to Z-class characters"? I assume in this ranking you are an A+ player yourself, which is very impressive.

bredren 6y ago

This is a big problem with GitHub Actions. It took a community drumbeat and persistence from an enterprise customer to get the status message to even show a problem last month. [1] It does suck and I do think there must be some political infighting going on that the service is having so many disruptions. There’s no excuse for something this important to not only have so much unplanned downtime, but no resources to connect with the community by offering post mortems or other reasonable interactions. That said, I’m still all in on GA. It’s amazing and the coupling with repos is great. It continues to be subtly refined. So I just hope whoever is holding this product back gets out of the way. [1] https://github.community/t/random-unknown-blob-error-when-pushing-to-docker-github-package-repository/16725/36?u=banagale https://github.community/t/random-unknown-blob-error-when-pu...

dustinmoris 6y ago

The Azure DevOps team has taken over. Things will only go downhill from here.

eurasiantiger 6y ago
The orcs are raiding the woods around Isengard. When Sauron promised your people stability, did you think he was going to do it for free? Welcome to AzureHub.
jbm 6y ago
You said it. I'm consulting for a company that uses Azure DevOps and I cannot believe how much harder it is to use than Github Enterprise for getting things done. The documentation is also strictly worse, and localization is just not there at all. -edit- I assume someone who works on Azure DevOps might be looking, so a few small specific things so you don't think I'm just a hater. - It is hard to use markdown consistently, and it is particularly painful when doing any project management work on DevOps (which I assume is one of its strong points) - The lack of a Japanese menu really sucks for something that is supposedly aimed at enterprises. Having to explain both native and English language vocabulary terms is double plus ungood (again, in an enterprise product put out by a company that has usually done excellent business in Japan) - It's baffling that I can't make changes to the template when you give me a template option. I assume it is a permissions issue, but really?

danielheath 6y ago

I personally know multiple senior ex-githubbers who quit over the ICE contracts; it seems plausible to me that they’ve lost key expertise and can’t safely integrate with their older systems.

hiyou102 6y ago

In the last availability report they mentioned MySQL issues: https://github.blog/2021-02-02-github-availability-report-january-2021/ https://github.blog/2021-02-02-github-availability-report-ja... Based off these reports most of their recent issues have been them hitting scale limits for their MySQL configurations and not having sufficient monitoring.