10 ms·

GitHub Outage Currently Ongoing – Issues, PRs, Notifications

140 points by dazbradbury 7y ago

krzkaczor 7y ago
Events like this make me realize how much I rely on my CI/CD pipeline for working on new features, deploying etc. I am often too lazy to run E2E tests on my machine if I know that on CI it will take only a few minutes to be done.
- prepend 7y ago
  I feel guilty because I’ve never installed GitLab’s CI process locally and have only run it for the past few years through git commits. I keep expecting someone to complain, but so far, so good. I expect similar poor behavior testing GitHub’s actions because it’s so convenient to only run through the service. It is important though that it’s possible to run without CI/cd. If it wasn’t available I would complain because of the risk of lock-in.
lousken 7y ago
Seems like github will soon become part of office 356 /s
xchaotic 7y ago
Justyna yesterday I was complaining about GitHub monoculture in the context of Atlassian sunsetting Mercurial. Outages like this are normal, but the problem is lack of good competitors.
- arghwhat 7y ago
  There are quite a decent list of competitors, all of which have similar or sometimes better offerings. There are also plenty of fully free and open source solutions, should you want to do things yourself. The Github monopoly is driven by its users.
  - mschuetz 7y ago
    I tried gitlab but I hated the web interface. It was lacking in so many departments. Any other alternatives to github to try?
    - unixhero 7y ago
      What was lacking? The GUI on Gitlab is phenomenal imo.
      - hkitdog 7y ago
        On gitlab,repository and code searching function should be improved loll
        penagwin 7y ago
        Same could be said for github. I regularly literally copy and paste a variable name from a file on Github itself - paste it into the search, and it finds 0 results. HOW COULD YOU FIND NOTHING?! Not even the file I copied it from? A simple grep search would've had no issue and would've been faster!
        arghwhat 7y ago
        Yeah, I always end up having to clone the repo to grep.
    - danskyr711 7y ago
      Bitbucket is pretty good
      - stevekemp 7y ago
        Bitbucket suffers from all the usual problems of an Atlassian-product: A badly designed user-interface, and frequent stability/downtime problems.
- consolenaut 7y ago
  I think you're right about a general move towards a Git monoculture (as opposed to a GitHub one), but there are dozens of great Git repo hosting/collaboration companies - GitLab, Bitbucket & SourceForge to name a few of the bigger ones
- nickpsecurity 7y ago
  The problem is bad design and/or integration of components. VMS clusters over leased lines went years without outages. Record was 17 years at a rail yard. They also did rolling upgrades across both different OS versions and CPU architectures. The methods are public if any competitors want to match or exceed them: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04622808 https://support.hpe.com/hpsc/doc/public/display?docId=emr_na... These modern cloud services using Linux ecosystem are doing rolling outages instead. I'm amazed people keep making them dependencies. Maybe their local systems using similar technology went down more often. Maybe this is an improvement. I'd still put mine on an OpenBSD or OpenVMS cluster if it was business-critical with lots of money on the line. I want it staying up, up, up. :)
  - mugsie 7y ago
    Modern apps are designed to deal with part of them being down - it is the core upgrade strategy. VMS clusters were (and modern equivalents are) awesome, but they have a different speed of innovation / development. It is a trade off - perfect uptime, but that new feature could take a year, or 99.9% uptime, but it can be written and deployed tomorrow.
    - nickpsecurity 7y ago
      Their development velocity might have been due to their management or internal culture. You can get most of those results faster and on a budget. The best evidence was the startup behind FoundationDB continually improving their product in highly-competitive space while doing QA like this: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc Generally, on top of code reviews, one can get high return with minimal labor and hardware with a combo of Design-by-Contract, contract/spec/property-based testing, low-F.P. static analysis, and fuzzing with contracts in as runtime checks. That's my default recommendation. In Github's case, they also might have access to both closed and open tools that Microsoft Research makes. MSR is awesome if you've never checked them out. Two examples applicable to system reliability and security: https://www.microsoft.com/en-us/research/project/ironclad/ https://www.microsoft.com/en-us/research/project/ironclad/ https://www.microsoft.com/en-us/research/publication/p-safe-asynchronous-event-driven-programming/ https://www.microsoft.com/en-us/research/publication/p-safe-... Plus some of their other tools in various states of usability: https://rise4fun.com/ https://rise4fun.com/
yjftsjthsd-h 7y ago
Alright, cue the usual: Monoculture bad Microsoft bad Git is distributed but all the things around it that we really need aren't You can set up git to push to multiple remotes automatically Nobody is actually using git in distributed mode Did I forget anything?
- davestephens 7y ago
  The usual fluff about migrating away to Gitlab?
  - eionh 7y ago
    "We would migrate to Gitlab, but it's currently down"
    - giovannibonetti 7y ago
      Given the context I guess he is talking about the self-hosted version. See also https://gogs.io/ https://gogs.io/
  - afandian 7y ago
    Irony of ironies. Gitlab's import-from-GitHub functionality has been broken ever since we tried to migrate away. (They're going ot fix it soon.)
    - russtrotter 7y ago
      I did this for the first time 2 days ago on two medium-size GH repos and it while was crazy slow, it eventually completed after about 8 hours or so. The actual bits seemed to finish in a relatively quick amount of time, but the project landing page continually showed "import in progress" for all those 8 hours.
      - brodock 7y ago
        Git is simple to import, but all other related data may be subject to API rate limits. I've opened an issue with some improvements suggestions, like surfacing when we are rate limited and allowing gradual access to the data while the import is going on: https://gitlab.com/gitlab-org/gitlab-ce/issues/66525 https://gitlab.com/gitlab-org/gitlab-ce/issues/66525
smartmic 7y ago
Git is designed as distributed VCS, so an outage of a centralized server should not matter too much. So far, the theory. Using Github excessively is the way back to centralization, nowadays handing control over to Microsoft. Everybody who is concerned (probably not too much) should have a look into alternatives.
- dspillett 7y ago
  Git itself is decentralised, but many (most? (all?)) of the workflows people build around it are not. Individuals and teams who already have what they need in the local repos can continue to work through an outage of the VCS part of github, but at the points where they need to collaborate (merging each other's changes, issue tracking, etc.) the workflows break down. Yes you could share changes in a more distributed manner, or workaround the outage in other ways, but in reality people will stop and wait for the central repo to be available again. Also, pulling changes from my repo to yours directly to avoid that part of the downed service doesn't solve the issue tracker or CI manager also being down. The clue is in the name: git HUB. That said, the issue is similar to how people see the results of aviation accidents: a jumbo going down takes out a lot of people but statistically, considering all air journeys, that is a lot less than the deaths that would happen due to equivalent car journeys. I suspect that everyone being a bit inconvenienced for a while when github/gitlab/other-centralised-service has issues like today doesn't add up to as much as the many small individual inconveniences they'd experience over time with their own local instances of, for example, gitlab, especially if including the routine maintenance time required which doesn't exist as much when using github, or gitlab managed instead of self-hosted - the outage just feels bigger than all the little problems because everyone is affected at once.
- reacweb 7y ago
  There are so many ways we become dependent of github, that nobody will shame on us because we are impacted by a github outage. In this context, is it useful to protect against outages ? It may be cheaper to do nothing.
- csomar 7y ago
  Yea because the alternative (hosting git yourself on your own servers) will have better availability and few issues (UX just came to mind!)
enlyth 7y ago
Resolved, all systems operational, that was quick.
u801e 7y ago
I see posts like this make the front page of hacker news every so often, but I don't recall seeing posts about things like making list outages for something like the LKML (Linux Kernel Mailing List) or an IRC network like Freenode.
- objclxt 7y ago
  Perhaps because Freenode has about 0.25% the number of users of Github - ~100,000 vs 40 million?
  - megous 7y ago
    No, it's just the US tech bubble. When my whole country and some was suddenly denied all google services for several hours, including CDNs, DNS, all cloud, gmail, android app store, gdocs, all everything, it didn't make any news here either, despite being much more disruptive, affecting 10+ million people, and being a much more intersting reminder of how much techies make other people and themselves dependent on this megacorp. All kinds of stuff broke, including "the entire internet" for people who were made to use 8.8.8.8 by their ISPs. E-shops that load jQuery from google stopped working, for no real reason whatsoever, people getting lost because they couldn't navigate anymore in the real world, etc. Since then I try to flag all this repetitive nonsense about slack/github going down for a few minutes, to no avail. It keeps comming back to the first page of HN. :D
    - cameronbrown 7y ago
      > When my whole country and some was suddenly denied all google services > didn't make any news here either Denied by government by any chance? Makes sense why it din't hit the news.
      - megous 7y ago
        Denied by google engineers making a mess with router misconfiguration. The internet traffic at key points went down by ~30% just by google taking itself offline. Also what was fun was everyone being blocked from all services that use reCAPTCHA.
jmisavage 7y ago
I kind of wish HN wouldn't show these outages unless they're going on for multiple hours or at least hide them after they're back up. Usually by the time I see them and then check myself the site is already back up. Maybe I just need to read HN more often.
- djsumdog 7y ago
  Yea, there are a lot of outage posts. It does kinda illustrate how many people depend on these core services though; and how little distributeiness or elasticity is built into the system. Linux repos and ISO download sites have TONS of mirrors, sometimes contributed by companies, but also a lot of universities, government and public institutions. Why is one of the largest core open source repositories centralized and private? Why are we so dependent on AWS, Google and others for large chunks of Internet services to be accessible? There is a deeper question here than just the outages; but rather why they affect so many people.
  - inickt 7y ago
    > Why is one of the largest core open source repositories centralized and private? Why are we so dependent on AWS, Google and others for large chunks of Internet services to be accessible? Just speculating but I think distributed systems are just inherently more complex. Serving multiple mirrors is easy, hosting across a set of providers and keeping things in sync gets harder. It's a problem, and besides building completely open source systems that are designed to have distributed instances, I think its a pretty hard thing to fix.
  - starbugs 7y ago
    I have a theory: Centralized infrastructure tends to be less complex and therefore cheaper. Economies of scale drive people into using a centralized monolith and then network effects do the rest. If you want a decentralized system to sustainably compete with a centralized one, the decentralized system must provide tangible and obvious surplus value in comparison to the centralized one or otherwise it is not economical. In reality that's often not the case. Usability is often worse or infrastructure is in favor of centralization (e.g. asymmetric broadband internet connections).
kome 7y ago
it's time to switch to SourceForge.
- sergiotapia 7y ago
  I've migrated all of our repos to codeplex.
scandox 7y ago
We're self hosting Gogs on a Hetzner VM and it's been a great experience. Of course it has less features. But it's simple and fast and I do about 10 minutes sys admin per quarter.
olah_1 7y ago
A peer-to-peer (IPFS) stack for code collaboration. https://radicle.xyz/ https://radicle.xyz/ A Radicle project contains a git repository, plus the associated issues and proposals. ^ Neat project
justinator 7y ago
What's a, "PR". (I'm being serious)
- janpot 7y ago
  https://help.github.com/en/articles/about-pull-requests https://help.github.com/en/articles/about-pull-requests
  - justinator 7y ago
    Thank you, of course - I was wondering if PR was the same PR as in Agile.
alexeiz 7y ago
Funny thing, I didn't even notice. In fact, being heavily reliant on Git and Github, I encountered maybe one or two issues a year when I couldn't push to Github because it was unavailable. In those cases, the progress can still be made locally, my workflow was barely affected. Of course, the deployment was impossible, but the downtime was never significant enough to worry or do anything about.
cproctor 7y ago
I've been having trouble getting GitHub to send me a verification email (after adding a new email address) for over a week. Anybody else having this problem? I wonder if it's connected.
fortran77 7y ago
If you don't the the "community" features of GitHub, we have had very good results with Amazon Code Commit. I've never experienced an outage, it's easy to manage, and very inexpensive. https://aws.amazon.com/codecommit/ https://aws.amazon.com/codecommit/
knorker 7y ago
Also still ongoing: They don't have IPv6.
truegrithub 7y ago
Everyone needs more sex. End of discussion. Who cares about github
- truegrithub 7y ago
  If people trucked more, nothing would be a problem. Nothing would look like a nail or a hammer. Truck it up. Truck github. You trucking beanbag cellar pieces of grits. Trucking makes the world go around. <3

140 points by dazbradbury 7y ago

Events like this make me realize how much I rely on my CI/CD pipeline for working on new features, deploying etc. I am often too lazy to run E2E tests on my machine if I know that on CI it will take only a few minutes to be done.

prepend 7y ago
I feel guilty because I’ve never installed GitLab’s CI process locally and have only run it for the past few years through git commits. I keep expecting someone to complain, but so far, so good. I expect similar poor behavior testing GitHub’s actions because it’s so convenient to only run through the service. It is important though that it’s possible to run without CI/cd. If it wasn’t available I would complain because of the risk of lock-in.

lousken 7y ago

Seems like github will soon become part of office 356 /s

xchaotic 7y ago

Justyna yesterday I was complaining about GitHub monoculture in the context of Atlassian sunsetting Mercurial. Outages like this are normal, but the problem is lack of good competitors.

arghwhat 7y ago
There are quite a decent list of competitors, all of which have similar or sometimes better offerings. There are also plenty of fully free and open source solutions, should you want to do things yourself. The Github monopoly is driven by its users.
- mschuetz 7y ago
  I tried gitlab but I hated the web interface. It was lacking in so many departments. Any other alternatives to github to try?
  - unixhero 7y ago
    What was lacking? The GUI on Gitlab is phenomenal imo.
    - hkitdog 7y ago
      On gitlab,repository and code searching function should be improved loll
      - penagwin 7y ago
        Same could be said for github. I regularly literally copy and paste a variable name from a file on Github itself - paste it into the search, and it finds 0 results. HOW COULD YOU FIND NOTHING?! Not even the file I copied it from? A simple grep search would've had no issue and would've been faster!
        arghwhat 7y ago
        Yeah, I always end up having to clone the repo to grep.
  - danskyr711 7y ago
    Bitbucket is pretty good
    - stevekemp 7y ago
      Bitbucket suffers from all the usual problems of an Atlassian-product: A badly designed user-interface, and frequent stability/downtime problems.
consolenaut 7y ago
I think you're right about a general move towards a Git monoculture (as opposed to a GitHub one), but there are dozens of great Git repo hosting/collaboration companies - GitLab, Bitbucket & SourceForge to name a few of the bigger ones
nickpsecurity 7y ago
The problem is bad design and/or integration of components. VMS clusters over leased lines went years without outages. Record was 17 years at a rail yard. They also did rolling upgrades across both different OS versions and CPU architectures. The methods are public if any competitors want to match or exceed them: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04622808 https://support.hpe.com/hpsc/doc/public/display?docId=emr_na... These modern cloud services using Linux ecosystem are doing rolling outages instead. I'm amazed people keep making them dependencies. Maybe their local systems using similar technology went down more often. Maybe this is an improvement. I'd still put mine on an OpenBSD or OpenVMS cluster if it was business-critical with lots of money on the line. I want it staying up, up, up. :)
- mugsie 7y ago
  Modern apps are designed to deal with part of them being down - it is the core upgrade strategy. VMS clusters were (and modern equivalents are) awesome, but they have a different speed of innovation / development. It is a trade off - perfect uptime, but that new feature could take a year, or 99.9% uptime, but it can be written and deployed tomorrow.
  - nickpsecurity 7y ago
    Their development velocity might have been due to their management or internal culture. You can get most of those results faster and on a budget. The best evidence was the startup behind FoundationDB continually improving their product in highly-competitive space while doing QA like this: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc Generally, on top of code reviews, one can get high return with minimal labor and hardware with a combo of Design-by-Contract, contract/spec/property-based testing, low-F.P. static analysis, and fuzzing with contracts in as runtime checks. That's my default recommendation. In Github's case, they also might have access to both closed and open tools that Microsoft Research makes. MSR is awesome if you've never checked them out. Two examples applicable to system reliability and security: https://www.microsoft.com/en-us/research/project/ironclad/ https://www.microsoft.com/en-us/research/project/ironclad/ https://www.microsoft.com/en-us/research/publication/p-safe-asynchronous-event-driven-programming/ https://www.microsoft.com/en-us/research/publication/p-safe-... Plus some of their other tools in various states of usability: https://rise4fun.com/ https://rise4fun.com/

yjftsjthsd-h 7y ago

Alright, cue the usual: Monoculture bad Microsoft bad Git is distributed but all the things around it that we really need aren't You can set up git to push to multiple remotes automatically Nobody is actually using git in distributed mode Did I forget anything?

davestephens 7y ago
The usual fluff about migrating away to Gitlab?
- eionh 7y ago
  "We would migrate to Gitlab, but it's currently down"
  - giovannibonetti 7y ago
    Given the context I guess he is talking about the self-hosted version. See also https://gogs.io/ https://gogs.io/
- afandian 7y ago
  Irony of ironies. Gitlab's import-from-GitHub functionality has been broken ever since we tried to migrate away. (They're going ot fix it soon.)
  - russtrotter 7y ago
    I did this for the first time 2 days ago on two medium-size GH repos and it while was crazy slow, it eventually completed after about 8 hours or so. The actual bits seemed to finish in a relatively quick amount of time, but the project landing page continually showed "import in progress" for all those 8 hours.
    - brodock 7y ago
      Git is simple to import, but all other related data may be subject to API rate limits. I've opened an issue with some improvements suggestions, like surfacing when we are rate limited and allowing gradual access to the data while the import is going on: https://gitlab.com/gitlab-org/gitlab-ce/issues/66525 https://gitlab.com/gitlab-org/gitlab-ce/issues/66525

smartmic 7y ago

Git is designed as distributed VCS, so an outage of a centralized server should not matter too much. So far, the theory. Using Github excessively is the way back to centralization, nowadays handing control over to Microsoft. Everybody who is concerned (probably not too much) should have a look into alternatives.

dspillett 7y ago
Git itself is decentralised, but many (most? (all?)) of the workflows people build around it are not. Individuals and teams who already have what they need in the local repos can continue to work through an outage of the VCS part of github, but at the points where they need to collaborate (merging each other's changes, issue tracking, etc.) the workflows break down. Yes you could share changes in a more distributed manner, or workaround the outage in other ways, but in reality people will stop and wait for the central repo to be available again. Also, pulling changes from my repo to yours directly to avoid that part of the downed service doesn't solve the issue tracker or CI manager also being down. The clue is in the name: git HUB. That said, the issue is similar to how people see the results of aviation accidents: a jumbo going down takes out a lot of people but statistically, considering all air journeys, that is a lot less than the deaths that would happen due to equivalent car journeys. I suspect that everyone being a bit inconvenienced for a while when github/gitlab/other-centralised-service has issues like today doesn't add up to as much as the many small individual inconveniences they'd experience over time with their own local instances of, for example, gitlab, especially if including the routine maintenance time required which doesn't exist as much when using github, or gitlab managed instead of self-hosted - the outage just feels bigger than all the little problems because everyone is affected at once.
reacweb 7y ago
There are so many ways we become dependent of github, that nobody will shame on us because we are impacted by a github outage. In this context, is it useful to protect against outages ? It may be cheaper to do nothing.
csomar 7y ago
Yea because the alternative (hosting git yourself on your own servers) will have better availability and few issues (UX just came to mind!)

enlyth 7y ago

Resolved, all systems operational, that was quick.

u801e 7y ago

I see posts like this make the front page of hacker news every so often, but I don't recall seeing posts about things like making list outages for something like the LKML (Linux Kernel Mailing List) or an IRC network like Freenode.

objclxt 7y ago
Perhaps because Freenode has about 0.25% the number of users of Github - ~100,000 vs 40 million?
- megous 7y ago
  No, it's just the US tech bubble. When my whole country and some was suddenly denied all google services for several hours, including CDNs, DNS, all cloud, gmail, android app store, gdocs, all everything, it didn't make any news here either, despite being much more disruptive, affecting 10+ million people, and being a much more intersting reminder of how much techies make other people and themselves dependent on this megacorp. All kinds of stuff broke, including "the entire internet" for people who were made to use 8.8.8.8 by their ISPs. E-shops that load jQuery from google stopped working, for no real reason whatsoever, people getting lost because they couldn't navigate anymore in the real world, etc. Since then I try to flag all this repetitive nonsense about slack/github going down for a few minutes, to no avail. It keeps comming back to the first page of HN. :D
  - cameronbrown 7y ago
    > When my whole country and some was suddenly denied all google services > didn't make any news here either Denied by government by any chance? Makes sense why it din't hit the news.
    - megous 7y ago
      Denied by google engineers making a mess with router misconfiguration. The internet traffic at key points went down by ~30% just by google taking itself offline. Also what was fun was everyone being blocked from all services that use reCAPTCHA.

jmisavage 7y ago

I kind of wish HN wouldn't show these outages unless they're going on for multiple hours or at least hide them after they're back up. Usually by the time I see them and then check myself the site is already back up. Maybe I just need to read HN more often.

djsumdog 7y ago
Yea, there are a lot of outage posts. It does kinda illustrate how many people depend on these core services though; and how little distributeiness or elasticity is built into the system. Linux repos and ISO download sites have TONS of mirrors, sometimes contributed by companies, but also a lot of universities, government and public institutions. Why is one of the largest core open source repositories centralized and private? Why are we so dependent on AWS, Google and others for large chunks of Internet services to be accessible? There is a deeper question here than just the outages; but rather why they affect so many people.
- inickt 7y ago
  > Why is one of the largest core open source repositories centralized and private? Why are we so dependent on AWS, Google and others for large chunks of Internet services to be accessible? Just speculating but I think distributed systems are just inherently more complex. Serving multiple mirrors is easy, hosting across a set of providers and keeping things in sync gets harder. It's a problem, and besides building completely open source systems that are designed to have distributed instances, I think its a pretty hard thing to fix.
- starbugs 7y ago
  I have a theory: Centralized infrastructure tends to be less complex and therefore cheaper. Economies of scale drive people into using a centralized monolith and then network effects do the rest. If you want a decentralized system to sustainably compete with a centralized one, the decentralized system must provide tangible and obvious surplus value in comparison to the centralized one or otherwise it is not economical. In reality that's often not the case. Usability is often worse or infrastructure is in favor of centralization (e.g. asymmetric broadband internet connections).

kome 7y ago

it's time to switch to SourceForge.

sergiotapia 7y ago
I've migrated all of our repos to codeplex.

scandox 7y ago

We're self hosting Gogs on a Hetzner VM and it's been a great experience. Of course it has less features. But it's simple and fast and I do about 10 minutes sys admin per quarter.

olah_1 7y ago

A peer-to-peer (IPFS) stack for code collaboration. https://radicle.xyz/ https://radicle.xyz/ A Radicle project contains a git repository, plus the associated issues and proposals. ^ Neat project

justinator 7y ago

What's a, "PR". (I'm being serious)

janpot 7y ago
https://help.github.com/en/articles/about-pull-requests https://help.github.com/en/articles/about-pull-requests
- justinator 7y ago
  Thank you, of course - I was wondering if PR was the same PR as in Agile.

alexeiz 7y ago

Funny thing, I didn't even notice. In fact, being heavily reliant on Git and Github, I encountered maybe one or two issues a year when I couldn't push to Github because it was unavailable. In those cases, the progress can still be made locally, my workflow was barely affected. Of course, the deployment was impossible, but the downtime was never significant enough to worry or do anything about.

cproctor 7y ago

I've been having trouble getting GitHub to send me a verification email (after adding a new email address) for over a week. Anybody else having this problem? I wonder if it's connected.

fortran77 7y ago

If you don't the the "community" features of GitHub, we have had very good results with Amazon Code Commit. I've never experienced an outage, it's easy to manage, and very inexpensive. https://aws.amazon.com/codecommit/ https://aws.amazon.com/codecommit/

knorker 7y ago

Also still ongoing: They don't have IPv6.

truegrithub 7y ago

Everyone needs more sex. End of discussion. Who cares about github

truegrithub 7y ago
If people trucked more, nothing would be a problem. Nothing would look like a nail or a hammer. Truck it up. Truck github. You trucking beanbag cellar pieces of grits. Trucking makes the world go around. <3