4 ms·

I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger. The statu

I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger. The status page says all is well, though: https://www.githubstatus.com/ https://www.githubstatus.com/. Hilarious.

TacticalCoder 2y ago
> I've never seen an outage this big. I remember a time when systems would boast about their "five nines" uptime. It was before anything "cloud" appeared.
ergocoder 2y ago
I wonder why the status just doesn't ping github.com for 200. That seems easy to do.
- tinyhitman 2y ago
  delaying SLA
  - sebmellen 2y ago
    This is at least a multi-million dollar payout (if they admit to it). All GitHub Pages say > We're having a really bad day. > The Unicorns have taken over. We're doing our best to get them under control and get GitHub back up and running.
    - cbates 2y ago
      Seems slightly unproffesional for a massive company like Github/Microsoft.
      - xp84 2y ago
        I disagree. This hurts no one, and not everything needs to be sanitized and painted over with bland corporatespeak.
        majewsky 2y ago
        I don't think they were asking for corporate speak. But at least I would find a plain technical error message like "cannot contact file server" much more respectable than something like "unicorns are hugging our servers uwu".
        COMMENT___ 2y ago
        This “ironic” and “humorous” style of errors and UI captions is the actual new corporate speak. I’d prefer dumb error messages rather than some shit someone over the ocean thinks is smart and humorous. And it’s not funny at all when it’s a global outage impacting my business and my $$$.
      - colimbarna 2y ago
        It's closer to the truth than you usually get. They're having a bad day, it's completely true. It's the start of my day, but I guess this is the middle of the night for them. There's no such thing as unicorns, but that just highlights the metaphorical nature of the remaining claim - getting Unicorns under control means solving their problems. Normally "professional" corporate speak means avoiding saying anything whose meaning is plain on its face and disconfirmable while avoiding the implication that the company is run and operated by humans. This is a model. (Obviously the came up with the message in advance, which just goes to show that someone in the company is well enough rounded to know that if it is displayed, they're having a bad day.)
        wrs 2y ago
        GitHub is (was?) a Rails application, so it was probably originally running behind Unicorn [0], if it isn’t still. So the unicorns are (were) real. [0] https://en.wikipedia.org/wiki/Unicorn_(web_server) https://en.wikipedia.org/wiki/Unicorn_(web_server)
    - ljahier 2y ago
      At the moment, all github services seem to be restored, and the github status indicates that the problem is still ongoing. I don't think it's related to the SLA, but rather to the monitoring, which is not live. There are a few minutes of delay.
- fragmede 2y ago
  from where? they don't only have one load balancer, so you'd still have the problem of the page showing green when it's not loading for some folk?
  - ergocoder 2y ago
    At Github's scale, why wouldn't they put a ping monitor from every continent at least? Then, you would show the status based on the continent.
    - intelVISA 2y ago
      That would be self-defeating given that it's a Rails app.
    - fragmede 2y ago
      Where on the continent? GitHub is undoubtedly doing blackbox testing internally and has multiple such monitors but that's not going to capture every customer's route to them, leading to the same problem - customers experience GitHub being down, despite monitoring saying it's mostly up. Thus the impass. Even doing whitebox testing, where you know the internals and can this place sensors intelligently, even just for ingress, you're still at the mercy of the Internet. If a sensor that's basically in the same datacenter says you're up, but the route into the datacenter is down, then what? multiply this by the complexity of the whole site, and monitoring it all with 100% fidelity is impossible. Not that it's not worth it to try, there's a team at GitHub that works on monitoring, but beyond motivation about keeping the SLA up, as a customer, unless you notice it's down, is it really down? In a globally distributed system, downtime, except for catastrophic downtime like this, is hard to define on a whole-site basis for all customers.
      - laserlight 2y ago
        > 100% fidelity is impossible I don't think anybody asked for 100% fidelity. We are talking about a complete outage that affected at least North America and Europe. If the status page shows green in such a case, its fidelity is around 50%. People expect better from GitHub.
- bigiain 2y ago
  To be fair - I really couldn't care less is the homepage is loading or not. So long as I can fetch/commit to my repos, pretty much everything else is of secondary, tertiary, or no real importance to me. (At work, I do indeed have systems running that monitor 200 statuses from client project homepages, almost all of which show better that 99.999% uptimes. And are practically useless. Most of them also monitor "canary" API requests which I strive to keep at 99.99% but don't always manage to achieve 99.9% - which is the very best and most expensive SLA we'll commit to.)
karmakaze 2y ago
I get the angry unicorn page "No server is currently available to service your request. Sorry about that. Please try refreshing and contact us if the problem persists. Contact Support — GitHub Status — @githubstatus" with that last link going to https://x.com/githubstatus https://x.com/githubstatus showing "GitHub Status Oct 22, 2018 Everything operating normally."
- kalkin 2y ago
  I think this is because logged-out Twitter now shows top Tweets of all time from a user, rather than most recent Tweets. Good reason why companies shouldn't be using Twitter/X for status updates anymore!
  - xp84 2y ago
    Thank you! I was wondering why all I could see was useless content there!
    - gamer191 2y ago
      Use https://xcancel.com/ https://xcancel.com/ (eg https://xcancel.com/githubstatus https://xcancel.com/githubstatus)
- TwiztidK 2y ago
  The era of Twitter/X status pages needs to come to an end given how unusable it is if you aren't logged in.
  - blt 2y ago
    Making logins required to view twitter was the ultimate bed shitting move. The whole point of twitter was to be a broadcast medium. Tweets were viewable without following or logging in. There is a huge vacuum in that space now.
    - raxxorraxor 2y ago
kinduff 2y ago
They are flipping the switches now, status page just changed.
manquer 2y ago
Status page updates with "degraded availability". lol
RIMR 2y ago
Wow, the status page only just now started reporting issues, and it still doesn't seem to communicate the scale of the issue. People use this page for guidance. I guess now we know how much it can be trusted.
- ikiris 2y ago
  It’s used to ease their comms, not a real time status board pointing at their monitoring.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
purkka 2y ago
I have to wonder how a company at the scale of GitHub can be so bad at keeping track of their status. Now 4 out of 10 services are marked as "Incident", yet most of the others are also completely dead.
- x86a 2y ago
  This is intentional. It's mostly a matter of discussing how to communicate it publicly and when to flip the switch to start the SLA timer. Also coordinating incident response during a huge outage is always challenging.
  - thiagocsf 2y ago
    That it may be but there’s no excuse. Declare an incident first, investigate later. Cheating SLAs by delaying the incident is a good way to erode trust within and without.
    - antimemetics 2y ago
      > Declare an incident first, investigate later. If that would be the best way to deal with it- why is literally no one doing it this way and what does that tell you?
      - adgjlsfhk1 2y ago
        because it involves admitting that you messed up which companies are often disensentivized to do
      - ErikBjare 2y ago
        False positives?
- xuancanh 2y ago
  It's because of the way most companies build their status dashboards. There are usually at least 2 dashboards, one internal dashboard and one external dashboard. The internal dashboard is the actual monitoring dashboard, where it will be hooked up with other monitoring data sources. The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the external dashboard will be updated to avoid flaky monitors and alerts. It will also affect SLAs so it needs multiple levels of approval to change the status, that's why there are some delays.
rvz 2y ago
Looks like we have a full house outage at GitHub with everything down. Much worse than the so-called Twitter / X recent speed-bump that was screeched at and quickly forgotten. I don't think GitHub has recovered from the monthly incidents that keeps occurring. Quite frankly it is the expectation that something will go down every month at GitHub which shows how unreliable the service is and this has happened for years. I guess this 4 year old prediction post really aged well after all about self-hosting and not going all in on Github [0] [0] https://news.ycombinator.com/item?id=22868406 https://news.ycombinator.com/item?id=22868406
- dataspun 2y ago
  statute of limitations for HN comment predictions is 3 years.
Lanedo 2y ago
Twitter now has: We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back. https://x.com/githubstatus/status/1823864449494569023 https://x.com/githubstatus/status/1823864449494569023
Lanedo 2y ago
Github seems to be coming back up: https://downdetector.com/status/github/ https://downdetector.com/status/github/
temp0826 2y ago
Used to work ops at AWS. I don't know if it's still the case but it required VERY HIGH management approval to actually flip any lights on their "status page" (likely it was referenced in some way for SLAs and refunding customers).
- smsm42 2y ago
  That is an excellent illustration to Goodhart's law. We're going to have this avesome status page, but since if we update it the clients would notice the system is down, we're going to put a lot of barriers to putting the actual status on that page. Also probably a class action suit lurking somewhere in there eventually.
saul-paterson 2y ago
FWIW, our self-hosted Gitea instance has not had a single second of unplanned downtime in five years we've been running it. And there wasn't much _planned_ downtime because it's really easy to upgrade (pull a new image and recreate the container — takes out the instance for maybe 15 seconds late at night), and full backups are handled live thanks to zfs. Migration to a new host takes another 15 seconds thanks to both zfs and containers. I don't know how many GitHub downtime reports I've seen during that time, we're probably into high dozens by now.
- chrisallenlane 2y ago
  I've been running Gitea on my homelab for a few months now. It's fantastic. It's like a snapshot of a point in time when GitHub was actually good, before it got enshittified by all of the social and AI nonsense. I've been moving most of my projects off of GitHub and into Gitea, and will continue to do so.
rozenmd 2y ago
Here, we caught 35 minutes of downtime: https://github.onlineornot.com/incidents/6Yyj8YWD94zE https://github.onlineornot.com/incidents/6Yyj8YWD94zE

TacticalCoder 2y ago

> I've never seen an outage this big. I remember a time when systems would boast about their "five nines" uptime. It was before anything "cloud" appeared.

ergocoder 2y ago

I wonder why the status just doesn't ping github.com for 200. That seems easy to do.

tinyhitman 2y ago
delaying SLA
- sebmellen 2y ago
  This is at least a multi-million dollar payout (if they admit to it). All GitHub Pages say > We're having a really bad day. > The Unicorns have taken over. We're doing our best to get them under control and get GitHub back up and running.
  - cbates 2y ago
    Seems slightly unproffesional for a massive company like Github/Microsoft.
    - xp84 2y ago
      I disagree. This hurts no one, and not everything needs to be sanitized and painted over with bland corporatespeak.
      - majewsky 2y ago
        I don't think they were asking for corporate speak. But at least I would find a plain technical error message like "cannot contact file server" much more respectable than something like "unicorns are hugging our servers uwu".
      - COMMENT___ 2y ago
        This “ironic” and “humorous” style of errors and UI captions is the actual new corporate speak. I’d prefer dumb error messages rather than some shit someone over the ocean thinks is smart and humorous. And it’s not funny at all when it’s a global outage impacting my business and my $$$.
    - colimbarna 2y ago
      It's closer to the truth than you usually get. They're having a bad day, it's completely true. It's the start of my day, but I guess this is the middle of the night for them. There's no such thing as unicorns, but that just highlights the metaphorical nature of the remaining claim - getting Unicorns under control means solving their problems. Normally "professional" corporate speak means avoiding saying anything whose meaning is plain on its face and disconfirmable while avoiding the implication that the company is run and operated by humans. This is a model. (Obviously the came up with the message in advance, which just goes to show that someone in the company is well enough rounded to know that if it is displayed, they're having a bad day.)
      - wrs 2y ago
        GitHub is (was?) a Rails application, so it was probably originally running behind Unicorn [0], if it isn’t still. So the unicorns are (were) real. [0] https://en.wikipedia.org/wiki/Unicorn_(web_server) https://en.wikipedia.org/wiki/Unicorn_(web_server)
  - ljahier 2y ago
    At the moment, all github services seem to be restored, and the github status indicates that the problem is still ongoing. I don't think it's related to the SLA, but rather to the monitoring, which is not live. There are a few minutes of delay.
fragmede 2y ago
from where? they don't only have one load balancer, so you'd still have the problem of the page showing green when it's not loading for some folk?
- ergocoder 2y ago
  At Github's scale, why wouldn't they put a ping monitor from every continent at least? Then, you would show the status based on the continent.
  - intelVISA 2y ago
    That would be self-defeating given that it's a Rails app.
  - fragmede 2y ago
    Where on the continent? GitHub is undoubtedly doing blackbox testing internally and has multiple such monitors but that's not going to capture every customer's route to them, leading to the same problem - customers experience GitHub being down, despite monitoring saying it's mostly up. Thus the impass. Even doing whitebox testing, where you know the internals and can this place sensors intelligently, even just for ingress, you're still at the mercy of the Internet. If a sensor that's basically in the same datacenter says you're up, but the route into the datacenter is down, then what? multiply this by the complexity of the whole site, and monitoring it all with 100% fidelity is impossible. Not that it's not worth it to try, there's a team at GitHub that works on monitoring, but beyond motivation about keeping the SLA up, as a customer, unless you notice it's down, is it really down? In a globally distributed system, downtime, except for catastrophic downtime like this, is hard to define on a whole-site basis for all customers.
    - laserlight 2y ago
      > 100% fidelity is impossible I don't think anybody asked for 100% fidelity. We are talking about a complete outage that affected at least North America and Europe. If the status page shows green in such a case, its fidelity is around 50%. People expect better from GitHub.
bigiain 2y ago
To be fair - I really couldn't care less is the homepage is loading or not. So long as I can fetch/commit to my repos, pretty much everything else is of secondary, tertiary, or no real importance to me. (At work, I do indeed have systems running that monitor 200 statuses from client project homepages, almost all of which show better that 99.999% uptimes. And are practically useless. Most of them also monitor "canary" API requests which I strive to keep at 99.99% but don't always manage to achieve 99.9% - which is the very best and most expensive SLA we'll commit to.)

karmakaze 2y ago

I get the angry unicorn page "No server is currently available to service your request. Sorry about that. Please try refreshing and contact us if the problem persists. Contact Support — GitHub Status — @githubstatus" with that last link going to https://x.com/githubstatus https://x.com/githubstatus showing "GitHub Status Oct 22, 2018 Everything operating normally."

kalkin 2y ago
I think this is because logged-out Twitter now shows top Tweets of all time from a user, rather than most recent Tweets. Good reason why companies shouldn't be using Twitter/X for status updates anymore!
- xp84 2y ago
  Thank you! I was wondering why all I could see was useless content there!
  - gamer191 2y ago
    Use https://xcancel.com/ https://xcancel.com/ (eg https://xcancel.com/githubstatus https://xcancel.com/githubstatus)
TwiztidK 2y ago
The era of Twitter/X status pages needs to come to an end given how unusable it is if you aren't logged in.
- blt 2y ago
  Making logins required to view twitter was the ultimate bed shitting move. The whole point of twitter was to be a broadcast medium. Tweets were viewable without following or logging in. There is a huge vacuum in that space now.
  - raxxorraxor 2y ago

kinduff 2y ago

They are flipping the switches now, status page just changed.

manquer 2y ago

Status page updates with "degraded availability". lol

RIMR 2y ago

Wow, the status page only just now started reporting issues, and it still doesn't seem to communicate the scale of the issue. People use this page for guidance. I guess now we know how much it can be trusted.

ikiris 2y ago
It’s used to ease their comms, not a real time status board pointing at their monitoring.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

purkka 2y ago

I have to wonder how a company at the scale of GitHub can be so bad at keeping track of their status. Now 4 out of 10 services are marked as "Incident", yet most of the others are also completely dead.

x86a 2y ago
This is intentional. It's mostly a matter of discussing how to communicate it publicly and when to flip the switch to start the SLA timer. Also coordinating incident response during a huge outage is always challenging.
- thiagocsf 2y ago
  That it may be but there’s no excuse. Declare an incident first, investigate later. Cheating SLAs by delaying the incident is a good way to erode trust within and without.
  - antimemetics 2y ago
    > Declare an incident first, investigate later. If that would be the best way to deal with it- why is literally no one doing it this way and what does that tell you?
    - adgjlsfhk1 2y ago
      because it involves admitting that you messed up which companies are often disensentivized to do
    - ErikBjare 2y ago
      False positives?
xuancanh 2y ago
It's because of the way most companies build their status dashboards. There are usually at least 2 dashboards, one internal dashboard and one external dashboard. The internal dashboard is the actual monitoring dashboard, where it will be hooked up with other monitoring data sources. The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the external dashboard will be updated to avoid flaky monitors and alerts. It will also affect SLAs so it needs multiple levels of approval to change the status, that's why there are some delays.

rvz 2y ago

Looks like we have a full house outage at GitHub with everything down. Much worse than the so-called Twitter / X recent speed-bump that was screeched at and quickly forgotten. I don't think GitHub has recovered from the monthly incidents that keeps occurring. Quite frankly it is the expectation that something will go down every month at GitHub which shows how unreliable the service is and this has happened for years. I guess this 4 year old prediction post really aged well after all about self-hosting and not going all in on Github [0] [0] https://news.ycombinator.com/item?id=22868406 https://news.ycombinator.com/item?id=22868406

dataspun 2y ago
statute of limitations for HN comment predictions is 3 years.

Lanedo 2y ago

Twitter now has: We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back. https://x.com/githubstatus/status/1823864449494569023 https://x.com/githubstatus/status/1823864449494569023

Lanedo 2y ago

Github seems to be coming back up: https://downdetector.com/status/github/ https://downdetector.com/status/github/

temp0826 2y ago

Used to work ops at AWS. I don't know if it's still the case but it required VERY HIGH management approval to actually flip any lights on their "status page" (likely it was referenced in some way for SLAs and refunding customers).

smsm42 2y ago
That is an excellent illustration to Goodhart's law. We're going to have this avesome status page, but since if we update it the clients would notice the system is down, we're going to put a lot of barriers to putting the actual status on that page. Also probably a class action suit lurking somewhere in there eventually.

saul-paterson 2y ago

FWIW, our self-hosted Gitea instance has not had a single second of unplanned downtime in five years we've been running it. And there wasn't much _planned_ downtime because it's really easy to upgrade (pull a new image and recreate the container — takes out the instance for maybe 15 seconds late at night), and full backups are handled live thanks to zfs. Migration to a new host takes another 15 seconds thanks to both zfs and containers. I don't know how many GitHub downtime reports I've seen during that time, we're probably into high dozens by now.

chrisallenlane 2y ago
I've been running Gitea on my homelab for a few months now. It's fantastic. It's like a snapshot of a point in time when GitHub was actually good, before it got enshittified by all of the social and AI nonsense. I've been moving most of my projects off of GitHub and into Gitea, and will continue to do so.

rozenmd 2y ago

Here, we caught 35 minutes of downtime: https://github.onlineornot.com/incidents/6Yyj8YWD94zE https://github.onlineornot.com/incidents/6Yyj8YWD94zE