4 ms·

Can't they at least fix their status page? https://www.githubstatus.com/ https://www.githubstatus.com/ It returns `All Systems Operational`. I mean what's the p

Can't they at least fix their status page? https://www.githubstatus.com/ https://www.githubstatus.com/ It returns `All Systems Operational`. I mean what's the point of having a status page if it returns wrong info?

res0nat0r 5y ago
These things don't/can't get updated instantly. I was doing some work as of ~5 minutes ago and it was working fine, and is unavailable now. If it is a major outage it will likely be updated shortly.
- MrStonedOne 5y ago
  My status page for an open source multiplayer game updates the status page for each server on a minute by minute bases. They can do better, they just don't want to.
  - res0nat0r 5y ago
    Large orgs (like Github) don't want or use automated status updates. There is usually always some service having issues in complex systems and immediately updating a public status page does more harm than good, including false alarms which may not affect anything public facing.
    - Dylan16807 5y ago
      If you don't want to report sufficiently small issues, you can put that into the code, can't you? Besides that, how are you going to cause "more harm than good"?
      - derefr 5y ago
        More harm than good to the company's own long-term reputation. A status page is a kind of PR. Think of it like a policy for a flight attendant to come out into the cabin to tell everyone what's going on when the plane encounters turbulence. That policy is Public Relations -driven. You only do it if you expect that it's positive PR, compared to not doing it — i.e. if telling people what's going on is boosting your reputation compared to saying nothing at all. If a status page just makes your stakeholders think your service is crappy, such that you'd be better off with no status page at all... then why have a status page? It's not doing its job as a PR tool.
        mjw1007 5y ago
        It seems to me you are now agreeing with the original « They can do better, they just don't want to. »
        derefr 5y ago
        I read the line specifically as "The employees can do better; they just don't want to try." But that's not true. The company could do better. But the individual employees cannot. The individual employees are constrained by the profit motive of the company. They are not allowed by corporate policy to set up automatic status updates, for about the same reason they're not allowed to post their corporate log-in credentials: that the result would very likely be disastrous to the company's bottom line. (Though, really, the corporations in most verticals are in a race-to-the-bottom in most respects. Even if you treat GitHub as a single entity capable of coherent desires, it probably doesn't desire to avoid automatic status updates. It needs to avoid them, to survive in a competitive market where everyone else is also avoiding them. People — and corporations — do lots of things they don't want to do, to survive.)
  - derefr 5y ago
    No, I don't think they can. An MMO is a very simple system, in that there's only one Service Level Indicator (SLI) that devs, shareholders, and players all agree on. That SLI is "can a player connect to the server, and perform regular gameplay actions, without a ridiculous amount of per-action latency." GitHub, meanwhile, is a million different things to a hundred million different people. Users of e.g. Homebrew, with its big monolithic ports system hosted as a github repo, have a very different SLI for Github than do users of some language-ecosystem package manager that allows you to pull deps directly from Github; than do people who depend on GitHub Actions to CI their builds on push; than do people doing code-review to others' PRs; than do people using Github mostly for its Wiki, or Issues, or downloading Releases, or Github Pages, or even just reading single-page-with-a-README repos, ala the various $FOO-awesome projects. For many of these use-cases, Github isn't degraded right now. For others, it is. If you ask for Github (or any service with this many different use-cases and stakeholders) to measure by the union of all these SLIs, then the service would literally never be not-degraded. In systems of sufficient scale, there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once. Never has been, never will be. And anything less than just going for the union of all those SLIs, is asking Github to exercise human judgement over which kinds of service degradation qualify as part of their own SLOs. Which is exactly what they're doing. Certainly, internal to services like this, there are all sorts of alerting systems constantly going off to tell SREs what things need fixing. But not all of those things immediately, or even quickly, or even ever, translate to SLO violations. There are some outlier users whose use-cases just break the system's semantics, where those use-cases just aren't "in scope" for the SLO. As long as those users are only breaking the system for themselves, the degradation they experience won't ever translate to an SLO breakage.
    - thih9 5y ago
      You seem to be applying different rules to MMOs and Github, and I don't understand why. I'd say that there are many ways of looking at this; there exist complex MMOs; and one could look at Github from the point of view of an average user. E.g., a bit tongue in cheek: > An MMO is a very simple system, in that there's only one Service Level Indicator (SLI) that devs, shareholders, and players all agree on. That SLI is "can a player connect to the server, and perform regular gameplay actions, without a ridiculous amount of per-action latency." Wouldn't you say that in an MMO of sufficient scale there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once? > In systems of sufficient scale, there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once. Never has been, never will be. Couldn't we redefine SLIs as "can the user connect to the server and perform regular user actions without a ridiculous amount of per-action latency"?
qwertox 5y ago
They have a banner "Investigating - We are investigating reports of degraded performance for GitHub Actions. -- Nov 27, 20:43 UTC" which should suffice for a start.
- spapas82 5y ago
  Yes now it has been properly updated. However it was down for 10-15 minutes before their status page was updated...
drdaeman 5y ago
There is a point. Even two. 1. It clearly indicates that automatic systems are failing to detect the outage. 2. It also indicates that no one is aware about the incident to manually signal the outage (or that there is no manual override). Basically, it makes a difference between "yeah, shit happened, we know (and maybe working on it)" and "hah, they don't even know themselves".
- res0nat0r 5y ago
  Most of these over arching status pages are manually run, intentionally by design.
- the_duke 5y ago
  Almost no one has automatic status pages anymore. Partially because these large systems have some kind of ongoing issue at any given time, so it's challenging to provide a meaningful live status that isn't a bit misleading and could cause misdirected panic. Partially because you don't want to give potential attackers (eg ddos) any insight if/how their methods are affecting your systems. Partially because there are SLAs and reputation at risk, and you don't want to admit to any more downtime than you absolutely have to.
  - derefr 5y ago
    If you had a really robust system, it'd be fun to just slap a read-only mirror of your internal metrics dashboard onto the public Internet for anyone to browse and slice as they please. It'd be a brag, kinda. Of course, in the real world, I don't think there's a single IT admin who didn't just start nervously sweating and pulling at their collar after reading the above, imagining it as something their CEO said and encouraging them to target it. Nobody can really do this — and that in turn says something important about how those metrics really look in most systems.
    - Jweb_Guru 5y ago
      FWIW, I've worked on systems that have internal SLAs orders of magnitude higher than what they promise to the public. I think it's more just that there's no advantage to doing something like this as long as none of your competitors are. The status quo is that people's systems are really opaque and vastly underpromise what they should be capable of, and in exchange you get to absorb some unplanned downtime due to issues with underlying systems that you have little control over.
random_thoughts 5y ago
They are currently updating it one by one
- cube00 5y ago
  ...and firing off automated tweets per product.
ImprovedSilence 5y ago
lolol. Clearly it's just degraded service. It seems The Management hasn't approved declaring it an outage, that would just ruin uptime metrics!
charles_f 5y ago
It's updated now.
crecker 5y ago
I would not be impressed if they hear about the down for a trending thread on HN.
- JohnFen 5y ago
  Honestly, HN is the best status indicator on the net right now.
- ryantgtg 5y ago
  I don’t work for github, so on a personal note I was about to create an issue in a repo, and it wasn’t loading. My go to “is my router messed up?” check is to load HN because it’s so reliable and fast. And lo, the top post was about github being down!
  - crecker 5y ago
    Ha. I checked HN for the same reason, I was not able to reach Github via university network and I thought that was the time my university messed up with DNS. HN seems to be going down for the massive amount of requests!
danjac 5y ago
There's lies, damn lies and SAAS status pages.
Flankk 5y ago
Status page is also down and is returning a false positive. If you check the status page status page it shows the status page is down.
chippiewill 5y ago
Status pages are almost never automated these days, they cause more problems than they solve. To be fair, redditstatus.com is quite nice with their sparkline headline metrics. It at least lets you know _something_ is happening even if they haven't yet declared an incident.
hdjjhhvvhga 5y ago
The only status pages that make sense are the ones maintained by a third party, not the owner. Also for technical reasons.
- cyberge99 5y ago
  And legal reasons.
ranklord 5y ago
It's updated already. Relatively fast... :/

res0nat0r 5y ago

These things don't/can't get updated instantly. I was doing some work as of ~5 minutes ago and it was working fine, and is unavailable now. If it is a major outage it will likely be updated shortly.

MrStonedOne 5y ago
My status page for an open source multiplayer game updates the status page for each server on a minute by minute bases. They can do better, they just don't want to.
- res0nat0r 5y ago
  Large orgs (like Github) don't want or use automated status updates. There is usually always some service having issues in complex systems and immediately updating a public status page does more harm than good, including false alarms which may not affect anything public facing.
  - Dylan16807 5y ago
    If you don't want to report sufficiently small issues, you can put that into the code, can't you? Besides that, how are you going to cause "more harm than good"?
    - derefr 5y ago
      More harm than good to the company's own long-term reputation. A status page is a kind of PR. Think of it like a policy for a flight attendant to come out into the cabin to tell everyone what's going on when the plane encounters turbulence. That policy is Public Relations -driven. You only do it if you expect that it's positive PR, compared to not doing it — i.e. if telling people what's going on is boosting your reputation compared to saying nothing at all. If a status page just makes your stakeholders think your service is crappy, such that you'd be better off with no status page at all... then why have a status page? It's not doing its job as a PR tool.
      - mjw1007 5y ago
        It seems to me you are now agreeing with the original « They can do better, they just don't want to. »
        derefr 5y ago
        I read the line specifically as "The employees can do better; they just don't want to try." But that's not true. The company could do better. But the individual employees cannot. The individual employees are constrained by the profit motive of the company. They are not allowed by corporate policy to set up automatic status updates, for about the same reason they're not allowed to post their corporate log-in credentials: that the result would very likely be disastrous to the company's bottom line. (Though, really, the corporations in most verticals are in a race-to-the-bottom in most respects. Even if you treat GitHub as a single entity capable of coherent desires, it probably doesn't desire to avoid automatic status updates. It needs to avoid them, to survive in a competitive market where everyone else is also avoiding them. People — and corporations — do lots of things they don't want to do, to survive.)
- derefr 5y ago
  No, I don't think they can. An MMO is a very simple system, in that there's only one Service Level Indicator (SLI) that devs, shareholders, and players all agree on. That SLI is "can a player connect to the server, and perform regular gameplay actions, without a ridiculous amount of per-action latency." GitHub, meanwhile, is a million different things to a hundred million different people. Users of e.g. Homebrew, with its big monolithic ports system hosted as a github repo, have a very different SLI for Github than do users of some language-ecosystem package manager that allows you to pull deps directly from Github; than do people who depend on GitHub Actions to CI their builds on push; than do people doing code-review to others' PRs; than do people using Github mostly for its Wiki, or Issues, or downloading Releases, or Github Pages, or even just reading single-page-with-a-README repos, ala the various $FOO-awesome projects. For many of these use-cases, Github isn't degraded right now. For others, it is. If you ask for Github (or any service with this many different use-cases and stakeholders) to measure by the union of all these SLIs, then the service would literally never be not-degraded. In systems of sufficient scale, there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once. Never has been, never will be. And anything less than just going for the union of all those SLIs, is asking Github to exercise human judgement over which kinds of service degradation qualify as part of their own SLOs. Which is exactly what they're doing. Certainly, internal to services like this, there are all sorts of alerting systems constantly going off to tell SREs what things need fixing. But not all of those things immediately, or even quickly, or even ever, translate to SLO violations. There are some outlier users whose use-cases just break the system's semantics, where those use-cases just aren't "in scope" for the SLO. As long as those users are only breaking the system for themselves, the degradation they experience won't ever translate to an SLO breakage.
  - thih9 5y ago
    You seem to be applying different rules to MMOs and Github, and I don't understand why. I'd say that there are many ways of looking at this; there exist complex MMOs; and one could look at Github from the point of view of an average user. E.g., a bit tongue in cheek: > An MMO is a very simple system, in that there's only one Service Level Indicator (SLI) that devs, shareholders, and players all agree on. That SLI is "can a player connect to the server, and perform regular gameplay actions, without a ridiculous amount of per-action latency." Wouldn't you say that in an MMO of sufficient scale there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once? > In systems of sufficient scale, there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once. Never has been, never will be. Couldn't we redefine SLIs as "can the user connect to the server and perform regular user actions without a ridiculous amount of per-action latency"?

qwertox 5y ago

They have a banner "Investigating - We are investigating reports of degraded performance for GitHub Actions. -- Nov 27, 20:43 UTC" which should suffice for a start.

spapas82 5y ago
Yes now it has been properly updated. However it was down for 10-15 minutes before their status page was updated...

drdaeman 5y ago

There is a point. Even two. 1. It clearly indicates that automatic systems are failing to detect the outage. 2. It also indicates that no one is aware about the incident to manually signal the outage (or that there is no manual override). Basically, it makes a difference between "yeah, shit happened, we know (and maybe working on it)" and "hah, they don't even know themselves".

res0nat0r 5y ago
Most of these over arching status pages are manually run, intentionally by design.
the_duke 5y ago
Almost no one has automatic status pages anymore. Partially because these large systems have some kind of ongoing issue at any given time, so it's challenging to provide a meaningful live status that isn't a bit misleading and could cause misdirected panic. Partially because you don't want to give potential attackers (eg ddos) any insight if/how their methods are affecting your systems. Partially because there are SLAs and reputation at risk, and you don't want to admit to any more downtime than you absolutely have to.
- derefr 5y ago
  If you had a really robust system, it'd be fun to just slap a read-only mirror of your internal metrics dashboard onto the public Internet for anyone to browse and slice as they please. It'd be a brag, kinda. Of course, in the real world, I don't think there's a single IT admin who didn't just start nervously sweating and pulling at their collar after reading the above, imagining it as something their CEO said and encouraging them to target it. Nobody can really do this — and that in turn says something important about how those metrics really look in most systems.
  - Jweb_Guru 5y ago
    FWIW, I've worked on systems that have internal SLAs orders of magnitude higher than what they promise to the public. I think it's more just that there's no advantage to doing something like this as long as none of your competitors are. The status quo is that people's systems are really opaque and vastly underpromise what they should be capable of, and in exchange you get to absorb some unplanned downtime due to issues with underlying systems that you have little control over.

random_thoughts 5y ago

They are currently updating it one by one

cube00 5y ago
...and firing off automated tweets per product.

ImprovedSilence 5y ago

lolol. Clearly it's just degraded service. It seems The Management hasn't approved declaring it an outage, that would just ruin uptime metrics!

charles_f 5y ago

It's updated now.

crecker 5y ago

I would not be impressed if they hear about the down for a trending thread on HN.

JohnFen 5y ago
Honestly, HN is the best status indicator on the net right now.
ryantgtg 5y ago
I don’t work for github, so on a personal note I was about to create an issue in a repo, and it wasn’t loading. My go to “is my router messed up?” check is to load HN because it’s so reliable and fast. And lo, the top post was about github being down!
- crecker 5y ago
  Ha. I checked HN for the same reason, I was not able to reach Github via university network and I thought that was the time my university messed up with DNS. HN seems to be going down for the massive amount of requests!

danjac 5y ago

There's lies, damn lies and SAAS status pages.

Flankk 5y ago

Status page is also down and is returning a false positive. If you check the status page status page it shows the status page is down.

chippiewill 5y ago

Status pages are almost never automated these days, they cause more problems than they solve. To be fair, redditstatus.com is quite nice with their sparkline headline metrics. It at least lets you know _something_ is happening even if they haven't yet declared an incident.

hdjjhhvvhga 5y ago

The only status pages that make sense are the ones maintained by a third party, not the owner. Also for technical reasons.

cyberge99 5y ago
And legal reasons.

ranklord 5y ago

It's updated already. Relatively fast... :/