5 ms·

So down right now...I wonder why they still use https://www.githubstatus.com/ https://www.githubstatus.com/ that reports everything is alright when it's not!

maschultz 3y ago
Maybe the status page is down - it needs a status page to tell us if the status page is down
SV_BubbleTime 3y ago
If it takes someone to manually change it from green to red, that does seem to defeat the purpose.
- evulhotdog 3y ago
  Yep, and when money comes into play when you're supposed to meet SLAs, you certainly don't want it being automatic.
- klysm 3y ago
  Possibly, but sometimes with failures this bad you can't get to the page to update it.
  - munk-a 3y ago
    There was that hilarious multi-hour AWS failure a while back where the status page was updated via one of their internal services... and that service went down as part of the outage.
- jabart 3y ago
  No it doesn't. The amount of false alarm alerts you can get with internet based monitoring is more than 0. You could have a BGP route break things for one ISP your monitoring happens to use. You could have a failover event happening where it takes 30 seconds for everything to converge. I have multiple monitors on my app at 1 minute intervals from different vendors and ALWAYS a user will email us within 5 seconds of an issue. It's not realistic for a company to have automatic status updates trigger things without a person manually reviewing them because too many things can go wrong on the automatic status update to cause panic.
  - lucb1e 3y ago
    Who would panic? If nobody notices it's out because it's not, then nobody is going to be checking the status page. And if they do see the status page showing red while it's up, it's not like they're going to be unhappy about their SLA being met. Maybe you want human confirmation on historic figures, but the live thing might as well be live.
  - wongarsu 3y ago
    Most paid status monitoring services cover BGP route problems and ISP issues by only flagging an event if it's detected from X geographically diverse endpoints. For the 30 seconds where you wait for failover to complete: that is a 30 second outage. It's not necessarily profitable to admit to it, but showing it as a 30 second outage would be accurate
    - jabart 3y ago
      TCP default is more than 30 seconds. The internet itself has about a 99.9% uptime. If one company showed every 30 second blip on their outage page all their competitors would have that screenshot on the first page of their pitch deck even if they also had the same issue. 2-5 minutes is reasonable for a public service to announce an outage.
    - jabart 3y ago
      Forgot about that centurylink BGP infinite loop route bug they had where it took down their whole system nationwide. A lot of monitoring services showed red even though it was one ISP that was done.
- distortionfield 3y ago
  Unknown unknowns means you can have catastrophic system failures that automated alerts don't detect.
- numbsafari 3y ago
  I bet they could teach Co-Pilot to create a PR to make the change, and build some GitHub actions to automatically merge those changes.
- AYBABTME 3y ago
  Not really, things fail in unexpected ways. Automated anomaly detection is notoriously error prone, leading to a lot of false positive and false negatives, in the trivial case of monitoring a single timeseries. For a system the size of GitHub, you need to monitor a whole host of things and if it's quasi impossible to do one timeseries well, there's basically no hope of doing automated many timeseries anomaly detection with a signal-to-noise ratio that's better than "humans looking at the thing and realizing it's not going well". There's stuff like this that can't be automated well. The automated result is far worse than the human-based alternative.
Shekelphile 3y ago
Pretty much every company has been shown to have fake status pages at this point.
- ezekg 3y ago
  Pretty much. They want the burden of proof for SLAs to fall on the customer, not on themselves. If a customer has to prove that an outage specifically affected them, they are much less likely to have a successful case against the failure to meet their SLA. (Not directed at GitHub specifically, but at bogus status pages.)
- AYBABTME 3y ago
  Status pages are updated by humans and the humans need to (1) realize there's a problem and (2) understand the magnitude of the problem and (3) put that on the status page. It's not fake, it's just a human process. And automating this would be error prone just the same.
  - Macuyiko 3y ago
    Very good points. Meanwhile I have clients asking me why they can't have a status page to which I reply: you can, but ultimately to be completely fail proof it will be a human updating it slowly. To which they reply: but GitHub or X does it... Very infuriating, that.
    - AYBABTME 3y ago
      There's some nice tooling these days for this. E.g. https://firehydrant.com/ https://firehydrant.com/ and https://incident.io https://incident.io both make this a faster, more embedded process.
      - sjwhitworth 3y ago
        Hey, incident.io CEO here! Thanks for mentioning us.
      - amandamacleod 3y ago
munk-a 3y ago
https://downdetector.com/status/github/ https://downdetector.com/status/github/ is a far more reliable source - it's just powered by user reports and often will show issues long before the status page ever receives an update.
- jachee 3y ago
  Keep in mind that downdetector can be brigaded and/or show knock-on problems instead of root causes. e.g. A couple weeks ago there were fairly major spikes across a rather huge variety of services on there, but it turned out that it was actually Comcast that was having trouble, rather than any of the “down” services.

maschultz 3y ago

Maybe the status page is down - it needs a status page to tell us if the status page is down

SV_BubbleTime 3y ago

If it takes someone to manually change it from green to red, that does seem to defeat the purpose.

evulhotdog 3y ago
Yep, and when money comes into play when you're supposed to meet SLAs, you certainly don't want it being automatic.
klysm 3y ago
Possibly, but sometimes with failures this bad you can't get to the page to update it.
- munk-a 3y ago
  There was that hilarious multi-hour AWS failure a while back where the status page was updated via one of their internal services... and that service went down as part of the outage.
jabart 3y ago
No it doesn't. The amount of false alarm alerts you can get with internet based monitoring is more than 0. You could have a BGP route break things for one ISP your monitoring happens to use. You could have a failover event happening where it takes 30 seconds for everything to converge. I have multiple monitors on my app at 1 minute intervals from different vendors and ALWAYS a user will email us within 5 seconds of an issue. It's not realistic for a company to have automatic status updates trigger things without a person manually reviewing them because too many things can go wrong on the automatic status update to cause panic.
- lucb1e 3y ago
  Who would panic? If nobody notices it's out because it's not, then nobody is going to be checking the status page. And if they do see the status page showing red while it's up, it's not like they're going to be unhappy about their SLA being met. Maybe you want human confirmation on historic figures, but the live thing might as well be live.
- wongarsu 3y ago
  Most paid status monitoring services cover BGP route problems and ISP issues by only flagging an event if it's detected from X geographically diverse endpoints. For the 30 seconds where you wait for failover to complete: that is a 30 second outage. It's not necessarily profitable to admit to it, but showing it as a 30 second outage would be accurate
  - jabart 3y ago
    TCP default is more than 30 seconds. The internet itself has about a 99.9% uptime. If one company showed every 30 second blip on their outage page all their competitors would have that screenshot on the first page of their pitch deck even if they also had the same issue. 2-5 minutes is reasonable for a public service to announce an outage.
  - jabart 3y ago
    Forgot about that centurylink BGP infinite loop route bug they had where it took down their whole system nationwide. A lot of monitoring services showed red even though it was one ISP that was done.
distortionfield 3y ago
Unknown unknowns means you can have catastrophic system failures that automated alerts don't detect.
numbsafari 3y ago
I bet they could teach Co-Pilot to create a PR to make the change, and build some GitHub actions to automatically merge those changes.
AYBABTME 3y ago
Not really, things fail in unexpected ways. Automated anomaly detection is notoriously error prone, leading to a lot of false positive and false negatives, in the trivial case of monitoring a single timeseries. For a system the size of GitHub, you need to monitor a whole host of things and if it's quasi impossible to do one timeseries well, there's basically no hope of doing automated many timeseries anomaly detection with a signal-to-noise ratio that's better than "humans looking at the thing and realizing it's not going well". There's stuff like this that can't be automated well. The automated result is far worse than the human-based alternative.

Shekelphile 3y ago

Pretty much every company has been shown to have fake status pages at this point.

ezekg 3y ago
Pretty much. They want the burden of proof for SLAs to fall on the customer, not on themselves. If a customer has to prove that an outage specifically affected them, they are much less likely to have a successful case against the failure to meet their SLA. (Not directed at GitHub specifically, but at bogus status pages.)
AYBABTME 3y ago
Status pages are updated by humans and the humans need to (1) realize there's a problem and (2) understand the magnitude of the problem and (3) put that on the status page. It's not fake, it's just a human process. And automating this would be error prone just the same.
- Macuyiko 3y ago
  Very good points. Meanwhile I have clients asking me why they can't have a status page to which I reply: you can, but ultimately to be completely fail proof it will be a human updating it slowly. To which they reply: but GitHub or X does it... Very infuriating, that.
  - AYBABTME 3y ago
    There's some nice tooling these days for this. E.g. https://firehydrant.com/ https://firehydrant.com/ and https://incident.io https://incident.io both make this a faster, more embedded process.
    - sjwhitworth 3y ago
      Hey, incident.io CEO here! Thanks for mentioning us.
    - amandamacleod 3y ago

munk-a 3y ago

https://downdetector.com/status/github/ https://downdetector.com/status/github/ is a far more reliable source - it's just powered by user reports and often will show issues long before the status page ever receives an update.

jachee 3y ago
Keep in mind that downdetector can be brigaded and/or show knock-on problems instead of root causes. e.g. A couple weeks ago there were fairly major spikes across a rather huge variety of services on there, but it turned out that it was actually Comcast that was having trouble, rather than any of the “down” services.