4 ms·

Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com

by drcongo 5mo ago

jaapz 5mo ago
All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.
- echelon 5mo ago
  In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262 https://news.ycombinator.com/item?id=48279262
  - TheDong 5mo ago
    Do you know of a single service at a single company that actually does that? I know all of Gmail, every GCE service I can think of, every AWS service I can think of, Amazon.com, Netflix, and Github all do not page on just a single 500. I know none of those are particularly "high performance" though. Curious where your experience is coming from.
    - CBLT 5mo ago
      I've been oncall for a different G service that nearly paged on every error. It used the standard error budget tooling, but on hundreds of user buckets because the engineering around locality-specific configuration was... suspect. Many of these buckets had single-digits user. If a user was on their phone and lost signal, I was paged. Very poor oncall experience.
    - echelon 5mo ago
      I worked at a large fintech moving billions of dollars in volume a day. I had a fairly long tenure, where I maintained multiple key services in critical online payments flow. Authentication, authorization, core business and risk data, as well as some cross-cutting control plane stuff, etc. You needed one or more of our services to take a payment, serve any request from the employee dashboard - pretty much everything hit our services. The entire company ground to a halt without my team. We paged for every single 500. In instances where a particular class of 500 was spurious or not worth fixing, we would leave it acked or mark it as noise. But typically we'd just put in a fix as soon as possible so we didn't page. Our graceful shutdown and traffic shaping stack was great, but occasionally we'd get a few pages during deploys or failovers. Oncall was typically not bad, but when it did get bad it was terrible. I've been involved in huge outages that cost hundreds of millions of dollars. Usually it was the fault of multiple teams having compounding runaway failures rather than one service or bug in particular. It's inexcusable to have a customer's payments not go through. We engineered around resilience. We had strict five nines SLAs and p99 targets and evaluated our adherence with even the smallest partial outage. Hundreds of other services depended on ours, and downstream impacts were huge, so we had to keep a tight ship. We didn't have "business hours"-only paging either as our platform was available globally, including a heavy install base in Asia.
      - sunrunner 5mo ago
        > We paged for every single 500. Assuming the existence of some kind of network (with zero guarantee of 100% reliability), how does this work in practice? Is each 500 treated as an event that needs investigation, even if the result of that would end up as 'a router dropped something from an internal buffer but the transaction as a whole was re-tried by a parent so the service itself recovered'?
        LPisGood 5mo ago
        A reliability engineer from Jane Street gave a great talk about this, five nine’s of correctness in reporting, etc isn’t enough for the SEC. https://youtu.be/zR9PpXWsKFQ https://youtu.be/zR9PpXWsKFQ
        5mo ago
    - theta_d 5mo ago
      The sub-service at IBM cloud I worked on had an insanely small error budget such that pages were nearly constant. On call was hell week until a few of us insisted on fixing the issues. The "few" of us were contractors. The employees seemed more than willing to just let the pages continue.
      - alexfoo 5mo ago
        Some companies pay more if people are paged. It can create a perverse incentive not to fix problems or, in extreme cases, to watch things going wrong, waiting for the page, and then being ready to fix it straight away.
  - Doohickey-d 5mo ago
    Im curious about this: because in my experience (working on smaller services though), a small number of errors is always there, as a "baseline". Recently there was this: https://news.ycombinator.com/item?id=47252971 https://news.ycombinator.com/item?id=47252971 "10% of Firefox crashes are caused by bitflips" Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on a network, there's potential for a lot more random errors.
    - KPGv2 5mo ago
      Bitflips are something that can happen in consumer-grade RAM, so that tracks (and it's comforting that wayward cosmic rays are a substantial reason for an application's crashes!), but on enterprise servers, they will run ECC RAM that is very resistant to bit flips. This is why data hoarders who have NASes with lots of space insist on running their servers with ECC RAM despite it being significantly more expensive. Because bit flips, for all intents and purposes, cannot happen. The RAM itself detects and corrects for them. I wouldn't expect bit flips to be a significant contributor to enterprise problems.
      - maccard 5mo ago
        You've completely missed the point - It's not about bitflips it's about errors that are outside the scope of what's fixable.
        KPGv2 5mo ago
        I suppose I misunderstood what the "random error" was supposed to mean. I wouldn't call a network error a "random error" because it's caused by things that are internal to the system (entities using a network). A bit flip is caused by an external factor: cosmic radiation. To me, that's what a "random" error is. If your network goes down because of a DDOS, or part of your system overheating, that's an internal issue you had control over. If a bit flips because of cosmic radiation, you can't really do anything about that, and it's utterly unpredictable. That's "random" to me.
      - 5mo ago
  - awithrow 5mo ago
    that is absolutely not the case for any system of size and scale. that would just burn out the on-call team and not result in improvements. Error rates/budgets are used instead.
    - hnlmorg 5mo ago
      It depends what you're monitoring. If it's response codes from user generated queries, then I'd agree with you. But if it is synthetic queries sent from the monitoring platform, then you control the user agent, payload, and endpoints. So any failed requests are a symptom of a misconfiguration and/or failure that should be investigated. Albeit not necessarily as a P1 priority.
  - jordemort 5mo ago
    forget it, Jake; it’s Azure
  - swiftcoder 5mo ago
    Yeah, no, nobody runs cloud services like that. At AWS most alarms required failures in 3 consecutive 5 minute periods. Critical things could be on 3 consecutive 1 minute windows - but that alarm starts a 15 minute escalation for the oncall engineer to check in, and they have to validate the issue isn't a false alarm before updating the status page would even be considered
  - compumike 5mo ago
    Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes: If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am! If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday. Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.
    - wasmitnetzen 5mo ago
      Shouldn't Github be large enough to not have anyone on-call, but just rotate the responsible team around the world?
      - bobthepanda 5mo ago
        At least when I worked at a Bigcorp a lot of that was being cut to save costs.
        lokar 5mo ago
        I've worked in large orgs where we could (at at some times did) have around the world rotations. They don't work well. It've very hard to maintain real team cohesion, and you end up with really superficial operations. People tend not to dig in really deep, find good fixes, etc. Lots of superficial bandages.
      - alexfoo 5mo ago
        One team can't troubleshoot AND FIX every possible subsystem, so you just end up with lots (growing to hundreds) of people "on-call" anyway. As others have said, follow-the-sun type models do exist, usually staffed by people in their normal working hours (EMEA, Americas, APAC) but this means you've still got to cover the weekend and public holidays (which there are a lot of when you factor in plenty of different countries). Where you need a quick response you can have a core ops/noc team that looks at things with lower thresholds and shorter windows, and their job is to do the initial triage and then page the appropriate team earlier than they would have been alerted by their own alert thresholds/monitoring. Actually clicking the button to change the status on a public status page is a whole different topic that becomes very political in certain companies.
  - hvb2 5mo ago
    > A noisy pager forces the team to fix the 500s. I'm sure you're not in ops. Or in a dev org of a service with decent request rates. What you're asking for is a service to fail silently. There's no way a service with a decent request rate to have 0 500s. Not when it still sees development. A 50 year old bank API? Maybe...
  - rhyperior 5mo ago
    You only do this when you’re trying to use incident management as a hammer to make a point to somebody whom you have otherwise failed to convince to fix something through persuasive argument. Ie, it’s punitive.
- hnlmorg 5mo ago
  You'd expect them to be monitoring more than just the HTTP response codes from user requests for precisely this reason. If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. But effective monitoring is harder than people assume.
  - dncornholio 5mo ago
    > If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.
    - hnlmorg 5mo ago
      No, monitoring for HTTP response code is a subset of observability and not one that generally gives you the best insights into which subsystems are misbehaving nor why. There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the observability platform (think like running a cron-job) and thus you're not tied to user activity to see when problems arise. There are other metrics outside of HTTP response codes too. Think like free RAM, CPU usage, disk space, etc. This is just naming some obvious ones because these types of metrics are generally bespoke to the type of application your monitoring. And with these types of monitors, you'd not just have an alert when things have failed, but ideally have alerts when an irregular trend is showing that things are likely to fail too. This latter type of monitors helps you get ahead of the problem before it become customer facing. Then you have more traditional stuff like logs. This will also be bespoke to the application. But you'd expect errors in logs to get surfaced quickly. Assuming Github have good hygiene in what's being logged. Tie that up with APMs, RUM, and other goodies like that and you'll have diagnostics to investigate issues when they appear. (this is just a super high level view of observability too)
      - lokar 5mo ago
- logifail 5mo ago
  > All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage. Is it true that official service status pages are updated automatically?
  - baby_souffle 5mo ago
    > it true that official service status pages are updated automatically? Depends. Typically no because there’s an art to crafting the actual message around impact… but sometimes yes it is automated
    - logifail 5mo ago
      > Typically no because there’s an art to crafting the actual message around impact I was thinking more of needing to notify/get sign-off from management...
      - baby_souffle 5mo ago
        > I was thinking more of needing to notify/get sign-off from management... Yeah, that's usually part of it. Precise language matters a TON when you might have some expensive breach-of-SLA terms. Sometimes the people first responding don't even have the full picture yet and can't fully articulate the impact so they leave it vague.
- registeredcorn 5mo ago
  I'm not arguing with what you're saying, but it does make me wonder: What exactly is the point of the status page, if "it is normal for users to already see errors before GitHub officially counts it as an outage"? Is it more so to have something to link to for managers who aren't using the service have a pretty bar to look at and feel like they are "doing something"? Or is it more of a kind of a way to prevent confirming what you already suspect to be true. E.g. "Huh. Me and Jim are seeing problems. How about you Tom? Oh wait, crud. The service page is confirming it's down now. Never mind! Who wants coffee?!"
  - filleduchaos 5mo ago
    There is oddly enough a middle ground between "zero errors whatsoever" and "outage".
    - deleted 5mo ago
      [deleted]
simonjgreen 5mo ago
More likely that 'update the Status site' lives a long way down their incident response plan, and they have alarms going off well before that
- jordemort 5mo ago
  yeah I mean a company the size of GitHub certainly can’t be expected to have enough staff to walk and chew gum at the same time
  - swiftcoder 5mo ago
    If it's like other BigTechs I have worked at, you need director-level signoff and comms team approval to post an outage notice
- PunchyHamster 5mo ago
  it should be automatic tho. Probably isn't so they can at least get the one nine on availability
  - simonjgreen 5mo ago
    Marketing definitely takes interest in status sites
re-thc 5mo ago
> It's weird how I always find out that GitHub is down before GitHub does No, it's not. Official updates = potential SLA penalties. Always requires approval.
- drcongo 5mo ago
  This is the most plausible reply.
deleted 5mo ago
[deleted]
chrisjj 5mo ago
> githubstatus.com There's a threshold. It shows only once 1000 users complain. /i
deleted 5mo ago
[deleted]

Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com

by drcongo 5mo ago

Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com

jaapz 5mo ago

All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.

echelon 5mo ago
In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262 https://news.ycombinator.com/item?id=48279262
- TheDong 5mo ago
  Do you know of a single service at a single company that actually does that? I know all of Gmail, every GCE service I can think of, every AWS service I can think of, Amazon.com, Netflix, and Github all do not page on just a single 500. I know none of those are particularly "high performance" though. Curious where your experience is coming from.
  - CBLT 5mo ago
    I've been oncall for a different G service that nearly paged on every error. It used the standard error budget tooling, but on hundreds of user buckets because the engineering around locality-specific configuration was... suspect. Many of these buckets had single-digits user. If a user was on their phone and lost signal, I was paged. Very poor oncall experience.
  - echelon 5mo ago
    I worked at a large fintech moving billions of dollars in volume a day. I had a fairly long tenure, where I maintained multiple key services in critical online payments flow. Authentication, authorization, core business and risk data, as well as some cross-cutting control plane stuff, etc. You needed one or more of our services to take a payment, serve any request from the employee dashboard - pretty much everything hit our services. The entire company ground to a halt without my team. We paged for every single 500. In instances where a particular class of 500 was spurious or not worth fixing, we would leave it acked or mark it as noise. But typically we'd just put in a fix as soon as possible so we didn't page. Our graceful shutdown and traffic shaping stack was great, but occasionally we'd get a few pages during deploys or failovers. Oncall was typically not bad, but when it did get bad it was terrible. I've been involved in huge outages that cost hundreds of millions of dollars. Usually it was the fault of multiple teams having compounding runaway failures rather than one service or bug in particular. It's inexcusable to have a customer's payments not go through. We engineered around resilience. We had strict five nines SLAs and p99 targets and evaluated our adherence with even the smallest partial outage. Hundreds of other services depended on ours, and downstream impacts were huge, so we had to keep a tight ship. We didn't have "business hours"-only paging either as our platform was available globally, including a heavy install base in Asia.
    - sunrunner 5mo ago
      > We paged for every single 500. Assuming the existence of some kind of network (with zero guarantee of 100% reliability), how does this work in practice? Is each 500 treated as an event that needs investigation, even if the result of that would end up as 'a router dropped something from an internal buffer but the transaction as a whole was re-tried by a parent so the service itself recovered'?
      - LPisGood 5mo ago
        A reliability engineer from Jane Street gave a great talk about this, five nine’s of correctness in reporting, etc isn’t enough for the SEC. https://youtu.be/zR9PpXWsKFQ https://youtu.be/zR9PpXWsKFQ
      - 5mo ago
  - theta_d 5mo ago
    The sub-service at IBM cloud I worked on had an insanely small error budget such that pages were nearly constant. On call was hell week until a few of us insisted on fixing the issues. The "few" of us were contractors. The employees seemed more than willing to just let the pages continue.
    - alexfoo 5mo ago
      Some companies pay more if people are paged. It can create a perverse incentive not to fix problems or, in extreme cases, to watch things going wrong, waiting for the page, and then being ready to fix it straight away.
- Doohickey-d 5mo ago
  Im curious about this: because in my experience (working on smaller services though), a small number of errors is always there, as a "baseline". Recently there was this: https://news.ycombinator.com/item?id=47252971 https://news.ycombinator.com/item?id=47252971 "10% of Firefox crashes are caused by bitflips" Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on a network, there's potential for a lot more random errors.
  - KPGv2 5mo ago
    Bitflips are something that can happen in consumer-grade RAM, so that tracks (and it's comforting that wayward cosmic rays are a substantial reason for an application's crashes!), but on enterprise servers, they will run ECC RAM that is very resistant to bit flips. This is why data hoarders who have NASes with lots of space insist on running their servers with ECC RAM despite it being significantly more expensive. Because bit flips, for all intents and purposes, cannot happen. The RAM itself detects and corrects for them. I wouldn't expect bit flips to be a significant contributor to enterprise problems.
    - maccard 5mo ago
      You've completely missed the point - It's not about bitflips it's about errors that are outside the scope of what's fixable.
      - KPGv2 5mo ago
        I suppose I misunderstood what the "random error" was supposed to mean. I wouldn't call a network error a "random error" because it's caused by things that are internal to the system (entities using a network). A bit flip is caused by an external factor: cosmic radiation. To me, that's what a "random" error is. If your network goes down because of a DDOS, or part of your system overheating, that's an internal issue you had control over. If a bit flips because of cosmic radiation, you can't really do anything about that, and it's utterly unpredictable. That's "random" to me.
    - 5mo ago
- awithrow 5mo ago
  that is absolutely not the case for any system of size and scale. that would just burn out the on-call team and not result in improvements. Error rates/budgets are used instead.
  - hnlmorg 5mo ago
    It depends what you're monitoring. If it's response codes from user generated queries, then I'd agree with you. But if it is synthetic queries sent from the monitoring platform, then you control the user agent, payload, and endpoints. So any failed requests are a symptom of a misconfiguration and/or failure that should be investigated. Albeit not necessarily as a P1 priority.
- jordemort 5mo ago
  forget it, Jake; it’s Azure
- swiftcoder 5mo ago
  Yeah, no, nobody runs cloud services like that. At AWS most alarms required failures in 3 consecutive 5 minute periods. Critical things could be on 3 consecutive 1 minute windows - but that alarm starts a 15 minute escalation for the oncall engineer to check in, and they have to validate the issue isn't a false alarm before updating the status page would even be considered
- compumike 5mo ago
  Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes: If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am! If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday. Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.
  - wasmitnetzen 5mo ago
    Shouldn't Github be large enough to not have anyone on-call, but just rotate the responsible team around the world?
    - bobthepanda 5mo ago
      At least when I worked at a Bigcorp a lot of that was being cut to save costs.
      - lokar 5mo ago
        I've worked in large orgs where we could (at at some times did) have around the world rotations. They don't work well. It've very hard to maintain real team cohesion, and you end up with really superficial operations. People tend not to dig in really deep, find good fixes, etc. Lots of superficial bandages.
    - alexfoo 5mo ago
      One team can't troubleshoot AND FIX every possible subsystem, so you just end up with lots (growing to hundreds) of people "on-call" anyway. As others have said, follow-the-sun type models do exist, usually staffed by people in their normal working hours (EMEA, Americas, APAC) but this means you've still got to cover the weekend and public holidays (which there are a lot of when you factor in plenty of different countries). Where you need a quick response you can have a core ops/noc team that looks at things with lower thresholds and shorter windows, and their job is to do the initial triage and then page the appropriate team earlier than they would have been alerted by their own alert thresholds/monitoring. Actually clicking the button to change the status on a public status page is a whole different topic that becomes very political in certain companies.
- hvb2 5mo ago
  > A noisy pager forces the team to fix the 500s. I'm sure you're not in ops. Or in a dev org of a service with decent request rates. What you're asking for is a service to fail silently. There's no way a service with a decent request rate to have 0 500s. Not when it still sees development. A 50 year old bank API? Maybe...
- rhyperior 5mo ago
  You only do this when you’re trying to use incident management as a hammer to make a point to somebody whom you have otherwise failed to convince to fix something through persuasive argument. Ie, it’s punitive.
hnlmorg 5mo ago
You'd expect them to be monitoring more than just the HTTP response codes from user requests for precisely this reason. If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. But effective monitoring is harder than people assume.
- dncornholio 5mo ago
  > If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.
  - hnlmorg 5mo ago
    No, monitoring for HTTP response code is a subset of observability and not one that generally gives you the best insights into which subsystems are misbehaving nor why. There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the observability platform (think like running a cron-job) and thus you're not tied to user activity to see when problems arise. There are other metrics outside of HTTP response codes too. Think like free RAM, CPU usage, disk space, etc. This is just naming some obvious ones because these types of metrics are generally bespoke to the type of application your monitoring. And with these types of monitors, you'd not just have an alert when things have failed, but ideally have alerts when an irregular trend is showing that things are likely to fail too. This latter type of monitors helps you get ahead of the problem before it become customer facing. Then you have more traditional stuff like logs. This will also be bespoke to the application. But you'd expect errors in logs to get surfaced quickly. Assuming Github have good hygiene in what's being logged. Tie that up with APMs, RUM, and other goodies like that and you'll have diagnostics to investigate issues when they appear. (this is just a super high level view of observability too)
    - lokar 5mo ago
logifail 5mo ago
> All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage. Is it true that official service status pages are updated automatically?
- baby_souffle 5mo ago
  > it true that official service status pages are updated automatically? Depends. Typically no because there’s an art to crafting the actual message around impact… but sometimes yes it is automated
  - logifail 5mo ago
    > Typically no because there’s an art to crafting the actual message around impact I was thinking more of needing to notify/get sign-off from management...
    - baby_souffle 5mo ago
      > I was thinking more of needing to notify/get sign-off from management... Yeah, that's usually part of it. Precise language matters a TON when you might have some expensive breach-of-SLA terms. Sometimes the people first responding don't even have the full picture yet and can't fully articulate the impact so they leave it vague.
registeredcorn 5mo ago
I'm not arguing with what you're saying, but it does make me wonder: What exactly is the point of the status page, if "it is normal for users to already see errors before GitHub officially counts it as an outage"? Is it more so to have something to link to for managers who aren't using the service have a pretty bar to look at and feel like they are "doing something"? Or is it more of a kind of a way to prevent confirming what you already suspect to be true. E.g. "Huh. Me and Jim are seeing problems. How about you Tom? Oh wait, crud. The service page is confirming it's down now. Never mind! Who wants coffee?!"
- filleduchaos 5mo ago
  There is oddly enough a middle ground between "zero errors whatsoever" and "outage".
  - deleted 5mo ago
    [deleted]

simonjgreen 5mo ago

More likely that 'update the Status site' lives a long way down their incident response plan, and they have alarms going off well before that

jordemort 5mo ago
yeah I mean a company the size of GitHub certainly can’t be expected to have enough staff to walk and chew gum at the same time
- swiftcoder 5mo ago
  If it's like other BigTechs I have worked at, you need director-level signoff and comms team approval to post an outage notice
PunchyHamster 5mo ago
it should be automatic tho. Probably isn't so they can at least get the one nine on availability
- simonjgreen 5mo ago
  Marketing definitely takes interest in status sites

re-thc 5mo ago

> It's weird how I always find out that GitHub is down before GitHub does No, it's not. Official updates = potential SLA penalties. Always requires approval.

drcongo 5mo ago
This is the most plausible reply.

deleted 5mo ago

[deleted]

chrisjj 5mo ago

> githubstatus.com There's a threshold. It shows only once 1000 users complain. /i

deleted 5mo ago

[deleted]