3 ms·

An outage is an outage, not a performance issue. A waitress can't keep telling a customer their food is coming when they've actually lost the order ticket and

• scottlamb 5y ago
  One thing about large systems, even ones that are quite reliable overall, is that if they reported an outage whenever a single customer saw a problem, their dashboards would literally never be green. That'd be true in some sense but not very useful. A teammate once made a "problem" light for Gmail as a joke and coded it to be lit when there were any alerts firing at any severity in the internal monitoring system. I think most people would say that Gmail is pretty reliable, but if the bulb ever went out, you could safely assume there was a problem with the bulb or the monitoring system. That's why big systems have SLOs like (to give a simple example) 99.9% of requests succeeding rather than 100%, or 99.999% of customer data being currently available rather than 100%. (Sometimes they can get more complex like at least x% of customers have no more than y% of time periods in which more than z% of requests fail.) And if they have per-customer SLOs, those are generally much less strict than site-wide SLOs and require a certain traffic level to take effect. In today's case, I think github's problems are widespread enough that whatever SLO they've set is being violated, but the point stands in general.
  • deleted 5y ago
    [deleted]