31 ms·

> Update - Git Operations is experiencing degraded performance. We are continuing to investigate. I hate that companies are saying degraded performance for "it

> Update - Git Operations is experiencing degraded performance. We are continuing to investigate. I hate that companies are saying degraded performance for "its not working at all" (even if only for some people). That's not degraded performance, that's broken. At least the status grid says Partial Outage, but stop with the degraded performance bullshit.

derefr 5y ago
Degraded performance is what it says. Things that took 20s are currently taking 60s. Things that took 60s are now taking 180s. Things are going slower. Maybe by a constant factor. Maybe exponentially. Where this sometimes turns into a "partial outage", is that there might be response timeouts somewhere in the backend of the stack, meaning that things that took 60s are now taking [100s and then a 504 error response]. ("Because," the backend dev thinks, "why would clients want a response after 100s? They've probably given up and gone home already at that point.") Or, equally likely, there's a response deadline somewhere in your client—where you figured "their API will never take more than 100s to respond", so even though the API is still chugging away at the 100s mark, your client cuts it off, drops the connection, and displays an error. Either way, the literal problem at hand is still, simply, "things are going slower." "Degraded performance" can help you predict whether you'll be one of the customers encountering failures, in a way that "partial outage" cannot. Were your particular querying workloads to the API fast before? You'll probably be okay. Were they slow before? Then you might experience the further slowdown as an outage. If you want to design for fault-tolerance of this case, it's important to know how APIs you talk to deal with long-running workloads. Do they have timeouts? Are they customizable? Are they different per plan level? These are things the API's documentation should communicate. Once you know these things—and you have metrics on how long requests to a given upstream API are taking to complete—you'll know exactly whether "degraded performance" will affect you or not, before actually getting customer reports.
- pera 5y ago
  Well I am getting 500s instantly, not after some internal timeout
  - derefr 5y ago
    In really complex systems, there's the ugly case where some middleware needs to fetch something from another service to start up... and then that fetch times out. So the middleware ends up in an invalid state. This is a distributed-systems design faux-pas. Systems that need information to start up, should have some sort of burned in or persistently-cached stale version of that needed information, such that they can start up without being able reach out for it. They should then probe for the newest info until they have it. (See e.g. Elixir's tzdata library.) If you don't do this, and instead put a synchronous remote call in the critical path for a service's startup, then you're turning "degraded performance" into a real outage condition, entirely unnecessarily. It's actually not that common, because you need to get to a certain scale before things like this start happening, and that scale brings with it hiring people who know about scenarios like this. But sadly, it does still happen sometimes.
    - dkersten 5y ago
      > In really complex systems, there's the ugly case where some middleware needs to fetch something from another service to start up... and then that fetch times out. So the middleware ends up in an invalid state. That's no longer a performance issue then though. It may have been triggered by a performance issue, but for downstream services or users, its a plain old outtage. Besides, who cares if its "technically" a performance issue if its unusable for you? Its the end result that's important and if I can't use the service, it make no difference to me what the technicalities are, only that I cannot use it, and my customers certainly don't care if my product stops working because microsoft or amazon or whoever have performance issues, they only care that my product stops working.
      - derefr 5y ago
        I didn't say that this case is not an outage. I was just trying to explain why Github were not reporting an outage as an outage. Let me try again: The reason sites report "performance degradation" in these cases, is that what can usually be detected immediately and automatically by metrics is the underlying performance degradation; while the fact that some process got unexpectedly wedged due to the performance degradation usually goes undetected for quite some time, until customers start reporting the outage. And, before the dashboard can be switched to "suspected outage", it has to first be determined whether any of these customers' reports are of a real "outage", and not just of "my own client is programmed to give up on the request after 100s" problems. Each of these weird edge-case distributed-systems failures is usually fixed — stopped from ever happening again — after the first time it gets triggered. Which means that every time one of these happens, it's the first time the system has failed in that particular way. So there's never any automated monitoring in place to notice that the system has failed in that particular way — as that would imply that the failure was in some way predictable. As such, while yes, the outage is real, I don't blame a company for not reporting the outage as such immediately, or for quite a while after it happens. Sometimes it's not clear right up until the root cause is resolved, that it actually was a partial customer-facing outage. Sometimes all your third-party downtime detector apps just aren't doing the right things to hit the code path / get routed through the middlebox that's failing for some customers; and all the real-customer reports getting through to you are vague and muddled. (One thing that can be immensely helpful to a SaaS's ops team, is to have at least one highly-technical company as a customer who reliably only reports "upstream fault" problems, rather than PEBKAC problems. If that canary customer is complaining, then the outage is likely real. They're better than any downtime detector service — and, surprisingly, often faster, too! Sadly, most SaaS businesses have literally no single such customer.)
- jeremyjh 5y ago
  When I push git commits it fails instantly each and every time. To me, that looks like an outage.
  - 015a 5y ago
    Its an outage for you. You do not have the information to assert that its an outage for everyone.
    - sombremesa 5y ago
      An outage is an outage, not a performance issue. A waitress can't keep telling a customer their food is coming when they've actually lost the order ticket and have no clue what was ordered anymore, just because other customers aren't having the same issue. Well, I guess she can in this case, and other customers will happily come to her defense!
      - scottlamb 5y ago
        One thing about large systems, even ones that are quite reliable overall, is that if they reported an outage whenever a single customer saw a problem, their dashboards would literally never be green. That'd be true in some sense but not very useful. A teammate once made a "problem" light for Gmail as a joke and coded it to be lit when there were any alerts firing at any severity in the internal monitoring system. I think most people would say that Gmail is pretty reliable, but if the bulb ever went out, you could safely assume there was a problem with the bulb or the monitoring system. That's why big systems have SLOs like (to give a simple example) 99.9% of requests succeeding rather than 100%, or 99.999% of customer data being currently available rather than 100%. (Sometimes they can get more complex like at least x% of customers have no more than y% of time periods in which more than z% of requests fail.) And if they have per-customer SLOs, those are generally much less strict than site-wide SLOs and require a certain traffic level to take effect. In today's case, I think github's problems are widespread enough that whatever SLO they've set is being violated, but the point stands in general.
        deleted 5y ago
- dboreham 5y ago
  Not too helpful when their UI timeout is 10s.
- dkersten 5y ago
  > Degraded performance is what it says. Things that took 20s are currently taking 60s. Things that took 60s are now taking 180s. Things are going slower. Maybe by a constant factor. Maybe exponentially. It wasn't a performance issue though. It was literally just plain not working. This is the error I got when trying to push: ! [remote rejected] master -> master (failure) Not after 10s, 30s, 60s -- instantly, every single time.
- scottlamb 5y ago
  > ("Because," the backend dev thinks, "why would clients want a response after 100s? They've probably given up and gone home already at that point.") Or worse: they might have given up on this request and issued another one. When you have a problem that might be caused by (or exacerbated by) overload and might cause heavy retrying (by the user and/or automatically at one or more layers), you want to give up on the requests that are least likely to matter. Deadlines are a standard way to do that. Other ways include propagating cancellation through the stack, sheddable request categories, and queue disciplines like adaptive LIFO. Anyway, I agree with everything you wrote, and I want to point out that whatever deadlines they have in the system are probably helping more than they're hurting. Of course I can't be sure of that without knowing the nature of the outage.
jdavis703 5y ago
I was able to merge a PR after trying a dozen or so times. So it would appear it’s degraded, not totally out.
- tyingq 5y ago
  That still feels like "not usable" to me.
- dkersten 5y ago
  Well, I wasn't able to push to one of my repositories, even though I tried many times. So it was effectively out for me, not "degraded". Partial Outage, like what the status grid said, seems much more accurate. But this goes beyond just this particular case. Remember when half the internet was down because of an AWS issue? Yet they still said "degraded performance" or similar.
howolduis 5y ago
that's what you get when Microsoft buys a platform
- tmpz22 5y ago
  Except everyone does it
  - deleted 5y ago
    [deleted]

derefr 5y ago

Degraded performance is what it says. Things that took 20s are currently taking 60s. Things that took 60s are now taking 180s. Things are going slower. Maybe by a constant factor. Maybe exponentially. Where this sometimes turns into a "partial outage", is that there might be response timeouts somewhere in the backend of the stack, meaning that things that took 60s are now taking [100s and then a 504 error response]. ("Because," the backend dev thinks, "why would clients want a response after 100s? They've probably given up and gone home already at that point.") Or, equally likely, there's a response deadline somewhere in your client—where you figured "their API will never take more than 100s to respond", so even though the API is still chugging away at the 100s mark, your client cuts it off, drops the connection, and displays an error. Either way, the literal problem at hand is still, simply, "things are going slower." "Degraded performance" can help you predict whether you'll be one of the customers encountering failures, in a way that "partial outage" cannot. Were your particular querying workloads to the API fast before? You'll probably be okay. Were they slow before? Then you might experience the further slowdown as an outage. If you want to design for fault-tolerance of this case, it's important to know how APIs you talk to deal with long-running workloads. Do they have timeouts? Are they customizable? Are they different per plan level? These are things the API's documentation should communicate. Once you know these things—and you have metrics on how long requests to a given upstream API are taking to complete—you'll know exactly whether "degraded performance" will affect you or not, before actually getting customer reports.

pera 5y ago
Well I am getting 500s instantly, not after some internal timeout
- derefr 5y ago
  In really complex systems, there's the ugly case where some middleware needs to fetch something from another service to start up... and then that fetch times out. So the middleware ends up in an invalid state. This is a distributed-systems design faux-pas. Systems that need information to start up, should have some sort of burned in or persistently-cached stale version of that needed information, such that they can start up without being able reach out for it. They should then probe for the newest info until they have it. (See e.g. Elixir's tzdata library.) If you don't do this, and instead put a synchronous remote call in the critical path for a service's startup, then you're turning "degraded performance" into a real outage condition, entirely unnecessarily. It's actually not that common, because you need to get to a certain scale before things like this start happening, and that scale brings with it hiring people who know about scenarios like this. But sadly, it does still happen sometimes.
  - dkersten 5y ago
    > In really complex systems, there's the ugly case where some middleware needs to fetch something from another service to start up... and then that fetch times out. So the middleware ends up in an invalid state. That's no longer a performance issue then though. It may have been triggered by a performance issue, but for downstream services or users, its a plain old outtage. Besides, who cares if its "technically" a performance issue if its unusable for you? Its the end result that's important and if I can't use the service, it make no difference to me what the technicalities are, only that I cannot use it, and my customers certainly don't care if my product stops working because microsoft or amazon or whoever have performance issues, they only care that my product stops working.
    - derefr 5y ago
      I didn't say that this case is not an outage. I was just trying to explain why Github were not reporting an outage as an outage. Let me try again: The reason sites report "performance degradation" in these cases, is that what can usually be detected immediately and automatically by metrics is the underlying performance degradation; while the fact that some process got unexpectedly wedged due to the performance degradation usually goes undetected for quite some time, until customers start reporting the outage. And, before the dashboard can be switched to "suspected outage", it has to first be determined whether any of these customers' reports are of a real "outage", and not just of "my own client is programmed to give up on the request after 100s" problems. Each of these weird edge-case distributed-systems failures is usually fixed — stopped from ever happening again — after the first time it gets triggered. Which means that every time one of these happens, it's the first time the system has failed in that particular way. So there's never any automated monitoring in place to notice that the system has failed in that particular way — as that would imply that the failure was in some way predictable. As such, while yes, the outage is real, I don't blame a company for not reporting the outage as such immediately, or for quite a while after it happens. Sometimes it's not clear right up until the root cause is resolved, that it actually was a partial customer-facing outage. Sometimes all your third-party downtime detector apps just aren't doing the right things to hit the code path / get routed through the middlebox that's failing for some customers; and all the real-customer reports getting through to you are vague and muddled. (One thing that can be immensely helpful to a SaaS's ops team, is to have at least one highly-technical company as a customer who reliably only reports "upstream fault" problems, rather than PEBKAC problems. If that canary customer is complaining, then the outage is likely real. They're better than any downtime detector service — and, surprisingly, often faster, too! Sadly, most SaaS businesses have literally no single such customer.)
jeremyjh 5y ago
When I push git commits it fails instantly each and every time. To me, that looks like an outage.
- 015a 5y ago
  Its an outage for you. You do not have the information to assert that its an outage for everyone.
  - sombremesa 5y ago
    An outage is an outage, not a performance issue. A waitress can't keep telling a customer their food is coming when they've actually lost the order ticket and have no clue what was ordered anymore, just because other customers aren't having the same issue. Well, I guess she can in this case, and other customers will happily come to her defense!
    - scottlamb 5y ago
      One thing about large systems, even ones that are quite reliable overall, is that if they reported an outage whenever a single customer saw a problem, their dashboards would literally never be green. That'd be true in some sense but not very useful. A teammate once made a "problem" light for Gmail as a joke and coded it to be lit when there were any alerts firing at any severity in the internal monitoring system. I think most people would say that Gmail is pretty reliable, but if the bulb ever went out, you could safely assume there was a problem with the bulb or the monitoring system. That's why big systems have SLOs like (to give a simple example) 99.9% of requests succeeding rather than 100%, or 99.999% of customer data being currently available rather than 100%. (Sometimes they can get more complex like at least x% of customers have no more than y% of time periods in which more than z% of requests fail.) And if they have per-customer SLOs, those are generally much less strict than site-wide SLOs and require a certain traffic level to take effect. In today's case, I think github's problems are widespread enough that whatever SLO they've set is being violated, but the point stands in general.
      - deleted 5y ago
dboreham 5y ago
Not too helpful when their UI timeout is 10s.
dkersten 5y ago
> Degraded performance is what it says. Things that took 20s are currently taking 60s. Things that took 60s are now taking 180s. Things are going slower. Maybe by a constant factor. Maybe exponentially. It wasn't a performance issue though. It was literally just plain not working. This is the error I got when trying to push: ! [remote rejected] master -> master (failure) Not after 10s, 30s, 60s -- instantly, every single time.
scottlamb 5y ago
> ("Because," the backend dev thinks, "why would clients want a response after 100s? They've probably given up and gone home already at that point.") Or worse: they might have given up on this request and issued another one. When you have a problem that might be caused by (or exacerbated by) overload and might cause heavy retrying (by the user and/or automatically at one or more layers), you want to give up on the requests that are least likely to matter. Deadlines are a standard way to do that. Other ways include propagating cancellation through the stack, sheddable request categories, and queue disciplines like adaptive LIFO. Anyway, I agree with everything you wrote, and I want to point out that whatever deadlines they have in the system are probably helping more than they're hurting. Of course I can't be sure of that without knowing the nature of the outage.

jdavis703 5y ago

I was able to merge a PR after trying a dozen or so times. So it would appear it’s degraded, not totally out.

tyingq 5y ago
That still feels like "not usable" to me.
dkersten 5y ago
Well, I wasn't able to push to one of my repositories, even though I tried many times. So it was effectively out for me, not "degraded". Partial Outage, like what the status grid said, seems much more accurate. But this goes beyond just this particular case. Remember when half the internet was down because of an AWS issue? Yet they still said "degraded performance" or similar.

howolduis 5y ago

that's what you get when Microsoft buys a platform

tmpz22 5y ago
Except everyone does it
- deleted 5y ago
  [deleted]