3 ms·

Architecture Patterns: The Circuit-Breaker

94 points by kiyanwang 3y ago

hinkley 3y ago
I’m starting to think circuit breakers are not long for this world. You get your front end reliably serving web pages to thousands of people and the bevy of search engine bots out there, and you start wondering why only the front end has to be robust like this. Why don’t your services use the same set of tools to approach a similar level of reliability? In particular, if you have seven services or apps all calling the same service, a pretty common arrangement is one circuit per client-service pair, so you have five different clusters trying to track and react to the health of a sixth, when a load balancer in front of that service could be handling it instead. They aren’t robustness in depth. They’re a coping mechanism for not having it.
- pluto_modadic 3y ago
  circuit breakers /can/ make sense, if done rationally. kinda like that line in fallbacks vs failovers where "if the second system is so reliable why bother with the first?" https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/ https://aws.amazon.com/builders-library/avoiding-fallback-in... (fallback: e.g., using direct DB instead of cache, failover: second cache/API)
  - hinkley 3y ago
    > if the second system is so reliable why bother with the first. Hmm. That's a difficult one to unpack. Sometimes services need to be reliable in different dimensions. And that first service just might not be able to scale high enough to do some more esoteric operation on the data. Though I could see someone arguing that from the user's perspective the first and second system are trying to pretend to be the same thing. I think the services I have the hardest time shooting are the ones that exist to prevent priority inversion. I maintain a batch processing system I have a hard time justifying in some ways. I get more value out of it as a guineapig for new tools and techniques we may want to apply more broadly, than I get from the actual task it performs, but I certainly don't want the core system trying to handle it either. It'd be a lot more noise. So I want to kill it, and I want to keep it. Some tasks have escalating priority over time, and trying to mix them with tasks that don't can be problematic.
- sgarland 3y ago
  I’ve also seen people reinventing the wheel (poorly), seemingly to avoid existing and proven technology, like writing a bespoke DB load balancer / health checker in Node using Redis and Dynamo, instead of just standing up HAProxy. “My JavaScript will surely be more performant and less buggy than production-grade C that’s existed for decades.”
  - baq 3y ago
    I guess it’s more ‘I don’t need another layer in my stack which I don’t know how to configure; I just need this one feature, how hard can it be’.
    - hinkley 3y ago
      Nginx is not a joy to configure but it’s probably easier than hystrix.
  - hinkley 3y ago
    I have my fingers in some of our reloadable configuration code and one of the places I see it used a lot is basically a bespoke service discovery system. It's so stupid, and it adds a lot of overhead. There are at least three competing solutions (architectures, not vendors) for this problem and two of them were old hat before this code was ever written. Stop it.
rbanffy 3y ago
I came to think of circuit breakers as a crude tool that has a very limited set of uses, much smaller than what it's used for. It might work as a last resort, but it should never be deployed as your first way to deal with failures. If at all possible, APIs should communicate server health status back to their clients, and clients should have ways to deal with partial outages of supporting services and systems. Also, that sequence diagram should have been a state diagram. Don't people teach proper UML in schools these days?
- bsaul 3y ago
  I had the same problem understanding the state diagram but didn’t even realize he used the wrong type of UML diagram :))
- brodouevencode 3y ago
  UML is taught, sometimes, but once you get into the industry it is a coin flip as to whether or not it is properly used - if it is used at all.
- jerf 3y ago
  I would think the first resort in most of the non-flooding-related cases[1] is backpressure. If you've got too many requests coming in, be sure that internally you start parking the excess incoming requests in some low-resource-usage configuration (e.g., simply noting the incoming TCP connection but then not even read()ing from it until you are ready), and let the system push back naturally by the way its latency increases. This obviously naturally pairs with both client-side and server-side timeouts, giving both a vote in how long they're willing to wait. This is generally what you want. Actively breaking the connections if I think I'm overloaded is only something I'd add if there is some additional constraint on the system that suggests it's a good idea due to some specific local concerns. Circuit breaking produces a much sharper (for lack of a better term) performance graph, where instead of degrading gracefully like backpressure will, you suddenly start getting active failures. I've built a lot of backpressure in my systems; I'm yet to need a circuit breaker. YMMV, of course, but I see some people referring to it as a basic tool and I would disagree; the basic tool is backpressure. [1]: Flooding is its own thing. In general, to weather DOS attacks but still have the service functional requires another system or likely set of systems in front of the service that can handle the DOS issues. I say this just to point out that circuit breakers aren't really for this case, because no system can on its own handle a full-on DOS flood, no matter what code you write, when the DOS flood may literally be coming in at a speed where it is faster than you can even reject connections, let alone do any work.
jdwyah 3y ago
When people think of circuit breakers, I think most people think of the automated things like hystrix / istio circuit breaking. I feel like those have their place, but in practice are often overkill and they are complex enough that it's hard to keep them loaded in my brain when I rarely think about them. What I find to be more broadly useful is the ability to change things like timeout and retry on the fly. Or other more manual kill switches via dynamic configuration. When something is on fire and my service is getting overloaded by another service that is crushing it with retries, I can then make a surgical change. Not automatic, but still a good MTTR. I think this is one of the better ways to use what we're building at Prefab.
- brodouevencode 3y ago
  Agreed. Configurability (which is a proxy of circuit-breaking), especially at the border, is often overlooked when we build systems. If it's not configurable it probably (but not always) have some "smarts" to deal with problems.

94 points by kiyanwang 3y ago

I’m starting to think circuit breakers are not long for this world. You get your front end reliably serving web pages to thousands of people and the bevy of search engine bots out there, and you start wondering why only the front end has to be robust like this. Why don’t your services use the same set of tools to approach a similar level of reliability? In particular, if you have seven services or apps all calling the same service, a pretty common arrangement is one circuit per client-service pair, so you have five different clusters trying to track and react to the health of a sixth, when a load balancer in front of that service could be handling it instead. They aren’t robustness in depth. They’re a coping mechanism for not having it.

pluto_modadic 3y ago
circuit breakers /can/ make sense, if done rationally. kinda like that line in fallbacks vs failovers where "if the second system is so reliable why bother with the first?" https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/ https://aws.amazon.com/builders-library/avoiding-fallback-in... (fallback: e.g., using direct DB instead of cache, failover: second cache/API)
- hinkley 3y ago
  > if the second system is so reliable why bother with the first. Hmm. That's a difficult one to unpack. Sometimes services need to be reliable in different dimensions. And that first service just might not be able to scale high enough to do some more esoteric operation on the data. Though I could see someone arguing that from the user's perspective the first and second system are trying to pretend to be the same thing. I think the services I have the hardest time shooting are the ones that exist to prevent priority inversion. I maintain a batch processing system I have a hard time justifying in some ways. I get more value out of it as a guineapig for new tools and techniques we may want to apply more broadly, than I get from the actual task it performs, but I certainly don't want the core system trying to handle it either. It'd be a lot more noise. So I want to kill it, and I want to keep it. Some tasks have escalating priority over time, and trying to mix them with tasks that don't can be problematic.
sgarland 3y ago
I’ve also seen people reinventing the wheel (poorly), seemingly to avoid existing and proven technology, like writing a bespoke DB load balancer / health checker in Node using Redis and Dynamo, instead of just standing up HAProxy. “My JavaScript will surely be more performant and less buggy than production-grade C that’s existed for decades.”
- baq 3y ago
  I guess it’s more ‘I don’t need another layer in my stack which I don’t know how to configure; I just need this one feature, how hard can it be’.
  - hinkley 3y ago
    Nginx is not a joy to configure but it’s probably easier than hystrix.
- hinkley 3y ago
  I have my fingers in some of our reloadable configuration code and one of the places I see it used a lot is basically a bespoke service discovery system. It's so stupid, and it adds a lot of overhead. There are at least three competing solutions (architectures, not vendors) for this problem and two of them were old hat before this code was ever written. Stop it.

rbanffy 3y ago

I came to think of circuit breakers as a crude tool that has a very limited set of uses, much smaller than what it's used for. It might work as a last resort, but it should never be deployed as your first way to deal with failures. If at all possible, APIs should communicate server health status back to their clients, and clients should have ways to deal with partial outages of supporting services and systems. Also, that sequence diagram should have been a state diagram. Don't people teach proper UML in schools these days?

bsaul 3y ago
I had the same problem understanding the state diagram but didn’t even realize he used the wrong type of UML diagram :))
brodouevencode 3y ago
UML is taught, sometimes, but once you get into the industry it is a coin flip as to whether or not it is properly used - if it is used at all.
jerf 3y ago
I would think the first resort in most of the non-flooding-related cases[1] is backpressure. If you've got too many requests coming in, be sure that internally you start parking the excess incoming requests in some low-resource-usage configuration (e.g., simply noting the incoming TCP connection but then not even read()ing from it until you are ready), and let the system push back naturally by the way its latency increases. This obviously naturally pairs with both client-side and server-side timeouts, giving both a vote in how long they're willing to wait. This is generally what you want. Actively breaking the connections if I think I'm overloaded is only something I'd add if there is some additional constraint on the system that suggests it's a good idea due to some specific local concerns. Circuit breaking produces a much sharper (for lack of a better term) performance graph, where instead of degrading gracefully like backpressure will, you suddenly start getting active failures. I've built a lot of backpressure in my systems; I'm yet to need a circuit breaker. YMMV, of course, but I see some people referring to it as a basic tool and I would disagree; the basic tool is backpressure. [1]: Flooding is its own thing. In general, to weather DOS attacks but still have the service functional requires another system or likely set of systems in front of the service that can handle the DOS issues. I say this just to point out that circuit breakers aren't really for this case, because no system can on its own handle a full-on DOS flood, no matter what code you write, when the DOS flood may literally be coming in at a speed where it is faster than you can even reject connections, let alone do any work.

jdwyah 3y ago

When people think of circuit breakers, I think most people think of the automated things like hystrix / istio circuit breaking. I feel like those have their place, but in practice are often overkill and they are complex enough that it's hard to keep them loaded in my brain when I rarely think about them. What I find to be more broadly useful is the ability to change things like timeout and retry on the fly. Or other more manual kill switches via dynamic configuration. When something is on fire and my service is getting overloaded by another service that is crushing it with retries, I can then make a surgical change. Not automatic, but still a good MTTR. I think this is one of the better ways to use what we're building at Prefab.

brodouevencode 3y ago
Agreed. Configurability (which is a proxy of circuit-breaking), especially at the border, is often overlooked when we build systems. If it's not configurable it probably (but not always) have some "smarts" to deal with problems.