4 ms·
I’m starting to think circuit breakers are not long for this world. You get your front end reliably serving web pages to thousands of people and the bevy of se
by hinkley 3y ago
I’m starting to think circuit breakers are not long for this world.
You get your front end reliably serving web pages to thousands of people and the bevy of search engine bots out there, and you start wondering why only the front end has to be robust like this. Why don’t your services use the same set of tools to approach a similar level of reliability?
In particular, if you have seven services or apps all calling the same service, a pretty common arrangement is one circuit per client-service pair, so you have five different clusters trying to track and react to the health of a sixth, when a load balancer in front of that service could be handling it instead.
They aren’t robustness in depth. They’re a coping mechanism for not having it.
- pluto_modadic 3y agocircuit breakers /can/ make sense, if done rationally. kinda like that line in fallbacks vs failovers where "if the second system is so reliable why bother with the first?" https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/ https://aws.amazon.com/builders-library/avoiding-fallback-in... (fallback: e.g., using direct DB instead of cache, failover: second cache/API)
- hinkley 3y ago> if the second system is so reliable why bother with the first. Hmm. That's a difficult one to unpack. Sometimes services need to be reliable in different dimensions. And that first service just might not be able to scale high enough to do some more esoteric operation on the data. Though I could see someone arguing that from the user's perspective the first and second system are trying to pretend to be the same thing. I think the services I have the hardest time shooting are the ones that exist to prevent priority inversion. I maintain a batch processing system I have a hard time justifying in some ways. I get more value out of it as a guineapig for new tools and techniques we may want to apply more broadly, than I get from the actual task it performs, but I certainly don't want the core system trying to handle it either. It'd be a lot more noise. So I want to kill it, and I want to keep it. Some tasks have escalating priority over time, and trying to mix them with tasks that don't can be problematic.
- sgarland 3y agoI’ve also seen people reinventing the wheel (poorly), seemingly to avoid existing and proven technology, like writing a bespoke DB load balancer / health checker in Node using Redis and Dynamo, instead of just standing up HAProxy. “My JavaScript will surely be more performant and less buggy than production-grade C that’s existed for decades.”
- baq 3y agoI guess it’s more ‘I don’t need another layer in my stack which I don’t know how to configure; I just need this one feature, how hard can it be’.
- hinkley 3y agoNginx is not a joy to configure but it’s probably easier than hystrix.
- hinkley 3y agoI have my fingers in some of our reloadable configuration code and one of the places I see it used a lot is basically a bespoke service discovery system. It's so stupid, and it adds a lot of overhead. There are at least three competing solutions (architectures, not vendors) for this problem and two of them were old hat before this code was ever written. Stop it.