5 ms·

I very much want to hear the words "failure isolation" during a systems design interview. Usually as the answer to "Why did you break that functionality out int

I very much want to hear the words "failure isolation" during a systems design interview. Usually as the answer to "Why did you break that functionality out into a separate service?". The answer should involve "independent scaling" and "failure isolation".

mfukar 10y ago
Great, now anyone knows how to temporarily satisfy you while delivering nothing of what you actually want in the long run.
- phamilton 10y ago
  Surely, you understand I'm not literally only looking for buzzwords. I would certainly ask deeper questions on what they are trying to accomplish and why.
  - mfukar 10y ago
    I'd heard the bar for humour had been lowered, but I didn't know how low.
zaptheimpaler 10y ago
I really don't buy this, its mostly a cultural fad. You can design your system as multiple services interacting via a strict API even in a monolith. Failure isolation is a pattern of how to handle errors, whether they come across as a 40x HTTP status code, a JSON field called "error" or a method call throwing an exception. And a single system enables you to work with simple method calls calling across services instead of all the complication SOA can bring with signaling errors/serialization over a network layer. Not to mention all the operational overhead that comes with managing more hosts and more services.
- phamilton 10y ago
  You make the assumption that failures will be covered by defensive programming techniques. Unexpected failures will take down a monolith. One bad db query can deadlock all consumers. The greater the separation between components, the more easily you can isolate all failures, not just expected ones.
Waterluvian 10y ago
"Because I can hoard access to it over the other teams and make myself indispensable."
- vacri 10y ago
  In my experience as a small-business ops guy, other devs just aren't interested. I try to get them to learn a bit and avoid the personnel SPOF, but they have their own bailiwicks to lounge in. It's in my own interest to spread the knowledge around, because I like holidays-that-are-holidays.
sbov 10y ago
Why is independent scaling necessarily a good thing?
- nindalf 10y ago
  Say you have a single section of your code that's very compute intensive. If it's part of a larger monolith then you would spin up multiple instances of the entire monolith to keep up with traffic. To take a concrete example, say you have an app that does login, some social stuff, some billing, pulling from an external API and finally your special sauce - rendering a PDF of what your user has created. The last step is a bottle neck but none of the others are. You could increase the number of instances of renderers and your entire system would chug along. You've asked the right question though. Is independent scaling "necessarily" better? No, it's a trade-off. * To get this feature you've sacrificed the simplicity of having a single codebase leading to a single binary that can be deployed with ease. * You've added extra components within your system like load balancers. * What would earlier have been a function call is now RPC with network and serialisation overhead. * Your system might be a little more difficult to debug unless you collect logs in one place and it's possible to follow a single user across multiple components. * It's possible for your system to fail in ways that weren't possible before. For instance there might be a network issue between two specific nodes and it's hard to figure that out unless you have proper monitoring in place. Of course there are advantages other than failure isolation and independent scaling but I haven't gone into those.
  - sbov 10y ago
    Note that your example seems to only matter if I use differently spec'd instances for different services. If they all have the same ratio of e.g. memory to cpu to disk, I'm not sure what slicing my instances across services really gets me. If you slice it across services, you add a few instances, and your system chugs along. If you don't, you still add a few instances, and it still chugs along. I could maybe see if you have some services that take huge amounts of memory and others that take huge amounts of CPU. If you have a standard "monolithic app instance", and had to scale up your CPU by 10x but memory only by 1x because of your pdf renderer, you will likely be wasting large amounts of memory. But unless you have huge disparities in memory vs cpu use (and services don't vary in use together) I don't really know what sort of cost savings you can get here - wouldn't this be the actual value add of being able to independently scale - less cost because you can more accurately hit your resource (cpu vs memory vs disk and so on) requirements for your project? In contrast, from my experience, separating your web/app/database/cache layer from eachother tends to be extremely beneficial for independent scaling because they almost always vary widely in how they consume which resources (memory vs cpu vs disk and so on). They also tend to be written with this in mind and so it is basically free to do so. An aside, but many of your downsides apply not to just services, but also having to scale an application horizontally. If the name of the game is scaling, then many of these you will pay for regardless of the question of monolith vs. services. All that above aside, I definitely feel the other benefits. But I really don't get it - every article I read about services seems to mention independent scaling, when it seems like a fairly suspect benefit. Maybe I just haven't worked on the correct project.
lomnakkus 10y ago
If you're looking for "key" (dare I say "buzz") words or phrases, you're only going to get people who are gaming the system -- unless you're exceptionally lucky. That's assuming you're a large-enough employer to actually do this type of thing. If you're not a large employer, I can elaborate, but I'll have to tread a bit carefully, hence my reticence to do so.
- phamilton 10y ago
  I of course would follow up with "Why?" and "How?". And I'm really not looking for those exact words, but the fact that nothing even close was on this post made me want to succinctly bring up the topic.
tlow 10y ago
That's not enough. Having fail condition parameters understood and cases designed for is so often overlooked.
ma2rten 10y ago
Also independent deployment.
sshumaker 10y ago
Honestly, a big reason for separate services is cultural. It tends to be easier to assign accountability and ownership when you are operating a service (monitoring, independent deployment, KPIs fall out more naturally). With a monolith you have to work really hard to prevent a diffusion of responsibility. Yes, to some extent this means you are "shipping your org chart", but that's mostly inevitable at scale anyway.
- phamilton 10y ago
  Conways law at its finest
- hinkley 10y ago
  Or organizing your shipping chart. One of them has to bend.
newman314 10y ago
I've been calling it "fault domains" but pretty much the same thing. Also, "designing for failure modes". Things working at scale is to be expected. How to gracefully fail is what is hard.
gravyboat 10y ago
I tried to explain this once in an interview and the interviewer was not happy. They didn't like the idea of deploying a dozen different services. They also didn't like my ideas around keeping the services relatively simple and scaling out as needed. I can say I'm glad I didn't take that job.
- golergka 10y ago
  Technical interviews go both ways.
- riffraff 10y ago
  to be fair, it is quite possible an app does not need a dozen different services, and using them would just introduce unnecessary complexity. E.g. slashdot used to run with, IIRC, only 4 roles: memcached, reverse proxy, apache, and mysql. Your interviewer might have just failed in explaining the problem constraints properly.
  - javajosh 10y ago
    >Your interviewer might have just failed in explaining the problem constraints properly. The simpler and more common explanation is that the interviewer was looking to have his biases reflected by the interviewee. In my experience it is difficult for a certain kind of mind to distinguish between "This person is stupid/crazy/wrong for the job" and "This person knows something I don't".
- hinkley 10y ago
  Having the wrong service boundaries can obscure the data flow in ways that can be hard to fix. You end up with high fanout or ugly caching layers where a simpler structure might have been possible before process boundaries were introduced. Best to give it some time, figure out where the important interactions are and separate things that don't interact heavily.

mfukar 10y ago

Great, now anyone knows how to temporarily satisfy you while delivering nothing of what you actually want in the long run.

phamilton 10y ago
Surely, you understand I'm not literally only looking for buzzwords. I would certainly ask deeper questions on what they are trying to accomplish and why.
- mfukar 10y ago
  I'd heard the bar for humour had been lowered, but I didn't know how low.

zaptheimpaler 10y ago

I really don't buy this, its mostly a cultural fad. You can design your system as multiple services interacting via a strict API even in a monolith. Failure isolation is a pattern of how to handle errors, whether they come across as a 40x HTTP status code, a JSON field called "error" or a method call throwing an exception. And a single system enables you to work with simple method calls calling across services instead of all the complication SOA can bring with signaling errors/serialization over a network layer. Not to mention all the operational overhead that comes with managing more hosts and more services.

phamilton 10y ago
You make the assumption that failures will be covered by defensive programming techniques. Unexpected failures will take down a monolith. One bad db query can deadlock all consumers. The greater the separation between components, the more easily you can isolate all failures, not just expected ones.

Waterluvian 10y ago

"Because I can hoard access to it over the other teams and make myself indispensable."

vacri 10y ago
In my experience as a small-business ops guy, other devs just aren't interested. I try to get them to learn a bit and avoid the personnel SPOF, but they have their own bailiwicks to lounge in. It's in my own interest to spread the knowledge around, because I like holidays-that-are-holidays.

sbov 10y ago

Why is independent scaling necessarily a good thing?

nindalf 10y ago
Say you have a single section of your code that's very compute intensive. If it's part of a larger monolith then you would spin up multiple instances of the entire monolith to keep up with traffic. To take a concrete example, say you have an app that does login, some social stuff, some billing, pulling from an external API and finally your special sauce - rendering a PDF of what your user has created. The last step is a bottle neck but none of the others are. You could increase the number of instances of renderers and your entire system would chug along. You've asked the right question though. Is independent scaling "necessarily" better? No, it's a trade-off. * To get this feature you've sacrificed the simplicity of having a single codebase leading to a single binary that can be deployed with ease. * You've added extra components within your system like load balancers. * What would earlier have been a function call is now RPC with network and serialisation overhead. * Your system might be a little more difficult to debug unless you collect logs in one place and it's possible to follow a single user across multiple components. * It's possible for your system to fail in ways that weren't possible before. For instance there might be a network issue between two specific nodes and it's hard to figure that out unless you have proper monitoring in place. Of course there are advantages other than failure isolation and independent scaling but I haven't gone into those.
- sbov 10y ago
  Note that your example seems to only matter if I use differently spec'd instances for different services. If they all have the same ratio of e.g. memory to cpu to disk, I'm not sure what slicing my instances across services really gets me. If you slice it across services, you add a few instances, and your system chugs along. If you don't, you still add a few instances, and it still chugs along. I could maybe see if you have some services that take huge amounts of memory and others that take huge amounts of CPU. If you have a standard "monolithic app instance", and had to scale up your CPU by 10x but memory only by 1x because of your pdf renderer, you will likely be wasting large amounts of memory. But unless you have huge disparities in memory vs cpu use (and services don't vary in use together) I don't really know what sort of cost savings you can get here - wouldn't this be the actual value add of being able to independently scale - less cost because you can more accurately hit your resource (cpu vs memory vs disk and so on) requirements for your project? In contrast, from my experience, separating your web/app/database/cache layer from eachother tends to be extremely beneficial for independent scaling because they almost always vary widely in how they consume which resources (memory vs cpu vs disk and so on). They also tend to be written with this in mind and so it is basically free to do so. An aside, but many of your downsides apply not to just services, but also having to scale an application horizontally. If the name of the game is scaling, then many of these you will pay for regardless of the question of monolith vs. services. All that above aside, I definitely feel the other benefits. But I really don't get it - every article I read about services seems to mention independent scaling, when it seems like a fairly suspect benefit. Maybe I just haven't worked on the correct project.

lomnakkus 10y ago

If you're looking for "key" (dare I say "buzz") words or phrases, you're only going to get people who are gaming the system -- unless you're exceptionally lucky. That's assuming you're a large-enough employer to actually do this type of thing. If you're not a large employer, I can elaborate, but I'll have to tread a bit carefully, hence my reticence to do so.

phamilton 10y ago
I of course would follow up with "Why?" and "How?". And I'm really not looking for those exact words, but the fact that nothing even close was on this post made me want to succinctly bring up the topic.

tlow 10y ago

That's not enough. Having fail condition parameters understood and cases designed for is so often overlooked.

ma2rten 10y ago

Also independent deployment.

sshumaker 10y ago

Honestly, a big reason for separate services is cultural. It tends to be easier to assign accountability and ownership when you are operating a service (monitoring, independent deployment, KPIs fall out more naturally). With a monolith you have to work really hard to prevent a diffusion of responsibility. Yes, to some extent this means you are "shipping your org chart", but that's mostly inevitable at scale anyway.

phamilton 10y ago
Conways law at its finest
hinkley 10y ago
Or organizing your shipping chart. One of them has to bend.

newman314 10y ago

I've been calling it "fault domains" but pretty much the same thing. Also, "designing for failure modes". Things working at scale is to be expected. How to gracefully fail is what is hard.

gravyboat 10y ago

I tried to explain this once in an interview and the interviewer was not happy. They didn't like the idea of deploying a dozen different services. They also didn't like my ideas around keeping the services relatively simple and scaling out as needed. I can say I'm glad I didn't take that job.

golergka 10y ago
Technical interviews go both ways.
riffraff 10y ago
to be fair, it is quite possible an app does not need a dozen different services, and using them would just introduce unnecessary complexity. E.g. slashdot used to run with, IIRC, only 4 roles: memcached, reverse proxy, apache, and mysql. Your interviewer might have just failed in explaining the problem constraints properly.
- javajosh 10y ago
  >Your interviewer might have just failed in explaining the problem constraints properly. The simpler and more common explanation is that the interviewer was looking to have his biases reflected by the interviewee. In my experience it is difficult for a certain kind of mind to distinguish between "This person is stupid/crazy/wrong for the job" and "This person knows something I don't".
hinkley 10y ago
Having the wrong service boundaries can obscure the data flow in ways that can be hard to fix. You end up with high fanout or ugly caching layers where a simpler structure might have been possible before process boundaries were introduced. Best to give it some time, figure out where the important interactions are and separate things that don't interact heavily.