4 ms·

an antipattern I've observed when giving system design interviews is that a lot of people, when faced with a performance problem, will throw out "we should add

an antipattern I've observed when giving system design interviews is that a lot of people, when faced with a performance problem, will throw out "we should add a caching layer" as their first instinct, without considering whether it's really appropriate or not. for example, if the problem we're talking about is related to slow _writes_, not slow reads, the typical usage of a cache isn't going to help you at all. implementing write-through caching is certainly possible, but has additional pitfalls related to things like transactional integrity between your cache and your authoritative data store.

GaryNumanVevo 2y ago
It's a super common "new to SRE" behavior to overindex on caching as a silver bullet, especially because literally every DB has mechanisms to scale reads fairly easily. In my experience, redis is often needed when you have a DB team that doesn't want to put in the effort to scale reads
- sgarland 2y ago
  Or when the devs don’t want to rewrite their schemata in a way that would massively reduce I/O requirements. Then when you lose a cache node, the DB gets slammed and falls over, because when the DB team implemented service-based rate-limiting, the teams cried that they were violating their SLOs so the rate limits were bumped waaaay up.
re-thc 2y ago
> an antipattern I've observed when giving system design interviews is that It's an interview though. Most people just watch youtube videos and "copy and paste" the answer. In a way it's the format of the interview that's the problem. Similar to leet code style interviews a lot of the times we're not checking for what we need.
- dcow 2y ago
  At this point, what format of interview isn’t a problem?
  - re-thc 2y ago
    > At this point, what format of interview isn’t a problem? The original 1. Is a surgeon going to have a surgery test before they get hired? Is a chef going to live stream cooking a dish and then for the interviewer to virtually "taste" it before deciding on a hire?
    - dcow 2y ago
      Chefs actually are subject to being tasted before hire. It doesn’t happen in a 1hr interview, no. But the owners of a restaurant either know the chef’s cooking from previous experience or invote them in to cook. Doctors go through excessive amounts if additional schooling and board certification to prove they know how to e.g. cut open a body. Maybe you’re asking for the software industry to become more formalized in skillset requirements? Or maybe engineers should bring their portfolio instead?
      - re-thc 2y ago
        > Chefs actually are subject to being tasted before hire. It doesn’t happen in a 1hr interview Exactly. What you're describing is more closer to doing a take home and then explaining it. > Doctors go through excessive amounts if additional schooling and board certification to prove they know how to e.g. cut open a body. So not part of the interview? It's not like there aren't certification in the industry, e.g. AWS, Microsoft, Java, etc etc. > Maybe you’re asking for the software industry to become more formalized in skillset requirements? Or maybe engineers should bring their portfolio instead? I doubt "formalization" helps (see above, some of it exists and those are off the mark just as well). In essence leet code is the "formalized" requirement. It's just a bad 1. I'm asking to come back to common sense. Again, doctors don't cut open a body live as part of the interview. So even if I have my AWS certifications (equivalent of your example) the interviewer will and still does ask me to do a live system design interview. How is that the same?
- Too 2y ago
  Disagree on this one. In an interview there is no "the answer", it's a dialogue. I've interviewed a lot of people, often using performance related questions, and trust me, there are lots of candidates whose only answer to those is "add a cache", even after multiple follow-up questions or hints like "is there anything else that can be done?", "try thinking outside the box", "what can be done with the database itself", etc. Only a novice interviewer will be fooled by the first answer. If you cannot demonstrate more solutions after that, it shows that you clearly have no experience or problem-solving ability, which is the whole point of the interview to find out, not whether you have studied through a set of common questions. btw, "scale up" is the second most common answer from those who can't provide better solutions. :)
  - re-thc 2y ago
    > and trust me, there are lots of candidates whose only answer to those is "add a cache", even after multiple follow-up questions or hints My point isn't that the interview can't weed out bad candidates. That's in a way the easy part. The problem is it can't identify not-bad candidates. The interview is broken because of how standardized it is. It's like a certain game genre and most people will play it the same way. It's more like a memory test. > In an interview there is no "the answer", it's a dialogue. It pretends to be or you assume it is. There are numerous 'tutorials' / videos / guides on system design it's >90% rehearsed. So again, my point is the interviewee is trained and will give you the standard answer even if you deviate some. There are just too many risks otherwise. If I had a more novel approach I'd risk the interviewer not understanding or taking longer than the allocated time to finish. Especially in big tech - interviewers are trained to look for "signals" and not whether you're good or bad. They need to tick certain boxes. Even if you have a "better" answer if it's outside the box it fails.
- evil-olive 2y ago
  > Similar to leet code style interviews a lot of the times we're not checking for what we need. right, all interview formats are imperfect...but some are more or less imperfect than others. a crucial difference in my mind, is that leetcode-on-the-whiteboard style interviews correspond quite poorly to the actual day-to-day job of coding. a well-prepared system design question, on the other hand, does correspond fairly well to part of the actual job - we have an existing system, it has a performance problem, or needs a new feature we didn't anticipate originally that requires the design to be reworked in some way. and we're sitting in a meeting trying to come up with what we think will be the Least Bad option. (importantly, the question I like to use is not a greenfield "design Tinder for dogs" / "design Youtube for cats" type of question, because as you say those can be reduced to a formula that candidates can regurgitate, instead I'm intentionally asking a question about a brownfield system that I summarize for them first) ultimately, that's what I'm probing for with that interview style - I don't particularly care whether a candidate arrives at some "right answer" or not, I'm looking for "do I want to sit in a meeting with you and try to hammer out a design to some non-trivial problem that you and me and other people on the team will then go and implement?"
  - re-thc 2y ago
    > a crucial difference in my mind > a well-prepared system design question > importantly, the question I like to use i.e. you're just saying your version is better (than the general 1). Even if I agree with you, that's not the point. The point is that the water is polluted in >80% of the places so anyone coming to drink water will be weary regardless. > reduced to a formula that candidates can regurgitate > instead I'm intentionally asking a question about a brownfield system that I summarize for them first You'll get the filtered version even if you claim your water is clean. The problem isn't the question. > do I want to sit in a meeting with you and try to hammer out a design i.e. it's not a technical skills test but a behavioral or personal preference interview. As above but to make it 200% clear I've never done ANY real world system design like I would in ANY interview. So you're not likely going to get this outcome either. In the real world someone is going to get this task e.g. a principal engineer and they're going to come up with some draft (maybe ask for help for a bit) and then hold a meeting to discuss / refine it. No 1 is creating these diagrams live with other people unless there's some place with enough "architect-level" engineers that have nothing to do. Furthermore, it'd be really expensive if all the stakeholders are present. What you do get is the principal filling different gaps based on different discussions potentially over a week or so. The discussion potentially happens over many short segments as well, e.g. "should we add a cache (to the performance engineer)" ... (2 hours later) ... "I think we need a WAF, thoughts? (to the security engineer)". In conclusion you'd not want to sit in a meeting with me and try to hammer out a design because: - I'm trying to force myself to do something I don't do (and no 1 does) and so it's not the real me - I'm under pressure from the interview and the broken situation so behaving differently - I've had so little time to consider your "unique Brownfield" scenario that I'm always going to go with safe options instead of more novel or closer to my personality approaches i.e. again not me
lucb1e 2y ago
> throw out "we should add a caching layer" as their first instinct, without considering whether it's really appropriate or not Could be worse: you could have met me! I used to laugh at caching and thought that if your website is so slow that you need a caching layer (Wordpress comes to mind), you're just doing it wrong: perhaps you're missing indexes on your database or you simply can't code properly and made it more complex than necessary (I was young, once). Most of my projects are PHP scripts invoked by Apache, so they have no state and compute everything fresh. This is fine (think <30ms typical page generation time) for 95% of the types of things I make, but in more recent years I had two projects where I really struggled with that non-pragmatic mentality and spent long hours experimenting with different writing strategies (so data wouldn't change as often and MariaDB's built-in optimizations better), indexes on low-cardinality columns, indexes on combined columns in specific orders, documenting with each query which index it requires and maps to, optimizing the query itself of course, in one experiment writing my own on-disk index file to search through some gigabytes of data much faster than the database seemed to be able to do for geospatial information, upgraded the physical hardware from HDD to SSD... Long story short, I now run Redis and the website is no longer primarily bound by computation power but, instead, roughly equally by bandwidth I'm still very wary of introducing Redis to projects lest I doom them: it'll inevitably outgrow RAM if I indiscriminately stick things in there, which means turning them off (so far, nearly no links or tools on my website ever turned 404 because they're all on a "keep it simple" WAMP/LAMP stack that can do its thing for many years, perhaps search-and-replacing something like mysql_query() with mysqli->query() every five years but that's about the extent of the maintenance) So anyway I think we're in agreement about "apply where appropriate" but figured I'd share the counter-example of how one can also be counterproductive in the other direction and that there is something to be said for the pragmatic people that consider/try a cache, which often does help even if there's often a different underlying problem and my perfectionism wouldn't like it
- JamesSwift 2y ago
  I appreciate that you came around, but I think its important to highlight this common misunderstanding of the role of caching in achieving scaling beyond certain hard boundaries. You previously thought caching was bad, and instead should look at your indexes.... which are just a different kind of cache! I see this disconnect a lot, especially when in ruby/rails world when people think that the bottleneck is going to be the framework, while ignoring that the actual solution to scale is caching in various forms.
JamesSwift 2y ago
Well, thats probably because caching is generally the answer to all scaling problems. Once you hit the theoretical wall of performance, all you can do is cheat. And that generally means caching.

GaryNumanVevo 2y ago

It's a super common "new to SRE" behavior to overindex on caching as a silver bullet, especially because literally every DB has mechanisms to scale reads fairly easily. In my experience, redis is often needed when you have a DB team that doesn't want to put in the effort to scale reads

sgarland 2y ago
Or when the devs don’t want to rewrite their schemata in a way that would massively reduce I/O requirements. Then when you lose a cache node, the DB gets slammed and falls over, because when the DB team implemented service-based rate-limiting, the teams cried that they were violating their SLOs so the rate limits were bumped waaaay up.

re-thc 2y ago

> an antipattern I've observed when giving system design interviews is that It's an interview though. Most people just watch youtube videos and "copy and paste" the answer. In a way it's the format of the interview that's the problem. Similar to leet code style interviews a lot of the times we're not checking for what we need.

dcow 2y ago
At this point, what format of interview isn’t a problem?
- re-thc 2y ago
  > At this point, what format of interview isn’t a problem? The original 1. Is a surgeon going to have a surgery test before they get hired? Is a chef going to live stream cooking a dish and then for the interviewer to virtually "taste" it before deciding on a hire?
  - dcow 2y ago
    Chefs actually are subject to being tasted before hire. It doesn’t happen in a 1hr interview, no. But the owners of a restaurant either know the chef’s cooking from previous experience or invote them in to cook. Doctors go through excessive amounts if additional schooling and board certification to prove they know how to e.g. cut open a body. Maybe you’re asking for the software industry to become more formalized in skillset requirements? Or maybe engineers should bring their portfolio instead?
    - re-thc 2y ago
      > Chefs actually are subject to being tasted before hire. It doesn’t happen in a 1hr interview Exactly. What you're describing is more closer to doing a take home and then explaining it. > Doctors go through excessive amounts if additional schooling and board certification to prove they know how to e.g. cut open a body. So not part of the interview? It's not like there aren't certification in the industry, e.g. AWS, Microsoft, Java, etc etc. > Maybe you’re asking for the software industry to become more formalized in skillset requirements? Or maybe engineers should bring their portfolio instead? I doubt "formalization" helps (see above, some of it exists and those are off the mark just as well). In essence leet code is the "formalized" requirement. It's just a bad 1. I'm asking to come back to common sense. Again, doctors don't cut open a body live as part of the interview. So even if I have my AWS certifications (equivalent of your example) the interviewer will and still does ask me to do a live system design interview. How is that the same?
Too 2y ago
Disagree on this one. In an interview there is no "the answer", it's a dialogue. I've interviewed a lot of people, often using performance related questions, and trust me, there are lots of candidates whose only answer to those is "add a cache", even after multiple follow-up questions or hints like "is there anything else that can be done?", "try thinking outside the box", "what can be done with the database itself", etc. Only a novice interviewer will be fooled by the first answer. If you cannot demonstrate more solutions after that, it shows that you clearly have no experience or problem-solving ability, which is the whole point of the interview to find out, not whether you have studied through a set of common questions. btw, "scale up" is the second most common answer from those who can't provide better solutions. :)
- re-thc 2y ago
  > and trust me, there are lots of candidates whose only answer to those is "add a cache", even after multiple follow-up questions or hints My point isn't that the interview can't weed out bad candidates. That's in a way the easy part. The problem is it can't identify not-bad candidates. The interview is broken because of how standardized it is. It's like a certain game genre and most people will play it the same way. It's more like a memory test. > In an interview there is no "the answer", it's a dialogue. It pretends to be or you assume it is. There are numerous 'tutorials' / videos / guides on system design it's >90% rehearsed. So again, my point is the interviewee is trained and will give you the standard answer even if you deviate some. There are just too many risks otherwise. If I had a more novel approach I'd risk the interviewer not understanding or taking longer than the allocated time to finish. Especially in big tech - interviewers are trained to look for "signals" and not whether you're good or bad. They need to tick certain boxes. Even if you have a "better" answer if it's outside the box it fails.
evil-olive 2y ago
> Similar to leet code style interviews a lot of the times we're not checking for what we need. right, all interview formats are imperfect...but some are more or less imperfect than others. a crucial difference in my mind, is that leetcode-on-the-whiteboard style interviews correspond quite poorly to the actual day-to-day job of coding. a well-prepared system design question, on the other hand, does correspond fairly well to part of the actual job - we have an existing system, it has a performance problem, or needs a new feature we didn't anticipate originally that requires the design to be reworked in some way. and we're sitting in a meeting trying to come up with what we think will be the Least Bad option. (importantly, the question I like to use is not a greenfield "design Tinder for dogs" / "design Youtube for cats" type of question, because as you say those can be reduced to a formula that candidates can regurgitate, instead I'm intentionally asking a question about a brownfield system that I summarize for them first) ultimately, that's what I'm probing for with that interview style - I don't particularly care whether a candidate arrives at some "right answer" or not, I'm looking for "do I want to sit in a meeting with you and try to hammer out a design to some non-trivial problem that you and me and other people on the team will then go and implement?"
- re-thc 2y ago
  > a crucial difference in my mind > a well-prepared system design question > importantly, the question I like to use i.e. you're just saying your version is better (than the general 1). Even if I agree with you, that's not the point. The point is that the water is polluted in >80% of the places so anyone coming to drink water will be weary regardless. > reduced to a formula that candidates can regurgitate > instead I'm intentionally asking a question about a brownfield system that I summarize for them first You'll get the filtered version even if you claim your water is clean. The problem isn't the question. > do I want to sit in a meeting with you and try to hammer out a design i.e. it's not a technical skills test but a behavioral or personal preference interview. As above but to make it 200% clear I've never done ANY real world system design like I would in ANY interview. So you're not likely going to get this outcome either. In the real world someone is going to get this task e.g. a principal engineer and they're going to come up with some draft (maybe ask for help for a bit) and then hold a meeting to discuss / refine it. No 1 is creating these diagrams live with other people unless there's some place with enough "architect-level" engineers that have nothing to do. Furthermore, it'd be really expensive if all the stakeholders are present. What you do get is the principal filling different gaps based on different discussions potentially over a week or so. The discussion potentially happens over many short segments as well, e.g. "should we add a cache (to the performance engineer)" ... (2 hours later) ... "I think we need a WAF, thoughts? (to the security engineer)". In conclusion you'd not want to sit in a meeting with me and try to hammer out a design because: - I'm trying to force myself to do something I don't do (and no 1 does) and so it's not the real me - I'm under pressure from the interview and the broken situation so behaving differently - I've had so little time to consider your "unique Brownfield" scenario that I'm always going to go with safe options instead of more novel or closer to my personality approaches i.e. again not me

lucb1e 2y ago

> throw out "we should add a caching layer" as their first instinct, without considering whether it's really appropriate or not Could be worse: you could have met me! I used to laugh at caching and thought that if your website is so slow that you need a caching layer (Wordpress comes to mind), you're just doing it wrong: perhaps you're missing indexes on your database or you simply can't code properly and made it more complex than necessary (I was young, once). Most of my projects are PHP scripts invoked by Apache, so they have no state and compute everything fresh. This is fine (think <30ms typical page generation time) for 95% of the types of things I make, but in more recent years I had two projects where I really struggled with that non-pragmatic mentality and spent long hours experimenting with different writing strategies (so data wouldn't change as often and MariaDB's built-in optimizations better), indexes on low-cardinality columns, indexes on combined columns in specific orders, documenting with each query which index it requires and maps to, optimizing the query itself of course, in one experiment writing my own on-disk index file to search through some gigabytes of data much faster than the database seemed to be able to do for geospatial information, upgraded the physical hardware from HDD to SSD... Long story short, I now run Redis and the website is no longer primarily bound by computation power but, instead, roughly equally by bandwidth I'm still very wary of introducing Redis to projects lest I doom them: it'll inevitably outgrow RAM if I indiscriminately stick things in there, which means turning them off (so far, nearly no links or tools on my website ever turned 404 because they're all on a "keep it simple" WAMP/LAMP stack that can do its thing for many years, perhaps search-and-replacing something like mysql_query() with mysqli->query() every five years but that's about the extent of the maintenance) So anyway I think we're in agreement about "apply where appropriate" but figured I'd share the counter-example of how one can also be counterproductive in the other direction and that there is something to be said for the pragmatic people that consider/try a cache, which often does help even if there's often a different underlying problem and my perfectionism wouldn't like it

JamesSwift 2y ago
I appreciate that you came around, but I think its important to highlight this common misunderstanding of the role of caching in achieving scaling beyond certain hard boundaries. You previously thought caching was bad, and instead should look at your indexes.... which are just a different kind of cache! I see this disconnect a lot, especially when in ruby/rails world when people think that the bottleneck is going to be the framework, while ignoring that the actual solution to scale is caching in various forms.

JamesSwift 2y ago

Well, thats probably because caching is generally the answer to all scaling problems. Once you hit the theoretical wall of performance, all you can do is cheat. And that generally means caching.