8 ms·
I am on the interviewer side of ~1 system design interview per week and have done probably around 100 of them at Google and other companies. Most of the advice
by mgraczyk 4y ago
I am on the interviewer side of ~1 system design interview per week and have done probably around 100 of them at Google and other companies.
Most of the advice here is good, although it could be condensed quite a bit.
The biggest red flag for me as an interviewer is when candidates list off concepts or technologies they don't understand. DO NOT say things like "we can't do this because of the CAP theorem" or "we should add a cache to reduce latency" unless what you are saying is true and you can explain why this is true (the guide mentions this).
The biggest green flag is when the candidate obviously has thought deeply about related problems and made design decisions based on real world constraints and requirements. If you bring up a lot of related things you have worked on, explain why you made the decisions you did in those scenarios, if its obviously related to the question, and if you don't spend too much doing it, then you will definitely get a positive recommendation from me.
The purpose of the system design interview is to convince me that you could build, or lead a team to build, the thing that we are discussing at a company like the one where you are interviewing. I feel like I can easily see through interview practice or coaching, so I don't recommend spending more than an hour or so "preparing" for system design interviews. Unless you are interviewing somewhere with very inexperienced interviewers...
- leonidasv 4y ago>I feel like I can easily see through interview practice or coaching How do you spot it? I've had candidates with stellar performance on system design interviews, and I've never had any suspect of them (and the resumé also seemed pretty strong in those cases). But now I'm curious if I missed something.
- moandcompany 4y agoOne of the "heavy practice/coaching/leetcode grinding" signals is when the interviewee immediately dives into proposing a solution and discusses facets of that solution in great detail without bothering to ask clarifying questions or discussing assumptions and the risks/consequences of those assumptions.
- ditonal 4y agoI think the opposite is true. An experienced engineer not as versed in interview prep might hear the question to design something, and go ahead and show how they would design it based on their experience doing it in the real world. The leetcode grinder would almost certainly ask clarifying questions since it’s heavily emphasized as part of the rubric in pretty much all system design interview prep material. Of course, the non leetcode grinder might ask those questions, but even the social cues that you’re allowed to ask clarifying questions might not be understood. In my experience, I’ve seen a lot of people say that systems design test the seniority of a candidate, but then stick to a rubric that the leetcode grinder has memorized and the senior engineer might miss several important points on despite having the experience because they didn’t understand all the unwritten rules of the song and dance.
- apsurd 4y agoIt's funny cuz the leetcode grinder also learns that "asking clarifying questions" is part of the right answer structure too. So the cat-&-mouse game becomes more nuanced, but I do agree that it usually does become apparent that less-experienced candidates are _blindly_ following a strategy they learned somewhere. Interestingly this is not _inherently_ wrong since if you think about it, don't experienced people do the same thing? playbooks and checklists etc. But TLDR: the hollowness of each step of the approach does become apparent in my experience.
- leonidasv 4y agoThe business goal of interviews is not to select the best one, but to narrow down candidates with an acceptable fit for the role. An interview system with a high false-negative rate is not seem as problematic if the false-positive rate is also low. The cost of not hiring a very good candidate is way lower than the cost of hiring a bad one (even considering the opportunity cost in most cases).
- mgraczyk 4y agoIt's possible in those cases that the candidates were really good. The main tells are a mismatch between apparent practical experience and apparent knowledge of the design space. I always dig into a few specific technical aspects of the design as far as possible to see how deep the candidate can go. For example, if there's a queue I'll ask what would happen if in production the queue starts to fill, and how to mitigate. Most candidates reflexively suggest making the queue bigger, some suggest adding more downstream capacity to drain faster. People with real experience will have encountered this problem and know that you have to spend time investigating to find the root cause of the backpressure, otherwise you could add capacity to random things without reducing the queue occupancy. Are your queue consumers getting throttled by some downstream service? Are your workers CPU bound? The "right" answer here is basically "we should figure out why the queue is filling up before changing anything", but very few candidates answer that way. Another kind of mismatch is the red flag I listed, where candidates will basically list buzzwords or ideas regardless of whether or not they are related to the question I asked. For example if I ask a question where the user-facing API is very simple (ie, write opaque data to a log) then they start talking about the tradeoffs between graphql vs REST, that's a bad sign.
- jefQuery 4y agoIt feels cool to me that my nontraditional degree for software (ChemE) drilled this into us and now it comes up in my current job: Rate of input - rate of output = rate of accumulation It’d be fine to just up the queue capacity if the input is just inconsistent/spikey but averages to the input, but under steady state (or steadily increasing growth) higher input, you must increase the output rate or you will outgrow whatever capacity the system has
- yunwal 4y ago> People with real experience will have encountered this problem and know that you have to spend time investigating to find the root cause of the backpressure, otherwise you could add capacity to random things without reducing the queue occupancy. Are your queue consumers getting throttled by some downstream service? Are your workers CPU bound? The "right" answer here is basically "we should figure out why the queue is filling up before changing anything", but very few candidates answer that way. This is the opposite of what "people with real experience" do, at least in production-down situations. The first step is to mitigate, the second step is to root cause. If there's some no-brainer step that has a chance of alleviating the issue while you root-cause, and is unlikely to make things worse, you should take it.
- coolspot 4y ago> DO NOT say things like "we should add a cache to reduce latency” We should add a cache to reduce latency tho…
- jeron 4y agoyeah, not sure what is too hard to understand about that. Perhaps it depends on the context but I highly doubt you can go wrong by saying a cache should be added to reduce latency.
- N_A_T_E 4y agoOnly two hard things in computer science, blah blah.
- lhorie 4y agoThe way it typically fall aparts for my candidates is I ask about optimizations, they say X could be cached, then I ask about cache invalidation and lo and behold, it didn't even occur to them that the cache needs to be invalidated at some point, so they blurt something along the lines of doing the entire expensive operation again and checking it against the cache, or something similarly nonsensical. More common than I'd like to admit.
- ThalesX 4y ago> then I ask about cache invalidation No doubt with a follow up on how the variable would be best named.
- old_hat 4y agoThat's obvious. It's called "x".
- Apocryphon 4y agoYou mean x1
- yodsanklai 4y ago> I feel like I can easily see through interview practice or coaching, so I don't recommend spending more than an hour or so "preparing" for system design interviews. I'm very surprised by this recommendation which doesn't match my experience at all. First, lots of candidates don't have experience building such systems, so they need to learn by reading books and articles. Second, the candidates need to be familiar with the structure of the interview and what's expected from them, and that takes practice too.
- mgraczyk 4y agoAs an interviewer, my job is to differentiate between people who have experience vs people who have just read books and articles. I interview mostly L6+, so I will not recommend people who I don't believe have real experience building complex systems at scale. I think it only takes an hour or so to become familiar with the structure. I can tell during the interview if that's the problem and I will be very patient in explaining what I'm looking for in that sense.
- Kinrany 4y agoWhat is the difference between real experience and books in your opinion? How similar does the experience need to be to the problem at hand for it to be relevant?
- deleted 4y ago[deleted]
- paxys 4y agoI can't say about other companies but I have personally done many, many of these interviews over the years and can say with 100% confidence that someone who has spent days/weeks/months studying system design interview material but not had any actual experience building such systems in a production setting will get weeded out in the first few minutes. Now some interview prep might actually help a lot of people, but it will help you better present your existing knowledge, not replace it.
- 4y ago
- nvarsj 4y agoNo offense meant, but your comment already shows your inherent bias. You are looking for people who have solved specific system design problems. That means you are already being overlay narrow in your selection criteria. I have a background in quant dev and HFT for example. When I was doing system design for big tech (at L6 level), I found that the kind of system design questions asked had very little overlap with my own experience. Because the fields are different and the problems are different. I absolutely had to prepare by spending a lot of time familiarizing myself with big data/internet scale type stuff. Like spending months studying the DDIA book. Which is a lot different from low latency systems. And there's no way I would have passed those system design interviews without doing so. The stark reality is most of the people passing these have prepared heavily and you probably don't pick up on that - because part of the practice (e.g. mock interviews) is to make you sound natural. Now that I'm in the system, the other stark reality is that little of this system design matters. Most of the challenge is understanding the internal technologies and figuring out who to talk to to get stuff done (like pretty much every big company). I think ultimately system design, just like LC interviews, are a proxy for "can you study hard and pass exams". Which is supposed to be a loose proxy for what makes you successful on the job. I have mixed feelings on whether it gets the right people for the job or not.
- roguas 4y ago> Most of the challenge is understanding the internal technologies and figuring out who to talk to to get stuff done. This is something interesting that often is skipped due to "sterile" nature of system design questions(solve puzzle). 90% of system design problems are tightly connected with how your org works internally. Can you depend on particular team or is it better to provide requested functionality internally. This is very often neglected but this moves or breaks stuffs. Imagine you have to build a complete traffic system architecture for the city that will start settling in a year? It will likely be ongoing thing for you, you will observe how street layout works and adjust, build new streets, make city highway from point A to point B etc. but over time. Doing this apriori is going to go horribly bad.
- mgraczyk 4y agoWell for my own startups I don't do system design interviews. I don't think they're perfect and there are better ways of measuring the same thing if you have more time per candidate. Big companies like Google have to sometimes choose mediocre processes that are Lindy and agreed on. System design interviews are less bad than coding interviews IMO, and they are much more scalable for Google. I think I'm better at spotting genuine experience vs preparation than most interviewers because I've done a lot of interviews. On the other hand I don't think I can tell 100% of the time, and it's not black and white to begin with. Another point, your specific background may not matter for L6 roles. I want to know if you can build the sort of thing you will be building in the role you're interviewing for. I don't care if you're good at building something else. This is less true for L4 and L5.
- Plasmoid 4y agoA cache is such a red flag when I'm doing a system design interview. I ask people to design a system but the system itself is very write heavy. Like 50 writes per read imbalanced. People will often suggest a cache because it's in the standard interview prep but not actually think about why they need it. They just go right to "I need a cache somewhere".
- dilyevsky 4y agoOk if not cache then it must be hashtable?
- rippercushions 4y agoTo crack any IT interview, just breadth-first search the hashtable in a dictionary and you're golden.
- sophonX 4y agolol, i don't get it.
- quadrifoliate 4y ago> I ask people to design a system but the system itself is very write heavy. Like 50 writes per read imbalanced. People will often suggest a cache because it's in the standard interview prep but not actually think about why they need it. I mean, it's a cool thought process to go "Ooh, maybe I can store up repeated things so I don't need to repeat an expensive access process often!" It's a byproduct of modern computing hardware that we frequently have the luxury of not thinking about caching very often. I don't think you should punish people for considering a cache. Write caching is a thing, it might just not be applicable for the system you're considering.
- paxys 4y agoWell yeah, if you can say all this in the interview then you will be fine. 90% of candidates will instead read "add cache to improve performance" in their interview prep guide and repeat it verbatim.
- emodendroket 4y ago> The purpose of the system design interview is to convince me that you could build, or lead a team to build, the thing that we are discussing at a company like the one where you are interviewing. I feel like I can easily see through interview practice or coaching, so I don't recommend spending more than an hour or so "preparing" for system design interviews. Unless you are interviewing somewhere with very inexperienced interviewers... I don't agree with this because I bombed a couple interviews till I spent the time going through the prep. I distinctly remember an interview completely going off the rails because I was asked about a retail Web site and said "honestly I'd usually start with a simple app server - SQL database setup to start; no reason for the complexity of anything more complex till you've proven you've got the traffic and have scaling problems." That's my honest opinion, borne out by real-world experience, for most people thinking of doing something like that, but it obviously wasn't what the interviewer wanted to hear and it's not like I would have been capable of giving a fancier design; I just didn't know how you're supposed to approach the question.
- okdood64 4y ago> I just didn't know how you're supposed to approach the question. For your example, I would ask them to define what scale we were operating in, and what the footprint would be, etc. Then you could start off with a more complex design (which seems like what the interviewer wanted.)
- emodendroket 4y agoWell I know that now because I watched a bunch of prep videos and then easily passed interviews despite my experience not significantly changing in the intervening time. But that's my point; if you don't prep you have to just guess which aspects of this pretend scenario you're meant to be taking seriously and which are OK to gloss over. I had other interviews go off the rails because I offered too much detail about a smaller part of the system and discussion of it dominated the entire time but even though the interview kept engaging this discussion it wasn't actually what they wanted. "Don't bother prepping because you can't; it's just measuring your real abilities" is just not right, in my opinion.
- paxys 4y agoYup, exactly this. The easiest way to weed out bad candidates is to listen to all the buzzwords they rattle out and ask them to explain a single one and how it is relevant to their design. My first instruction to candidates always is – only talk about things you know. If your system design interview prep book told you that you should use time series databases for a certain category of questions but you don't actually know how they work, you will be screwed by trying to incorporate them in your design. Just use MySQL instead and tell me its pros and cons for this use case.
- time_to_smile 4y ago> although it could be condensed quite a bit. I bet I can condense this even more: it's yet another FAANG shibboleth that starts out as a good idea and the evolves into a ritual candidates need to perform until they can identify that they are "in". Google Algorithm/Whiteboard interview has morphed into a generation of robots that can solve any LC problem but are incapable of building anything real in software. Currently I think there are more books out there on "passing the systems design interview" than there are on actual systems design.