19 ms·

Learn how to design systems at scale and prepare for system design interviews

215 points by vinnyglennon 3y ago

asicsp 3y ago
See also "The System Design Primer": https://github.com/donnemartin/system-design-primer https://github.com/donnemartin/system-design-primer
- totallywrong 3y ago
  Very nice thanks.
- danielvaughn 3y ago
  ByteByteGo is also an incredible resource: https://www.youtube.com/@ByteByteGo https://www.youtube.com/@ByteByteGo
nineplay 3y ago
There's no shortage of criticism of the leet coding interview questions, but I found the system design interviews even more asinine. I have never in my career had to do anything like designing a large scale system. Maybe I'm inadequate, maybe I've been insufficiently motivated, but it hasn't happened. If that's a requirement, say so and don't waste the time of applicants who don't know what a ring tokenizer is. As it was, it turned into a ridiculous charade session where I watched a bunch of videos and regurgitated them as though I knew what I was talking about. "Oh yes, I'd use a column oriented database and put a load balancer in front". Without any real-word experience it's just a bunch of BS. I'd never let someone like me design a large scale system - not even close. I don't want to design large scale systems, it sounds boring and like the type of job where you're expected to be on call 24/7. I've worked with the Linux kernel, I've written device drivers, I've programed in everything from C to Go, and that's what I want to keep doing. Why put me through this?
- packetlost 3y ago
  IMO being able to think about the wider implications for smaller subsets of a system is an important ability. That being said, if your organization allows ICs to make technical decisions without any sort of review from someone whose job it is to architect large systems, idk that seems like something you should have. It also depends greatly on what 'layer' of the stack you work in.
  - nineplay 3y ago
    If you want to ask me about the wider implications for smaller subsystems then ask me about the Linux kernel. How did it become that system design is the only way to demonstrate this particular skill?
  - BasedAnon 3y ago
    There's a difference between talking about the wider implications of a system and acting like software can be planned top down with any sense of accuracy
- myth_drannon 3y ago
  It's a good way to filter out experienced candidates. The requirement is to be a junior dev that can take any BS that is dumped on him.
- PragmaticPulp 3y ago
  > I have never in my career had to do anything like designing a large scale system. Giving large scale system design interview questions for a role where someone never has to work with large scale systems would be a weird cargo cult choice. However, when a job involves working with large scale systems, it's important to understand the bigger picture even if you're never going to be the one designing the entire thing from scratch. Knowing why decisions were made and the context within which you're operating is important for being able to make good decisions. > I've worked with the Linux kernel, I've written device drivers, I've programed in everything from Fortran to Go, and that's what I want to keep doing. Why put me through this? If you were applying to a job for Linux kernel development, device driver development, and Fortran then I wouldn't expect your interviewers to ask about large scale distributed web development either. However, if you're applying to a job that involves doing large scale web development, then your experience writing Linux kernel code and device drivers obviously isn't a substitute for understanding these large scale system design questions.
  - nineplay 3y ago
    The problem is that the system design interview somehow became a necessary component of the FAANG hiring process.
    - Zetice 3y ago
      Why would you interview for a role at a FAANG company in the first place?
      - nineplay 3y ago
        In retrospect it was a horrible mistake.
      - bagels 3y ago
        They pay a lot?
      - scarface_74 3y ago
        They want to exchange the most money possible for their labor ?
- xyzzy_plugh 3y ago
  I find system design interviews generally provide significantly more signal than coding interviews, but still less signal than a lunch interview. It's a test of breadth and depth and it takes an apt, possibly experienced interviewer to navigate the candidate's domain knowledge effectively. The goal isn't to build some elaborate, buzzwordy house of cards, but to understand where tools are appropriate and where they are not, to think on your feet and work with an interviewer to build a system that makes sense. And, just like the real world, criteria should shift and change as you flesh out the design. In particular people trip over the same things every time: reaching for things they do not understand, not understanding fundamental properties of infrastructure (CPUs, memory, networking), and cache invalidation. When I interview folks I always preface the prompt with an offer to provide advice or information, acting as if I were a trusted colleague or stakeholder. If you want to be low level, then I'd question why we're conducting a large system design interview anyways. We could certainly frame it as a small system design instead, and focus on the universe contained within the injection-molded exoskeleton of the widget. If you say "column oriented" I'm going to ask you to explain why. I'm going to challenge what the load balancer is doing or what you expect it to do, and why. Building large systems in the real world well, and watching them scale up under load with grace (and without contorting your opex to have only lunar aspirations) is somewhat akin to watching your child ride their bike for the first time after the training wheels are off. It feels good. Just like seeing your hardware in the field produce a low failure yield. There is satisfaction in doing good work, or at least their should be.
  - nineplay 3y ago
    > If you say "column oriented" I'm going to ask you to explain why. I'm going to challenge what the load balancer is doing or what you expect it to do, and why. And I would have happily repeated something from a video I'd watched on YouTube two nights earlier. It's a cram test.
    - SpicyLemonZest 3y ago
      I don't know how I could prove or verify this, but in my experience it feels very easy to detect a difference between people who understand a systems topic and people who've treated it like a cram test. I recall one interview in particular where the guy gave textbook answers to anything like "what technology would you use here" or "what are the benefits of X vs. Y", but fell apart completely whenever I scratched the surface for an implementation detail.
- whalesalad 3y ago
  Interesting take. Different strokes for different folks? You aren't right or wrong here, it's preference. That being said I am in the opposite camp and I find that more and more, the systems that I am building and maintaining are large and distributed. Despite what a lot of commenters here on HN will say - yes there absolutely are businesses out there that need tooling like Kubernetes and huge column databases.
- endtime 3y ago
  Agreed 100%. I spent ten years at Google, got promoted three times, never did any distributed systems stuff. When I decided to leave about 18 months ago, trying to cram for these interviews and memorize how WhatsApp works was the worst part of interviewing. But I jumped through the hoop and got a couple offers, neither of which involve doing distributed systems work. Those were definitely my weakest interviews, and in the case of my current employer, I literally said to my HM, "I've never done this kind of work, and I'm not going to shine in these interviews. Here's the kind of work I have done and I'd love to talk to you about it instead." I still had to do the sys design interview, but I think maybe that helped get it down weighted?
  - yodsanklai 3y ago
    I never did any system design, and got offers at Meta and Google (I even aced the system design at Google). It's not a very discriminating interview I believe. And I found it quite fun to prepare.
    - operatingthetan 3y ago
      >And I found it quite fun to prepare. What resources did you use?
      - yodsanklai 3y ago
        Quite a lot. I read many times "Designing data-intensive applications" which I highly recommend. There's a book called "system design interview" I believe, that is a summary of the most typical designs. There are also a bunch of videos on youtube. I read some research papers of classical designs. I played with some typical components, such as nosql dbs. I even implemented some prototypes.
        shadowfox 3y ago
        Do you have any pointers to these research papers?
- didip 3y ago
  System design questions are way more relevant to day-to-day work than leetcode questions. Nothing stops the interviewer from asking you even more relevant system designs questions like: * How would you build a Linux kernel from scratch? * How would you design a common interface for any device drivers (that you are familiar of)?
- sanderjd 3y ago
  I have designed a few systems, but my issue with the system design interview is that this is not how it works. There is never a blank page in real life like it is in one of these interviews, and the stuff that's actually on the page matters more than the stuff you're "supposed" to say in these sessions. Yes, column-oriented databases absolutely do work better for OLAP use cases, but is it better enough for the specific use case to be worth introducing a new database technology into the organization, or would a new database within the existing managed psql instance be good enough for now? Those detailed organizational questions usually matter more in the first few iterations of systems than "principled" architectural concerns. The useful kind of design is: What is the next best iteration of this system? Maybe with an appendix at the end discussing some ideas for the iteration after that. Sometimes that next iteration is actually the first iteration of the system, in which case you should definitely not be drawing 20 boxes with different components of how it will fit together, you should be looking for the simplest possible thing that could work. One of the fun things at big successful companies is that there are actually a lot of systems that are quite a few iterations in, with a stable baseline of usage properties. With these, it actually can make sense to draw a bunch of boxes with different components targeting different well-known pain points in a way that avoids trading off important existing capabilities. But again, that's exactly the opposite of a blank page, and no amount of digging into the interviewer's toy system design question can get deep enough for that. All of my answers to these questions - which have always been very well-received - have been over-engineered solutions that I'd never actually pursue in a real job. But interviewers aren't really prepared for questions like "what frameworks are already being used and familiar to most teams at the company?" followed by "since we already have familiarity with postgresql+rails+react, we should set up a new non-replicated but periodically backed up database in that database instance, start with a few tables with some reasonable relationships, use activerecord and its migrations, and implement the front-end in react, then we should launch it and see what bottlenecks and pain points arise". I get it, these interviews are trying to see if you have the knowledge and chops to solve those pain points and bottlenecks as they arise, but I'm sorry, "do an up-front design of a huge fancy system" just doesn't answer that question.
  - nineplay 3y ago
    I was asked how I would replace an large scale but inadequate logging infrastructure and I stared with "In a way that minimally disrupts everything currently in place for monitoring and alerting". I'm not sure how well that answer played out but it's still the correct answer.
- yodsanklai 3y ago
  If you apply to a generalist SWE role at FAANG, you're expected to have some knowledge about these things since you're likely to encounter them. I didn't have such experience either, but it's now part of my day to day job. If you apply for a targeted specialized role, you may get a system design question that relates to your domain. If there's a general system design interview, it should have less weight in the process. It's still an indicator on how candidates communicate and think through a design. Plus, these kind of things should be general knowledge for a computer scientist nowadays.
  - htrp 3y ago
    > If you apply for a targeted specialized role, you may get a system design question that relates to your domain. If you apply for a specialized targeted role.... you still get the same generalized interview loop
    - yodsanklai 3y ago
      Depends on the company. Sometimes you get extra interviews in addition to the generalist ones.
- HumblyTossed 3y ago
  I think they're BS also. What you should be looking for is does the person understand how the project(s) they have worked on fit into a larger system. For instance, I have a high level understanding of how the other systems at the Co I work for work. I know what they do, I know what data they ingest and expel. I know how the data I consume is generated. I know how it all fits together and I can talk conversantly about it. I know that when a change request comes in if it affects other teams and I voice out when it does. I think this is what you should look for in most candidates except high level system engineers and jr developers.
- opportune 3y ago
  I have designed large scale systems, but I often feel most of my system design interviewers haven’t and are much more pedantic/nitpicky than is reasonable. Leetcode is better because most people at least understand the questions they’re asking. Last time I did a system design interview I mentioned database triggers as a way to maintain some kind of data invariant, which flustered my interviewer and I guess made it so their canned follow up questions didn’t work, so they asked if I could think of any other approach (the one they had in mind). I couldn’t and it made the interview very painful.
  - dumpster_fire 3y ago
    You just reminded me of the time I was explaining how to do consensus failover, and the interviewer asked me to do distributed single thread design. Great use of everyone's time. The market is full of LARPers who wish they had the chance to design something real, and they will take it out on you.
  - danielvaughn 3y ago
    This is a good example of a bad interviewer. Never have a specific concrete answer in mind when asking an open-ended question.
    - kagevf 3y ago
      Or, if you want the candidate to talk about a certain approach, maybe think of some ways you can nudge them in that direction. If something's not at top of mind, some subtle hints from the interviewer could trigger a discussion of what they want to talk about. The interviewer shouldn't be tied to a "script".
    - dekhn 3y ago
      For the really good candidates (IE, ones who can answer basic programming questions and explain how a hash table works), I have an open-ended question which is really an open question in an active field. I really hoped I'd get candidates who would get that far (https://arxiv.org/abs/1309.2975 https://arxiv.org/abs/1309.2975) so that I could finally get some interesting answers I haven't heard before but most candidates I end up interviewing struggle to explain a hash table and what its advantages/disadvantages for counting unique k-mers are.
- synergy20 3y ago
  I'm also a system programmer developer for the most part(kernel, hardware bring-up, low level firmware, performance tuning of embedded systems) and I got invited a couple times for FAANG interviews,and all of them had this system design interview with nosql column databases and loadbalancers behind nginx proxies of some sort. Problem is that, it's pretty far from my field. Are you going to ask a cloud architect who connects frontend to backend with databases at scale a lot of questions about how to write a pci express network driver in linux kernel with great performance too? I would like to be hired by a specialized hiring team in those big companies instead of going through the general hiring process that you're expected to be a technical God who knows everything at expert level. I rejected all those interview requests.
  - deleted 3y ago
    [deleted]
- dekhn 3y ago
  I built one large scale system in my career (when I worked at Google, I made a folding@home screensaver that used up idle cycles in production). When I built it I ignored 95% of what Google knew about large scale system design because that knowledge was really about building scalable web services and storage systems, while I needed to build a batch scheduler which could handle many millions of tasks simultaneously. We depended on a few core scalable resources available in production (borg, chubby, bigtable, colossus) and tried as hard as possible to avoid spamming them with excessive traffic without adding lots of complex caches and other hard-to-debug systems. In fact, "simplicity" was the primary design goal. The system worked, it scaled, and if I'd followed all the normal Google guidelines, it wouldn't have (because scientific computation and web load balancers differ). Not sure what to take away from that. These days in system design interviews I usually focus on limiting the use cases for the system so that I can architect something that: linear resource consumption for linear workloads over 2-3 orders of magnitude, is simple enough that a small group of engineers can understand the whole system and debug it when it breaks, and not try to optimize for future use cases (clearly documenting the limits of the system) or try to accomodate too many oddball one-off user requests.
- scarface_74 3y ago
  I agree. I am shocked even though I shouldn’t be at this part of the technical interview for devs and that devs can pass these interviews without demonstrating practical experience. I’ve had interviews and jobs for three smaller companies where I was actually coming in as an architect 2016-2020). But they wanted to know about my real world experience. Luckily my second job out of college way back in 1999 I actually had to manage servers as well as develop so I could both talk theory and practice. From 1999-2012 I was managing infrastructure as part of my job at two jobs. I’ve never interviewed at BigTech for a software engineering position. But I did do a slight pivot and interview (and presently work) in cloud consulting at BigTech specializing in “application modernization” - cloud DevOps + development. Sure I had the one initial phone screen where I had to talk theory about system design. But my entire loop consisted of my walking through my past system design experience - and not all centered around AWS. And yes I can talk intelligently about all of the sections that the page covers including your example of columnar vs row databases. But I wouldn’t expect that from most devs. I was never on call at my last job. We had “follow the sun” support. But our site was only business critical during the day. One of the first things I insisted on with my CTO is that we hire a manage service provider for non business hours support. Sort of related: at most tech companies, the difference between mid and Senior is not coding ability. It’s system design and “scope” and “impact”
Zetice 3y ago
When I see something like this I wonder if there are people out there who actually go chapter-by-chapter from start to finish, spending dozens of hours learning the content a random GitHub repo claimed was important.
- mynegation 3y ago
  You can always go for the fundamentals and go through DDIA. It is heavy on foundations but does not really give examples of specific real world systems. For that, Alex Xu’s books are probably the most popular.
- worksonmine 3y ago
  This repo is a very high level collection of topics with short descriptions rather than a learning source. You can skim it a couple of minutes. But if you're curious about a topic, why wouldn't you read a free resource on it if the quality seems good? Are you always this dismissive? How do you learn?
  - Zetice 3y ago
    “quality seems good” is the point of contention here; how would you know? You’re either already knowledgeable about this topic and therefore necessarily didn’t learn it from here or you’re not and therefore unqualified to judge its quality at all.
    - worksonmine 3y ago
      That applies to everything we learn. Most of us use previous experience to make an educated guess. Someone just coming into software won't be reading this, but it can be good start to know what one could dig deeper into. Most content on any topic is bullshit, and in tech quickly outdated. What's your gripe with this particular one? I have several similar repos starred for future reference, because why not? Could be useful. You never answered how you learn? Because I assume you still do? Since you have opinions on how others do it maybe you have better approaches you'd like to share?
OnionBlender 3y ago
Can someone suggest a book or resource to learn more about designing a software system but not "scalable" system design that so many interview prep books talk about? I'm especially interested in something that is _not_ web related. I've spent most of my career fixing bugs or taking over apps/systems that someone else started. I really don't know how to design something new. (I mostly work on APIs, Android apps/services, and a bit of game middleware).
- vishnugupta 3y ago
  Nothing like learning from popular software architecture https://aosabook.org/en/ https://aosabook.org/en/
zitterbewegung 3y ago
Honestly a better resource for this is the following book . It is very easy to read and understand to use what to do where. https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/ https://www.oreilly.com/library/view/designing-data-intensiv...
- Maxpowerfreak 3y ago
  And for those not too sure about buying yet another O'Reilly book they may or may not read, I recommend checking your local library or school (if a student). They may have the book physically or, even better, have a subscription to O'Reilly's online platform,letting you access any of their books and even some of them in pre-release. I find that I only use a few of my books enough to justify the purchase, and I prefer skimming chapters I find interesting through the platform access I get from my library account.
  - esafak 3y ago
    That particular book has a long reservation line in my library network, because it is so popular.
    - saratogacx 3y ago
      My local library has a subscription to O’Reilly books included in being a card-holder. No waiting needed if you are not set on the dead-tree version.
itissid 3y ago
Interviewer: How do you design twitter? Interviewee: Long answer with novice RDBMS choices. Interviewer(implicitly): I will judge you based on how you can "Problem Solve" many-many conundrum. Interviewee: Use Cache, scaling, edge nodes, kafka because.... .... HIRED .... Interviewee spends first 6 months writing a test framework to automatically run tests of deployment of an internal tool that runs python ingestion pipeline. Next 6 months figuring out how to help user add type hints to IDE for pipelines to help catch errors faster and how to roll it out to 100+ ICs in the company. Sadly fails. Never ever writes a line of code to make "twitter", learns a ton on how to work with people. .... Interviews again: "How do you design twitter"?
- linza 3y ago
  Not sure what the point is here, but ... in my experience as the hiring manager I haven't seen huge success in testing a candidate whether they can do the exact thing they will be doing in the first 6 months (in software engineering). I had better results in asking questions that bias towards a certain set of behavior traits (yes, this has other long-term problems) and not certain skills. I understand that some candidates don't like it, because they are obviously good at something and want to be asked questions in that field. But ... I already know you are good at it because you said it on the CV, so i'm not wasting time with that (yes, if you are a good liar it takes us longer to find out). So your example, working as intended?
  - itissid 3y ago
    My point was that the culture of system design interviews is not a good one. Its repeated to infinity at many companies. > I had better results in asking questions that bias towards a certain set of behavior traits I am curious. What do you ask? Like do you ask tech centric questions around Proactiveness, Empathy, Motivation, Conflict Resolution etc?
    - linza 3y ago
      I still ask to design Twitter, but make it clear that it's about the process, and follow ups are driven by where the candidate leads to. The material from OP won't prepare you for systems design interview (witch is also your point?). Only experience will, so we measure experience and leadership. Juniors can apply to systems design openings and will be interviewed, but they end up as generic SWEs if hired. If one thinks one can work on distributed systems coming out of university you will have a hard time interviewing or not like it.
farhanhubble 3y ago
I've seen people cram their heads with all kinds of prep problems like designing an elevator, a parking lot or a social media site and I have never understood the rationale behind such questions. Design isn't something you could do in an hour. Design is also a trial and error thing. Slight changes in data flow might change the entire system architecture.
itissid 3y ago
This is not how people learn. And I always found these books more useful after I have actually worked on a system than before. Do you know why people ask such questions at any company? Its because there is a rubrick at every company and 1/2 the people don't have a clue what else to ask except because they have no other good ideas or because they may have done it themselves and it was a nice problem without realizing what that experience gave them never actually gives the same to the new guy. Skill addition is not additive it is about comparing choices and critically analyzing risks on things you *have already worked on*. Its as much about knowing what won't work i.e. subtractive. If you have never worked on a distributed system and are asked to work on some part of it the reasonable way it happens is: 1. You will be asked to initially to start working on a small part. 2. You will/should have someone to help you if you have no clue. This is called seeking feedback and it is done on google docs where people critique your design and you iterate. 3. You should have the time and sense to read up when you do (1), (2). 4. Maintain mental sanity and find mentors to help you out. Notice that a system design interview prep will help you with exactly *zero* of the steps above. For all other situations where you are forced to do more than the above, run away in the opposite direction. I think if someone just published a Cheat Sheet on the actual Rubrick it would be so thoroughly gamed that it would prove the point that these things are pointless.
zartstrom 3y ago
I'm not sure if I would be prepared to design systems at scale after reading this ... extended glossary. I like the short explanations though.
birdymcbird 3y ago
until recently i was a principal engineer at amazon. so maybe my opinion has some weight. system design interview is more about interviewee asking questions..taking time to understand the problem..ask about product feature or SLA..understand functional and non functional requirement. then its about candidate showing some knowledge set showing they can think and reason behind some immediate coding task. Demonstrate ability to make judgment..simplify where possible..discuss costs and trade offs. this interview not about candidate building some system at scale themself. building and supporting has trials and lessons you only learn by doing and failing, not through interview prep or YouTube videos
- CharlieDigital 3y ago
  Candidates can't read minds. The best technical interviews I've been on as a interviewee have been those where the expectations are clear. In your example: "We're not expecting you to create Twitter in 15 minutes, but we want to understand how you think about the challenges and key considerations of building a large system like Twitter" Many interviewers fail to provide enough context and that leaves the interpretation of the prompt too wide open. At that point, the interview has failed since whether a candidate can provide an answer that is aligned with the expectations of the interviewer has an element of chance to it.
  - mrleinad 3y ago
    > Many interviewers fail to provide enough context and that leaves the interpretation of the prompt too wide open So do clients/customers. What's your point? That an interview to assess whether a developer can elicit requirements should be less hard than dealing with an actual customer?
    - CharlieDigital 3y ago
      An interview with a customer for requirements had a very clear context. An interview for a technical position could focus on either high level design, specific technical aspects, or process and so approach to problem solving. Maybe a bit of each. An interviewer that more clearly defines the context can get better responses.
mkl95 3y ago
I decline non big tech interviews that involve some kind of "system design". I've worked with so-called staff engineers who designed systems with textbook antipatterns (I literally studied them). This idea that you need to regurgitate some bloated mess onto the cloud even to solve even the simplest problems needs to stop.
moandcompany 3y ago
The problem we have today is that the Cracking the Coding Interview/leetcode grinder approach toward interviewing has now overrun the original purpose of system design interviews. It looks like people that lack the understanding of their purpose, or experience to conduct these interviews properly, now run these interviews, and/or often run the show, writing books/articles/etc, which perpetuate the cargo cult behavior of regurgitating patterns described in some influencer book/article/video. This is somewhat associated with the title-inflation we've seen in our industry where "senior" and "staff" level engineers today seem to have much less experience than persons with the same titles a decade ago, and the people running these interviews are today's "senior and staff" engineers, or managers with very shallow engineering experience. Systems design interviews were meant to be non-grindable, open-ended interview questions which allow a very experienced engineer as interviewer to understand the interviewee's experience and approach in solving systems-level problems. The original purpose of the systems design question in engineering interviews was to: 1) Give the candidate an open-ended question that is not meant to be memorizable or fully solved within the allotted time. 2) Give the candidate an opportunity to shine -- expose and demonstrate areas where they have particular depth from their past work experience Systems Design interviews employ the judgment of the highly experienced engineer as interviewer, rather than checking for a specific answer. As an interviewee, the first phase of a systems design interview is really about asking questions to better understand context/constraints/considerations/etc before formulating solutions. The second phase of a systems design interview is about communicating your understanding of the "whys" and clarifying/refining those understandings with your interviewer. Ironically, we also have a post on Hacker News today ("The hardest part of building software is not coding, it's requirements") [https://news.ycombinator.com/item?id=36597709 https://news.ycombinator.com/item?id=36597709] which speaks to the point that systems design interviews really being a test to see how a candidate elicits requirements and considers trade-offs, when developing solutions for problems.
Thaxll 3y ago
There are things in that doc that are completly wrong and ridiculous such as: https://github.com/karanpratapsingh/system-design#streaming-a-video https://github.com/karanpratapsingh/system-design#streaming-... It reminds me a Youtube video on how to create a Netflix clone, for streaming they were creating a Java class for each frame of a video.
VirusNewbie 3y ago
Has...the person who wrote this designed large scale systems?
- unmole 3y ago
  Doesn't matter. The person conducting the interview has probably never done that. And the job itself probably won't involve large scale system design.
  - VirusNewbie 3y ago
    My last three jobs involved large scale systems. I've designed numerous ones. The last company I was at was a small startup and still did billions of transactions, a non trivial amount that certainly can't be handled by a single machine.

215 points by vinnyglennon 3y ago

See also "The System Design Primer": https://github.com/donnemartin/system-design-primer https://github.com/donnemartin/system-design-primer

totallywrong 3y ago
Very nice thanks.
danielvaughn 3y ago
ByteByteGo is also an incredible resource: https://www.youtube.com/@ByteByteGo https://www.youtube.com/@ByteByteGo

nineplay 3y ago

There's no shortage of criticism of the leet coding interview questions, but I found the system design interviews even more asinine. I have never in my career had to do anything like designing a large scale system. Maybe I'm inadequate, maybe I've been insufficiently motivated, but it hasn't happened. If that's a requirement, say so and don't waste the time of applicants who don't know what a ring tokenizer is. As it was, it turned into a ridiculous charade session where I watched a bunch of videos and regurgitated them as though I knew what I was talking about. "Oh yes, I'd use a column oriented database and put a load balancer in front". Without any real-word experience it's just a bunch of BS. I'd never let someone like me design a large scale system - not even close. I don't want to design large scale systems, it sounds boring and like the type of job where you're expected to be on call 24/7. I've worked with the Linux kernel, I've written device drivers, I've programed in everything from C to Go, and that's what I want to keep doing. Why put me through this?

packetlost 3y ago
IMO being able to think about the wider implications for smaller subsets of a system is an important ability. That being said, if your organization allows ICs to make technical decisions without any sort of review from someone whose job it is to architect large systems, idk that seems like something you should have. It also depends greatly on what 'layer' of the stack you work in.
- nineplay 3y ago
  If you want to ask me about the wider implications for smaller subsystems then ask me about the Linux kernel. How did it become that system design is the only way to demonstrate this particular skill?
- BasedAnon 3y ago
  There's a difference between talking about the wider implications of a system and acting like software can be planned top down with any sense of accuracy
myth_drannon 3y ago
It's a good way to filter out experienced candidates. The requirement is to be a junior dev that can take any BS that is dumped on him.
PragmaticPulp 3y ago
> I have never in my career had to do anything like designing a large scale system. Giving large scale system design interview questions for a role where someone never has to work with large scale systems would be a weird cargo cult choice. However, when a job involves working with large scale systems, it's important to understand the bigger picture even if you're never going to be the one designing the entire thing from scratch. Knowing why decisions were made and the context within which you're operating is important for being able to make good decisions. > I've worked with the Linux kernel, I've written device drivers, I've programed in everything from Fortran to Go, and that's what I want to keep doing. Why put me through this? If you were applying to a job for Linux kernel development, device driver development, and Fortran then I wouldn't expect your interviewers to ask about large scale distributed web development either. However, if you're applying to a job that involves doing large scale web development, then your experience writing Linux kernel code and device drivers obviously isn't a substitute for understanding these large scale system design questions.
- nineplay 3y ago
  The problem is that the system design interview somehow became a necessary component of the FAANG hiring process.
  - Zetice 3y ago
    Why would you interview for a role at a FAANG company in the first place?
    - nineplay 3y ago
      In retrospect it was a horrible mistake.
    - bagels 3y ago
      They pay a lot?
    - scarface_74 3y ago
      They want to exchange the most money possible for their labor ?
xyzzy_plugh 3y ago
I find system design interviews generally provide significantly more signal than coding interviews, but still less signal than a lunch interview. It's a test of breadth and depth and it takes an apt, possibly experienced interviewer to navigate the candidate's domain knowledge effectively. The goal isn't to build some elaborate, buzzwordy house of cards, but to understand where tools are appropriate and where they are not, to think on your feet and work with an interviewer to build a system that makes sense. And, just like the real world, criteria should shift and change as you flesh out the design. In particular people trip over the same things every time: reaching for things they do not understand, not understanding fundamental properties of infrastructure (CPUs, memory, networking), and cache invalidation. When I interview folks I always preface the prompt with an offer to provide advice or information, acting as if I were a trusted colleague or stakeholder. If you want to be low level, then I'd question why we're conducting a large system design interview anyways. We could certainly frame it as a small system design instead, and focus on the universe contained within the injection-molded exoskeleton of the widget. If you say "column oriented" I'm going to ask you to explain why. I'm going to challenge what the load balancer is doing or what you expect it to do, and why. Building large systems in the real world well, and watching them scale up under load with grace (and without contorting your opex to have only lunar aspirations) is somewhat akin to watching your child ride their bike for the first time after the training wheels are off. It feels good. Just like seeing your hardware in the field produce a low failure yield. There is satisfaction in doing good work, or at least their should be.
- nineplay 3y ago
  > If you say "column oriented" I'm going to ask you to explain why. I'm going to challenge what the load balancer is doing or what you expect it to do, and why. And I would have happily repeated something from a video I'd watched on YouTube two nights earlier. It's a cram test.
  - SpicyLemonZest 3y ago
    I don't know how I could prove or verify this, but in my experience it feels very easy to detect a difference between people who understand a systems topic and people who've treated it like a cram test. I recall one interview in particular where the guy gave textbook answers to anything like "what technology would you use here" or "what are the benefits of X vs. Y", but fell apart completely whenever I scratched the surface for an implementation detail.
whalesalad 3y ago
Interesting take. Different strokes for different folks? You aren't right or wrong here, it's preference. That being said I am in the opposite camp and I find that more and more, the systems that I am building and maintaining are large and distributed. Despite what a lot of commenters here on HN will say - yes there absolutely are businesses out there that need tooling like Kubernetes and huge column databases.
endtime 3y ago
Agreed 100%. I spent ten years at Google, got promoted three times, never did any distributed systems stuff. When I decided to leave about 18 months ago, trying to cram for these interviews and memorize how WhatsApp works was the worst part of interviewing. But I jumped through the hoop and got a couple offers, neither of which involve doing distributed systems work. Those were definitely my weakest interviews, and in the case of my current employer, I literally said to my HM, "I've never done this kind of work, and I'm not going to shine in these interviews. Here's the kind of work I have done and I'd love to talk to you about it instead." I still had to do the sys design interview, but I think maybe that helped get it down weighted?
- yodsanklai 3y ago
  I never did any system design, and got offers at Meta and Google (I even aced the system design at Google). It's not a very discriminating interview I believe. And I found it quite fun to prepare.
  - operatingthetan 3y ago
    >And I found it quite fun to prepare. What resources did you use?
    - yodsanklai 3y ago
      Quite a lot. I read many times "Designing data-intensive applications" which I highly recommend. There's a book called "system design interview" I believe, that is a summary of the most typical designs. There are also a bunch of videos on youtube. I read some research papers of classical designs. I played with some typical components, such as nosql dbs. I even implemented some prototypes.
      - shadowfox 3y ago
        Do you have any pointers to these research papers?
didip 3y ago
System design questions are way more relevant to day-to-day work than leetcode questions. Nothing stops the interviewer from asking you even more relevant system designs questions like: * How would you build a Linux kernel from scratch? * How would you design a common interface for any device drivers (that you are familiar of)?
sanderjd 3y ago
I have designed a few systems, but my issue with the system design interview is that this is not how it works. There is never a blank page in real life like it is in one of these interviews, and the stuff that's actually on the page matters more than the stuff you're "supposed" to say in these sessions. Yes, column-oriented databases absolutely do work better for OLAP use cases, but is it better enough for the specific use case to be worth introducing a new database technology into the organization, or would a new database within the existing managed psql instance be good enough for now? Those detailed organizational questions usually matter more in the first few iterations of systems than "principled" architectural concerns. The useful kind of design is: What is the next best iteration of this system? Maybe with an appendix at the end discussing some ideas for the iteration after that. Sometimes that next iteration is actually the first iteration of the system, in which case you should definitely not be drawing 20 boxes with different components of how it will fit together, you should be looking for the simplest possible thing that could work. One of the fun things at big successful companies is that there are actually a lot of systems that are quite a few iterations in, with a stable baseline of usage properties. With these, it actually can make sense to draw a bunch of boxes with different components targeting different well-known pain points in a way that avoids trading off important existing capabilities. But again, that's exactly the opposite of a blank page, and no amount of digging into the interviewer's toy system design question can get deep enough for that. All of my answers to these questions - which have always been very well-received - have been over-engineered solutions that I'd never actually pursue in a real job. But interviewers aren't really prepared for questions like "what frameworks are already being used and familiar to most teams at the company?" followed by "since we already have familiarity with postgresql+rails+react, we should set up a new non-replicated but periodically backed up database in that database instance, start with a few tables with some reasonable relationships, use activerecord and its migrations, and implement the front-end in react, then we should launch it and see what bottlenecks and pain points arise". I get it, these interviews are trying to see if you have the knowledge and chops to solve those pain points and bottlenecks as they arise, but I'm sorry, "do an up-front design of a huge fancy system" just doesn't answer that question.
- nineplay 3y ago
  I was asked how I would replace an large scale but inadequate logging infrastructure and I stared with "In a way that minimally disrupts everything currently in place for monitoring and alerting". I'm not sure how well that answer played out but it's still the correct answer.
yodsanklai 3y ago
If you apply to a generalist SWE role at FAANG, you're expected to have some knowledge about these things since you're likely to encounter them. I didn't have such experience either, but it's now part of my day to day job. If you apply for a targeted specialized role, you may get a system design question that relates to your domain. If there's a general system design interview, it should have less weight in the process. It's still an indicator on how candidates communicate and think through a design. Plus, these kind of things should be general knowledge for a computer scientist nowadays.
- htrp 3y ago
  > If you apply for a targeted specialized role, you may get a system design question that relates to your domain. If you apply for a specialized targeted role.... you still get the same generalized interview loop
  - yodsanklai 3y ago
    Depends on the company. Sometimes you get extra interviews in addition to the generalist ones.
HumblyTossed 3y ago
I think they're BS also. What you should be looking for is does the person understand how the project(s) they have worked on fit into a larger system. For instance, I have a high level understanding of how the other systems at the Co I work for work. I know what they do, I know what data they ingest and expel. I know how the data I consume is generated. I know how it all fits together and I can talk conversantly about it. I know that when a change request comes in if it affects other teams and I voice out when it does. I think this is what you should look for in most candidates except high level system engineers and jr developers.
opportune 3y ago
I have designed large scale systems, but I often feel most of my system design interviewers haven’t and are much more pedantic/nitpicky than is reasonable. Leetcode is better because most people at least understand the questions they’re asking. Last time I did a system design interview I mentioned database triggers as a way to maintain some kind of data invariant, which flustered my interviewer and I guess made it so their canned follow up questions didn’t work, so they asked if I could think of any other approach (the one they had in mind). I couldn’t and it made the interview very painful.
- dumpster_fire 3y ago
  You just reminded me of the time I was explaining how to do consensus failover, and the interviewer asked me to do distributed single thread design. Great use of everyone's time. The market is full of LARPers who wish they had the chance to design something real, and they will take it out on you.
- danielvaughn 3y ago
  This is a good example of a bad interviewer. Never have a specific concrete answer in mind when asking an open-ended question.
  - kagevf 3y ago
    Or, if you want the candidate to talk about a certain approach, maybe think of some ways you can nudge them in that direction. If something's not at top of mind, some subtle hints from the interviewer could trigger a discussion of what they want to talk about. The interviewer shouldn't be tied to a "script".
  - dekhn 3y ago
    For the really good candidates (IE, ones who can answer basic programming questions and explain how a hash table works), I have an open-ended question which is really an open question in an active field. I really hoped I'd get candidates who would get that far (https://arxiv.org/abs/1309.2975 https://arxiv.org/abs/1309.2975) so that I could finally get some interesting answers I haven't heard before but most candidates I end up interviewing struggle to explain a hash table and what its advantages/disadvantages for counting unique k-mers are.
synergy20 3y ago
I'm also a system programmer developer for the most part(kernel, hardware bring-up, low level firmware, performance tuning of embedded systems) and I got invited a couple times for FAANG interviews,and all of them had this system design interview with nosql column databases and loadbalancers behind nginx proxies of some sort. Problem is that, it's pretty far from my field. Are you going to ask a cloud architect who connects frontend to backend with databases at scale a lot of questions about how to write a pci express network driver in linux kernel with great performance too? I would like to be hired by a specialized hiring team in those big companies instead of going through the general hiring process that you're expected to be a technical God who knows everything at expert level. I rejected all those interview requests.
- deleted 3y ago
  [deleted]
dekhn 3y ago
I built one large scale system in my career (when I worked at Google, I made a folding@home screensaver that used up idle cycles in production). When I built it I ignored 95% of what Google knew about large scale system design because that knowledge was really about building scalable web services and storage systems, while I needed to build a batch scheduler which could handle many millions of tasks simultaneously. We depended on a few core scalable resources available in production (borg, chubby, bigtable, colossus) and tried as hard as possible to avoid spamming them with excessive traffic without adding lots of complex caches and other hard-to-debug systems. In fact, "simplicity" was the primary design goal. The system worked, it scaled, and if I'd followed all the normal Google guidelines, it wouldn't have (because scientific computation and web load balancers differ). Not sure what to take away from that. These days in system design interviews I usually focus on limiting the use cases for the system so that I can architect something that: linear resource consumption for linear workloads over 2-3 orders of magnitude, is simple enough that a small group of engineers can understand the whole system and debug it when it breaks, and not try to optimize for future use cases (clearly documenting the limits of the system) or try to accomodate too many oddball one-off user requests.
scarface_74 3y ago
I agree. I am shocked even though I shouldn’t be at this part of the technical interview for devs and that devs can pass these interviews without demonstrating practical experience. I’ve had interviews and jobs for three smaller companies where I was actually coming in as an architect 2016-2020). But they wanted to know about my real world experience. Luckily my second job out of college way back in 1999 I actually had to manage servers as well as develop so I could both talk theory and practice. From 1999-2012 I was managing infrastructure as part of my job at two jobs. I’ve never interviewed at BigTech for a software engineering position. But I did do a slight pivot and interview (and presently work) in cloud consulting at BigTech specializing in “application modernization” - cloud DevOps + development. Sure I had the one initial phone screen where I had to talk theory about system design. But my entire loop consisted of my walking through my past system design experience - and not all centered around AWS. And yes I can talk intelligently about all of the sections that the page covers including your example of columnar vs row databases. But I wouldn’t expect that from most devs. I was never on call at my last job. We had “follow the sun” support. But our site was only business critical during the day. One of the first things I insisted on with my CTO is that we hire a manage service provider for non business hours support. Sort of related: at most tech companies, the difference between mid and Senior is not coding ability. It’s system design and “scope” and “impact”

Zetice 3y ago

When I see something like this I wonder if there are people out there who actually go chapter-by-chapter from start to finish, spending dozens of hours learning the content a random GitHub repo claimed was important.

mynegation 3y ago
You can always go for the fundamentals and go through DDIA. It is heavy on foundations but does not really give examples of specific real world systems. For that, Alex Xu’s books are probably the most popular.
worksonmine 3y ago
This repo is a very high level collection of topics with short descriptions rather than a learning source. You can skim it a couple of minutes. But if you're curious about a topic, why wouldn't you read a free resource on it if the quality seems good? Are you always this dismissive? How do you learn?
- Zetice 3y ago
  “quality seems good” is the point of contention here; how would you know? You’re either already knowledgeable about this topic and therefore necessarily didn’t learn it from here or you’re not and therefore unqualified to judge its quality at all.
  - worksonmine 3y ago
    That applies to everything we learn. Most of us use previous experience to make an educated guess. Someone just coming into software won't be reading this, but it can be good start to know what one could dig deeper into. Most content on any topic is bullshit, and in tech quickly outdated. What's your gripe with this particular one? I have several similar repos starred for future reference, because why not? Could be useful. You never answered how you learn? Because I assume you still do? Since you have opinions on how others do it maybe you have better approaches you'd like to share?

OnionBlender 3y ago

Can someone suggest a book or resource to learn more about designing a software system but not "scalable" system design that so many interview prep books talk about? I'm especially interested in something that is _not_ web related. I've spent most of my career fixing bugs or taking over apps/systems that someone else started. I really don't know how to design something new. (I mostly work on APIs, Android apps/services, and a bit of game middleware).

vishnugupta 3y ago
Nothing like learning from popular software architecture https://aosabook.org/en/ https://aosabook.org/en/

zitterbewegung 3y ago

Honestly a better resource for this is the following book . It is very easy to read and understand to use what to do where. https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/ https://www.oreilly.com/library/view/designing-data-intensiv...

Maxpowerfreak 3y ago
And for those not too sure about buying yet another O'Reilly book they may or may not read, I recommend checking your local library or school (if a student). They may have the book physically or, even better, have a subscription to O'Reilly's online platform,letting you access any of their books and even some of them in pre-release. I find that I only use a few of my books enough to justify the purchase, and I prefer skimming chapters I find interesting through the platform access I get from my library account.
- esafak 3y ago
  That particular book has a long reservation line in my library network, because it is so popular.
  - saratogacx 3y ago
    My local library has a subscription to O’Reilly books included in being a card-holder. No waiting needed if you are not set on the dead-tree version.

itissid 3y ago

Interviewer: How do you design twitter? Interviewee: Long answer with novice RDBMS choices. Interviewer(implicitly): I will judge you based on how you can "Problem Solve" many-many conundrum. Interviewee: Use Cache, scaling, edge nodes, kafka because.... .... HIRED .... Interviewee spends first 6 months writing a test framework to automatically run tests of deployment of an internal tool that runs python ingestion pipeline. Next 6 months figuring out how to help user add type hints to IDE for pipelines to help catch errors faster and how to roll it out to 100+ ICs in the company. Sadly fails. Never ever writes a line of code to make "twitter", learns a ton on how to work with people. .... Interviews again: "How do you design twitter"?

linza 3y ago
Not sure what the point is here, but ... in my experience as the hiring manager I haven't seen huge success in testing a candidate whether they can do the exact thing they will be doing in the first 6 months (in software engineering). I had better results in asking questions that bias towards a certain set of behavior traits (yes, this has other long-term problems) and not certain skills. I understand that some candidates don't like it, because they are obviously good at something and want to be asked questions in that field. But ... I already know you are good at it because you said it on the CV, so i'm not wasting time with that (yes, if you are a good liar it takes us longer to find out). So your example, working as intended?
- itissid 3y ago
  My point was that the culture of system design interviews is not a good one. Its repeated to infinity at many companies. > I had better results in asking questions that bias towards a certain set of behavior traits I am curious. What do you ask? Like do you ask tech centric questions around Proactiveness, Empathy, Motivation, Conflict Resolution etc?
  - linza 3y ago
    I still ask to design Twitter, but make it clear that it's about the process, and follow ups are driven by where the candidate leads to. The material from OP won't prepare you for systems design interview (witch is also your point?). Only experience will, so we measure experience and leadership. Juniors can apply to systems design openings and will be interviewed, but they end up as generic SWEs if hired. If one thinks one can work on distributed systems coming out of university you will have a hard time interviewing or not like it.

farhanhubble 3y ago

I've seen people cram their heads with all kinds of prep problems like designing an elevator, a parking lot or a social media site and I have never understood the rationale behind such questions. Design isn't something you could do in an hour. Design is also a trial and error thing. Slight changes in data flow might change the entire system architecture.

itissid 3y ago

This is not how people learn. And I always found these books more useful after I have actually worked on a system than before. Do you know why people ask such questions at any company? Its because there is a rubrick at every company and 1/2 the people don't have a clue what else to ask except because they have no other good ideas or because they may have done it themselves and it was a nice problem without realizing what that experience gave them never actually gives the same to the new guy. Skill addition is not additive it is about comparing choices and critically analyzing risks on things you *have already worked on*. Its as much about knowing what won't work i.e. subtractive. If you have never worked on a distributed system and are asked to work on some part of it the reasonable way it happens is: 1. You will be asked to initially to start working on a small part. 2. You will/should have someone to help you if you have no clue. This is called seeking feedback and it is done on google docs where people critique your design and you iterate. 3. You should have the time and sense to read up when you do (1), (2). 4. Maintain mental sanity and find mentors to help you out. Notice that a system design interview prep will help you with exactly *zero* of the steps above. For all other situations where you are forced to do more than the above, run away in the opposite direction. I think if someone just published a Cheat Sheet on the actual Rubrick it would be so thoroughly gamed that it would prove the point that these things are pointless.

zartstrom 3y ago

I'm not sure if I would be prepared to design systems at scale after reading this ... extended glossary. I like the short explanations though.

birdymcbird 3y ago

until recently i was a principal engineer at amazon. so maybe my opinion has some weight. system design interview is more about interviewee asking questions..taking time to understand the problem..ask about product feature or SLA..understand functional and non functional requirement. then its about candidate showing some knowledge set showing they can think and reason behind some immediate coding task. Demonstrate ability to make judgment..simplify where possible..discuss costs and trade offs. this interview not about candidate building some system at scale themself. building and supporting has trials and lessons you only learn by doing and failing, not through interview prep or YouTube videos

CharlieDigital 3y ago
Candidates can't read minds. The best technical interviews I've been on as a interviewee have been those where the expectations are clear. In your example: "We're not expecting you to create Twitter in 15 minutes, but we want to understand how you think about the challenges and key considerations of building a large system like Twitter" Many interviewers fail to provide enough context and that leaves the interpretation of the prompt too wide open. At that point, the interview has failed since whether a candidate can provide an answer that is aligned with the expectations of the interviewer has an element of chance to it.
- mrleinad 3y ago
  > Many interviewers fail to provide enough context and that leaves the interpretation of the prompt too wide open So do clients/customers. What's your point? That an interview to assess whether a developer can elicit requirements should be less hard than dealing with an actual customer?
  - CharlieDigital 3y ago
    An interview with a customer for requirements had a very clear context. An interview for a technical position could focus on either high level design, specific technical aspects, or process and so approach to problem solving. Maybe a bit of each. An interviewer that more clearly defines the context can get better responses.

mkl95 3y ago

I decline non big tech interviews that involve some kind of "system design". I've worked with so-called staff engineers who designed systems with textbook antipatterns (I literally studied them). This idea that you need to regurgitate some bloated mess onto the cloud even to solve even the simplest problems needs to stop.

moandcompany 3y ago

The problem we have today is that the Cracking the Coding Interview/leetcode grinder approach toward interviewing has now overrun the original purpose of system design interviews. It looks like people that lack the understanding of their purpose, or experience to conduct these interviews properly, now run these interviews, and/or often run the show, writing books/articles/etc, which perpetuate the cargo cult behavior of regurgitating patterns described in some influencer book/article/video. This is somewhat associated with the title-inflation we've seen in our industry where "senior" and "staff" level engineers today seem to have much less experience than persons with the same titles a decade ago, and the people running these interviews are today's "senior and staff" engineers, or managers with very shallow engineering experience. Systems design interviews were meant to be non-grindable, open-ended interview questions which allow a very experienced engineer as interviewer to understand the interviewee's experience and approach in solving systems-level problems. The original purpose of the systems design question in engineering interviews was to: 1) Give the candidate an open-ended question that is not meant to be memorizable or fully solved within the allotted time. 2) Give the candidate an opportunity to shine -- expose and demonstrate areas where they have particular depth from their past work experience Systems Design interviews employ the judgment of the highly experienced engineer as interviewer, rather than checking for a specific answer. As an interviewee, the first phase of a systems design interview is really about asking questions to better understand context/constraints/considerations/etc before formulating solutions. The second phase of a systems design interview is about communicating your understanding of the "whys" and clarifying/refining those understandings with your interviewer. Ironically, we also have a post on Hacker News today ("The hardest part of building software is not coding, it's requirements") [https://news.ycombinator.com/item?id=36597709 https://news.ycombinator.com/item?id=36597709] which speaks to the point that systems design interviews really being a test to see how a candidate elicits requirements and considers trade-offs, when developing solutions for problems.

Thaxll 3y ago

There are things in that doc that are completly wrong and ridiculous such as: https://github.com/karanpratapsingh/system-design#streaming-a-video https://github.com/karanpratapsingh/system-design#streaming-... It reminds me a Youtube video on how to create a Netflix clone, for streaming they were creating a Java class for each frame of a video.

VirusNewbie 3y ago

Has...the person who wrote this designed large scale systems?

unmole 3y ago
Doesn't matter. The person conducting the interview has probably never done that. And the job itself probably won't involve large scale system design.
- VirusNewbie 3y ago
  My last three jobs involved large scale systems. I've designed numerous ones. The last company I was at was a small startup and still did billions of transactions, a non trivial amount that certainly can't be handled by a single machine.