7 ms·

GitHub – system-design-primer: Learn how to design large-scale systems

112 points by msolujic 3y ago

alok-g 3y ago
Meta comment: We have come to very narrow definition of a 'system' when using terms like system design, system architect, etc., which here seems to mean a distributed computer system. That, as compared to a more open-ended definition found on, say, Wikipedia [1]. What about biological systems like a cell or a human body, a road system, a sound system, ... [1] https://en.m.wikipedia.org/wiki/System https://en.m.wikipedia.org/wiki/System
- qznc 3y ago
  Working in the automotive sector, I find it even worse. Sometimes “system” means an ECU (e.g. a radar sensor), sometimes the whole car, sometimes even includes infrastructure (cloud, other cars, etc). Sometimes it is just a software component within an ECU (e.g. the operating system). If I could I would taboo the term. It conveys practically zero information at this point.
- charcircuit 3y ago
  >which here seems to mean a distributed computer system. No, it can also be a design between multiple services on the same machine. Or it can be a design between multiple subsystems of a single application.
bananapub 3y ago
the Google NALSD guides are also worth reading for concrete-ising these sort of designs, too. there's some stuff in the SRE Workbook: https://sre.google/workbook/non-abstract-design/ https://sre.google/workbook/non-abstract-design/ and much more online.
petepete 3y ago
Slightly confusingly, this is nothing to do with GitHub's Primer design system. https://primer.style/ https://primer.style/
- esafak 3y ago
  I think the "large scale" qualifier should dispel confusion.
rollulus 3y ago
This thing has 233k stars. I dare to say, without evidence, that the vast majority of devs does not know what e.g. the CAP theorem is——and they’ll never need it. The vast majority of “systems” are small-scale. Who’s this massive audience?
- brunooliv 3y ago
  People who live for short tenured stays at FAANG companies and optimize their time and skills exclusively for interviews?
- robervin 3y ago
  Still seems quite high, but I imagine a lot of tech founders reach for tools like this if they've not been tasked with designing full systems before.
- mediascreen 3y ago
  I would guess that it's probably aspirational to some degree. People hope to work with larger systems. Personally I think it looks interesting and I work at a small web agency. We usually only work with a few parts of these kinds of systems, but occasionally we have clients like a large sports ticket vendor, where a few more of these patterns are helpful.
- nunez 3y ago
  Job hoppers trying for staff+ roles at FAANGs or VC-backed mid-stage startups. Also, to add to your comment about CAP. knowing CAP (consistency, availability, partition tolerance, choose two) can be useful when choosing which database to use for a production system, I.e. can you get away with a fancy distributed gossip database or do you need the robustness of a rdbms. 99.9% of the time a RDBMS is fine, but if you're vying for a senior staff role, you need to sound like you can talk about the 0.01% of the time where Riak or something like that makes sense as a primary data store. That's why this guide mentions it.
lifeisstillgood 3y ago
I fundamentally distrust these sort of guides. It was brought home reading something on the google SRE book - basically they talked about how google started with two data centres and a bunch of python scripts as automation. Did they do "up-front" design? Of course. Did they get it horribly wrong at times ? of course. but the large working system evolved from a small working system. Thinking you can cunningly plan a future system to hang together like some giant zipper with pieces as yet to be built pulling together perfectly is ludicrous.
- rajaman0 3y ago
  You are comparing apples and oranges. These interviews are for engineers interviewing at companies which have scale ; not for a 0-1 growth of a product like your google example. When you're launching products at scale, you absolutely need to design large systems that are durable and can operate at said scale. And when I'm hiring, I need to filter for folks who are able to rationalize at that scale.
  - lifeisstillgood 3y ago
    Well as you are the FAANG hiring manager I expect I am going to lose the argument but I wonder if my oranges are more like apples than you suspect. I see there being two options here apples: please take the existing set of distributed components that evolved in tandem at this company and design a new application on top of and using those components oranges: please start from scratch and explain how you would build a set of components that will work as well as those in 1. Apples is what we mostly look for - in any org we have a scale and ecosystem and we don't want to throw it all out. If it must evolve it evolves in tandem with rest of the ecosystem. So if we have some scalable data layer now that has chosen one side of CP/AP and then we come along and say actually throw that out We need full concistency all the time for this HR app, we are going to build our own different CP data layer globally, hold my beer, then ... The oranges part is fine - it tells us if someone has actually understood the principles behind apples. But honestly it's a fake out of we think oranges can actually be done. And this sort of stuff gives the impression you can design the application solution by also designing the ecosystem at the same time. That's the bit Inwant to emphasise - you build the platform then design the application on the quirks of what you built And anyone who has not used the apples components won't understand their quirks and can come unstuck when the well known but non obvious behaviour strikes. I am being too vague here. I feel there is an interesting set of discussions to come out of it - Inwill reread the article
nunez 3y ago
This is super helpful as a primer (especially the bits about knowing which kinds of infra go where and doing capacity planning BEFORE architecting anything) but is not a replacement for actually having done it somewhere, since there are so many tools and tricks that are difficult to convey in a single guide. I say this because this guide is often used by engineers trying to job hop their way into staff+ roles without having the requisite experience to back that up. Put another way, this guide will help you answer "design Netflix," (I.e. narrow the scope, capacity plan, diagram the ideal path, replace with infrastructure, discuss tradeoffs) but will NOT help you answer "tell me about a time where you designed a large-scale system and how you would've made it better given the chance"
marginalia_nu 3y ago
I had a look at the "design a crawler" part, since that's something I have experience with. It makes some pretty sketchy assumptions and weird design choices. First in the back-of-the-envelope calculation it overestimates the size of a document by about two orders of magnitude. It's closer to 7.5 KB than 500 KB on average, uncompressed. 500 KB is closer to the size of Homer's Odyssey, which I will state is much longer than most websites. I'm willing to give them the benefit of the doubt though, maybe I'm misunderstanding what these 500 KB is supposed to consist of. I also don't think the crawl frontier design would work at the sort of scales they imagine. This is a design that will get you to maybe 10 million documents. Apart from the profoundly questionable choice of using a nosql database for this, apart from the race condition in the crawling algorithm, Redis can only store 4 billion (2^32-1) items in a set, and let me tell you, crawl frontiers grow very quickly when you design them like this one has been designed. By the time you're indexing a billion documents, your crawl frontier will be of many orders of magnitude more. The biggest mistake is to think too link-oriented. You probably want to retrieve links one domain at a time as this permits connection pooling (otherwise your biggest crawling bottleneck will be waiting for the remote end to acknowledge the connection's closure), helps with deduplication and means you don't need to re-fetch robots.txt for every single document you retrieve. It also solves many problems limiting how much you fetch from a single domain, since you can just set a hard time and link limit. You don't have to re-fetch every document on that domain when you refresh it, just get the new ones and sample the existing ones. This also means you don't need a billion-entry priority queue, you can get away with a manageable queue of a few million instead.
- deleted 3y ago
  [deleted]
- supriyo-biswas 3y ago
  Last-Modified/Etags are your friends for detecting modifications. Having said that, link orientation may not matter if they have to run a headless browser rendering such performance optimizations moot.
  - marginalia_nu 3y ago
lokesh1729 3y ago
This guide gives the overview of breadth of concepts. This is a good starting point. There’s one more github repo called “awesome-scalability” which has the good blog posts by concept.
xwowsersx 3y ago
I don't think "GitHub" should lead in the title here. It's a bit misleading because it makes one think this is going to be a blog series from GitHub or something like that and, in any event isn't in line with standard HN practice. Only thing I could see making sense is to put "(GitHub)" at the end of the title, but even that I'm not sure is necessary. @dang?

112 points by msolujic 3y ago

Meta comment: We have come to very narrow definition of a 'system' when using terms like system design, system architect, etc., which here seems to mean a distributed computer system. That, as compared to a more open-ended definition found on, say, Wikipedia [1]. What about biological systems like a cell or a human body, a road system, a sound system, ... [1] https://en.m.wikipedia.org/wiki/System https://en.m.wikipedia.org/wiki/System

qznc 3y ago
Working in the automotive sector, I find it even worse. Sometimes “system” means an ECU (e.g. a radar sensor), sometimes the whole car, sometimes even includes infrastructure (cloud, other cars, etc). Sometimes it is just a software component within an ECU (e.g. the operating system). If I could I would taboo the term. It conveys practically zero information at this point.
charcircuit 3y ago
>which here seems to mean a distributed computer system. No, it can also be a design between multiple services on the same machine. Or it can be a design between multiple subsystems of a single application.

bananapub 3y ago

the Google NALSD guides are also worth reading for concrete-ising these sort of designs, too. there's some stuff in the SRE Workbook: https://sre.google/workbook/non-abstract-design/ https://sre.google/workbook/non-abstract-design/ and much more online.

petepete 3y ago

Slightly confusingly, this is nothing to do with GitHub's Primer design system. https://primer.style/ https://primer.style/

esafak 3y ago
I think the "large scale" qualifier should dispel confusion.

rollulus 3y ago

This thing has 233k stars. I dare to say, without evidence, that the vast majority of devs does not know what e.g. the CAP theorem is——and they’ll never need it. The vast majority of “systems” are small-scale. Who’s this massive audience?

brunooliv 3y ago
People who live for short tenured stays at FAANG companies and optimize their time and skills exclusively for interviews?
robervin 3y ago
Still seems quite high, but I imagine a lot of tech founders reach for tools like this if they've not been tasked with designing full systems before.
mediascreen 3y ago
I would guess that it's probably aspirational to some degree. People hope to work with larger systems. Personally I think it looks interesting and I work at a small web agency. We usually only work with a few parts of these kinds of systems, but occasionally we have clients like a large sports ticket vendor, where a few more of these patterns are helpful.
nunez 3y ago
Job hoppers trying for staff+ roles at FAANGs or VC-backed mid-stage startups. Also, to add to your comment about CAP. knowing CAP (consistency, availability, partition tolerance, choose two) can be useful when choosing which database to use for a production system, I.e. can you get away with a fancy distributed gossip database or do you need the robustness of a rdbms. 99.9% of the time a RDBMS is fine, but if you're vying for a senior staff role, you need to sound like you can talk about the 0.01% of the time where Riak or something like that makes sense as a primary data store. That's why this guide mentions it.

lifeisstillgood 3y ago

I fundamentally distrust these sort of guides. It was brought home reading something on the google SRE book - basically they talked about how google started with two data centres and a bunch of python scripts as automation. Did they do "up-front" design? Of course. Did they get it horribly wrong at times ? of course. but the large working system evolved from a small working system. Thinking you can cunningly plan a future system to hang together like some giant zipper with pieces as yet to be built pulling together perfectly is ludicrous.

rajaman0 3y ago
You are comparing apples and oranges. These interviews are for engineers interviewing at companies which have scale ; not for a 0-1 growth of a product like your google example. When you're launching products at scale, you absolutely need to design large systems that are durable and can operate at said scale. And when I'm hiring, I need to filter for folks who are able to rationalize at that scale.
- lifeisstillgood 3y ago
  Well as you are the FAANG hiring manager I expect I am going to lose the argument but I wonder if my oranges are more like apples than you suspect. I see there being two options here apples: please take the existing set of distributed components that evolved in tandem at this company and design a new application on top of and using those components oranges: please start from scratch and explain how you would build a set of components that will work as well as those in 1. Apples is what we mostly look for - in any org we have a scale and ecosystem and we don't want to throw it all out. If it must evolve it evolves in tandem with rest of the ecosystem. So if we have some scalable data layer now that has chosen one side of CP/AP and then we come along and say actually throw that out We need full concistency all the time for this HR app, we are going to build our own different CP data layer globally, hold my beer, then ... The oranges part is fine - it tells us if someone has actually understood the principles behind apples. But honestly it's a fake out of we think oranges can actually be done. And this sort of stuff gives the impression you can design the application solution by also designing the ecosystem at the same time. That's the bit Inwant to emphasise - you build the platform then design the application on the quirks of what you built And anyone who has not used the apples components won't understand their quirks and can come unstuck when the well known but non obvious behaviour strikes. I am being too vague here. I feel there is an interesting set of discussions to come out of it - Inwill reread the article

nunez 3y ago

This is super helpful as a primer (especially the bits about knowing which kinds of infra go where and doing capacity planning BEFORE architecting anything) but is not a replacement for actually having done it somewhere, since there are so many tools and tricks that are difficult to convey in a single guide. I say this because this guide is often used by engineers trying to job hop their way into staff+ roles without having the requisite experience to back that up. Put another way, this guide will help you answer "design Netflix," (I.e. narrow the scope, capacity plan, diagram the ideal path, replace with infrastructure, discuss tradeoffs) but will NOT help you answer "tell me about a time where you designed a large-scale system and how you would've made it better given the chance"

marginalia_nu 3y ago

I had a look at the "design a crawler" part, since that's something I have experience with. It makes some pretty sketchy assumptions and weird design choices. First in the back-of-the-envelope calculation it overestimates the size of a document by about two orders of magnitude. It's closer to 7.5 KB than 500 KB on average, uncompressed. 500 KB is closer to the size of Homer's Odyssey, which I will state is much longer than most websites. I'm willing to give them the benefit of the doubt though, maybe I'm misunderstanding what these 500 KB is supposed to consist of. I also don't think the crawl frontier design would work at the sort of scales they imagine. This is a design that will get you to maybe 10 million documents. Apart from the profoundly questionable choice of using a nosql database for this, apart from the race condition in the crawling algorithm, Redis can only store 4 billion (2^32-1) items in a set, and let me tell you, crawl frontiers grow very quickly when you design them like this one has been designed. By the time you're indexing a billion documents, your crawl frontier will be of many orders of magnitude more. The biggest mistake is to think too link-oriented. You probably want to retrieve links one domain at a time as this permits connection pooling (otherwise your biggest crawling bottleneck will be waiting for the remote end to acknowledge the connection's closure), helps with deduplication and means you don't need to re-fetch robots.txt for every single document you retrieve. It also solves many problems limiting how much you fetch from a single domain, since you can just set a hard time and link limit. You don't have to re-fetch every document on that domain when you refresh it, just get the new ones and sample the existing ones. This also means you don't need a billion-entry priority queue, you can get away with a manageable queue of a few million instead.

deleted 3y ago
[deleted]
supriyo-biswas 3y ago
Last-Modified/Etags are your friends for detecting modifications. Having said that, link orientation may not matter if they have to run a headless browser rendering such performance optimizations moot.
- marginalia_nu 3y ago

lokesh1729 3y ago

This guide gives the overview of breadth of concepts. This is a good starting point. There’s one more github repo called “awesome-scalability” which has the good blog posts by concept.

xwowsersx 3y ago

I don't think "GitHub" should lead in the title here. It's a bit misleading because it makes one think this is going to be a blog series from GitHub or something like that and, in any event isn't in line with standard HN practice. Only thing I could see making sense is to put "(GitHub)" at the end of the title, but even that I'm not sure is necessary. @dang?