12 ms·

The System Design Primer

508 points by donnemartin 10y ago

abraae 10y ago
Nice concept! A missing area is identity management. Most likely this should be separated from your system (e.g. don't have a table somewhere with username, password in it). In consumer facing systems, OpenID Connect (better) as practiced by Google, OAuth is used by most others. In enterprise software, SAML is the common parlance. That leads naturally to questions about API authorization (are API calls made on behalf of system users? If not, start probing further). Always enlightening to start asking questions about identity management very early on in designing systems.
- donnemartin 10y ago
  Good point, the security section could be beefed up some more. Pull requests are welcome :)
sna1l 10y ago
Great idea, thanks for doing this!
crudbug 10y ago
Great resource. Which tool is used for creating diagrams ?
- donnemartin 10y ago
  Thanks! I'm using OmniGraffle for the diagrams.
deleted 10y ago
[deleted]
phamilton 10y ago
I very much want to hear the words "failure isolation" during a systems design interview. Usually as the answer to "Why did you break that functionality out into a separate service?". The answer should involve "independent scaling" and "failure isolation".
- mfukar 10y ago
  Great, now anyone knows how to temporarily satisfy you while delivering nothing of what you actually want in the long run.
  - phamilton 10y ago
    Surely, you understand I'm not literally only looking for buzzwords. I would certainly ask deeper questions on what they are trying to accomplish and why.
    - mfukar 10y ago
      I'd heard the bar for humour had been lowered, but I didn't know how low.
- zaptheimpaler 10y ago
  I really don't buy this, its mostly a cultural fad. You can design your system as multiple services interacting via a strict API even in a monolith. Failure isolation is a pattern of how to handle errors, whether they come across as a 40x HTTP status code, a JSON field called "error" or a method call throwing an exception. And a single system enables you to work with simple method calls calling across services instead of all the complication SOA can bring with signaling errors/serialization over a network layer. Not to mention all the operational overhead that comes with managing more hosts and more services.
  - phamilton 10y ago
    You make the assumption that failures will be covered by defensive programming techniques. Unexpected failures will take down a monolith. One bad db query can deadlock all consumers. The greater the separation between components, the more easily you can isolate all failures, not just expected ones.
- Waterluvian 10y ago
  "Because I can hoard access to it over the other teams and make myself indispensable."
pfista 10y ago
This looks like a great guide, thanks! Makes me wonder how effective things like Google's app engine are in autoscaling your web apps. "Serverless" code seems too good to be true.
reacharavindh 10y ago
Haven't read the entire guide yet. But, I hope it has a few lines somewhere about over-engineering a solution. Yes, fault-tolerance, Asynchronousism, individual scalability are virtues you want, but not for a super simple problem that needs functional work. I've been in so many discussions with people that talk about all these virtues and speend too little a time on making that core function do what it is supposed to do.
arslanahmad 10y ago
I recently gave interviews and did my preparation from https://www.educative.io/collection/5668639101419520/5649050225344512 https://www.educative.io/collection/5668639101419520/5649050.... It was pretty useful.
chid 10y ago
This looks awesome, is there a collection of these around other fields?
oferzelig 10y ago
Beautiful document. Clearly written and clearly drawn.
movedx 10y ago
Brilliant work. I may convert this into MkDocs "formatted" project using the Materials theme. I've done the same thing for the Open Guide to AWS which I'm still working on. It vastly improves the readability and accessible of the information.
- donnemartin 10y ago
  Interesting, it does look great! I'm curious, how do we keep the MkDocs project updated as the guide evolves?
- mu_killnine 10y ago
  That would be great, never seen MkDocs but looks like it might be an improvement for jumping between topics.
- Dangeranger 10y ago
  Can you please post a link to your Open Guide to AWS if possible?
  - movedx 10y ago
    Most of the sections are in DRAFT mode as I move them away from just being a list of bullet points to something more readable and formatted. http://opsfactory.com.au/wisdom-aws-guide/ http://opsfactory.com.au/wisdom-aws-guide/
  - danenania 10y ago
    https://github.com/open-guides/og-aws https://github.com/open-guides/og-aws
javajosh 10y ago
I wonder what this would look like for Erlang system designers.
- cholantesh 10y ago
  I am totally ignorant to Erlang (though it is in my top 5 of languages to learn); why do you think it would be different?
  - javajosh 10y ago
    I am only learning Erlang now (through the futurelearn course[1], posted here, about a month ago now). One of the reasons it interests me is that it was designed from the start to support high-availability concurrent, distributed processes. It's a functional, dynamic language meaning that you can reprogram a system at runtime if you want (see: gen_server). Like I said, I'm only starting and I don't know how real Erlang systems are built. However, I suspect that they tend to eschew the orthodoxy of treating a relational database as a single source of truth, with stateless app servers (these two features are the core of all the systems in the OP's thing) and embrace distributed, redundant statefulness. If this can be done (without becoming impossible to reason about) I suspect it represents an optimal server system, in terms of resource usage, availability, and probably even in performance in a world where most organizations' datasets can fit entirely in RAM, from when they are a twinkle in someone's eye to their eventual dissolution. I should stress that "questioning orthodoxy" is something of a hobby, which probably biases me. [1] https://www.futurelearn.com/courses/functional-programming-erlang https://www.futurelearn.com/courses/functional-programming-e... [2] http://learnyousomeerlang.com http://learnyousomeerlang.com keeps coming up as a good learning resource.
0x54MUR41 10y ago
That's amazing. Thank you for creating this. It's very useful for preparing an interview in system design. I think you miss "Show HN" on your post.
contingencies 10y ago
High level observations: 1. Business level constraints (time, human, fiscal and other resources, stakeholders) trump technical constraints every time. Identifying these should be step zero in any design process. 2. A business-level risk model assists with appropriate design with respect to both security and availability and should ultimately drive component selection. 3. Content seems very much focused on public IP services provided through multiple networked subsystems. While this is a very popular category of modern systems design, not all systems fall in to this category (eg. embedded), and even if they do many complex systems are internal, and public-facing interfaces are partly shielded/outsourced (Cloudflare, AWS, etc.). 4. Existing depth in areas such as database replication could perhaps be grouped in a generic fashion as examples of fault tolerance and failure / issue-mitigation strategies. 5. Asynchronicity and communication could be grouped together under architectural paradigms (eg. state, consistency and recovery models), since they tend to define at least subsystem-local architectural paradigms. (Ever tried restoring a huge RDBMS backup or performing a backup between major RDBMS versions where downtime is a concern? What about debugging interactions between numerous message queues, or disparate views of a shared database (eg. blockchain, split-capable orchestration systems) with supposed eventual consistency?) 6. Legal and regulatory considerations are often very powerful architectural concerns. In a multinational system with components owned by disparate legal entities in different jurisdictions, potential regulatory ingress (eg. halt/seize/shut down national operations) can become a significant consideration. 7. The new/greenfield systems design perspective is a valid and common one. However, equally commonly, established organizations' subsystems are (re-)designed/upgraded, and in this case system interfaces may be internal or otherwise highly distinct from public service design. Often these sorts of projects are harder because of downtime concerns, migration complexity and organizational/technical inertia.
- vacri 10y ago
  > Business level constraints (time, human, fiscal and other resources, stakeholders) trump technical constraints every time. Sometimes I feel it's a pity that HN doesn't support bold text and increased font sizes.
- donnemartin 10y ago
  Thanks for the feedback! I'll see if I can work in some of these suggestions. Pull requests are welcome :)
alkonaut 10y ago
Does "system" here mean "system of internet services"? I'm designing large systems and hope to learn more - but none of my systems have servers. Anywhere.
- dwringer 10y ago
  Indeed, I think the headline is far too general. I came here hoping to find something from the perspective of systems theory or cybernetics.

508 points by donnemartin 10y ago

Nice concept! A missing area is identity management. Most likely this should be separated from your system (e.g. don't have a table somewhere with username, password in it). In consumer facing systems, OpenID Connect (better) as practiced by Google, OAuth is used by most others. In enterprise software, SAML is the common parlance. That leads naturally to questions about API authorization (are API calls made on behalf of system users? If not, start probing further). Always enlightening to start asking questions about identity management very early on in designing systems.

donnemartin 10y ago
Good point, the security section could be beefed up some more. Pull requests are welcome :)

sna1l 10y ago

Great idea, thanks for doing this!

crudbug 10y ago

Great resource. Which tool is used for creating diagrams ?

donnemartin 10y ago
Thanks! I'm using OmniGraffle for the diagrams.

deleted 10y ago

[deleted]

phamilton 10y ago

I very much want to hear the words "failure isolation" during a systems design interview. Usually as the answer to "Why did you break that functionality out into a separate service?". The answer should involve "independent scaling" and "failure isolation".

mfukar 10y ago
Great, now anyone knows how to temporarily satisfy you while delivering nothing of what you actually want in the long run.
- phamilton 10y ago
  Surely, you understand I'm not literally only looking for buzzwords. I would certainly ask deeper questions on what they are trying to accomplish and why.
  - mfukar 10y ago
    I'd heard the bar for humour had been lowered, but I didn't know how low.
zaptheimpaler 10y ago
I really don't buy this, its mostly a cultural fad. You can design your system as multiple services interacting via a strict API even in a monolith. Failure isolation is a pattern of how to handle errors, whether they come across as a 40x HTTP status code, a JSON field called "error" or a method call throwing an exception. And a single system enables you to work with simple method calls calling across services instead of all the complication SOA can bring with signaling errors/serialization over a network layer. Not to mention all the operational overhead that comes with managing more hosts and more services.
- phamilton 10y ago
  You make the assumption that failures will be covered by defensive programming techniques. Unexpected failures will take down a monolith. One bad db query can deadlock all consumers. The greater the separation between components, the more easily you can isolate all failures, not just expected ones.
Waterluvian 10y ago
"Because I can hoard access to it over the other teams and make myself indispensable."

pfista 10y ago

This looks like a great guide, thanks! Makes me wonder how effective things like Google's app engine are in autoscaling your web apps. "Serverless" code seems too good to be true.

reacharavindh 10y ago

Haven't read the entire guide yet. But, I hope it has a few lines somewhere about over-engineering a solution. Yes, fault-tolerance, Asynchronousism, individual scalability are virtues you want, but not for a super simple problem that needs functional work. I've been in so many discussions with people that talk about all these virtues and speend too little a time on making that core function do what it is supposed to do.

arslanahmad 10y ago

I recently gave interviews and did my preparation from https://www.educative.io/collection/5668639101419520/5649050225344512 https://www.educative.io/collection/5668639101419520/5649050.... It was pretty useful.

chid 10y ago

This looks awesome, is there a collection of these around other fields?

oferzelig 10y ago

Beautiful document. Clearly written and clearly drawn.

movedx 10y ago

Brilliant work. I may convert this into MkDocs "formatted" project using the Materials theme. I've done the same thing for the Open Guide to AWS which I'm still working on. It vastly improves the readability and accessible of the information.

donnemartin 10y ago
Interesting, it does look great! I'm curious, how do we keep the MkDocs project updated as the guide evolves?
mu_killnine 10y ago
That would be great, never seen MkDocs but looks like it might be an improvement for jumping between topics.
Dangeranger 10y ago
Can you please post a link to your Open Guide to AWS if possible?
- movedx 10y ago
  Most of the sections are in DRAFT mode as I move them away from just being a list of bullet points to something more readable and formatted. http://opsfactory.com.au/wisdom-aws-guide/ http://opsfactory.com.au/wisdom-aws-guide/
- danenania 10y ago
  https://github.com/open-guides/og-aws https://github.com/open-guides/og-aws

javajosh 10y ago

I wonder what this would look like for Erlang system designers.

cholantesh 10y ago
I am totally ignorant to Erlang (though it is in my top 5 of languages to learn); why do you think it would be different?
- javajosh 10y ago
  I am only learning Erlang now (through the futurelearn course[1], posted here, about a month ago now). One of the reasons it interests me is that it was designed from the start to support high-availability concurrent, distributed processes. It's a functional, dynamic language meaning that you can reprogram a system at runtime if you want (see: gen_server). Like I said, I'm only starting and I don't know how real Erlang systems are built. However, I suspect that they tend to eschew the orthodoxy of treating a relational database as a single source of truth, with stateless app servers (these two features are the core of all the systems in the OP's thing) and embrace distributed, redundant statefulness. If this can be done (without becoming impossible to reason about) I suspect it represents an optimal server system, in terms of resource usage, availability, and probably even in performance in a world where most organizations' datasets can fit entirely in RAM, from when they are a twinkle in someone's eye to their eventual dissolution. I should stress that "questioning orthodoxy" is something of a hobby, which probably biases me. [1] https://www.futurelearn.com/courses/functional-programming-erlang https://www.futurelearn.com/courses/functional-programming-e... [2] http://learnyousomeerlang.com http://learnyousomeerlang.com keeps coming up as a good learning resource.

0x54MUR41 10y ago

That's amazing. Thank you for creating this. It's very useful for preparing an interview in system design. I think you miss "Show HN" on your post.

contingencies 10y ago

High level observations: 1. Business level constraints (time, human, fiscal and other resources, stakeholders) trump technical constraints every time. Identifying these should be step zero in any design process. 2. A business-level risk model assists with appropriate design with respect to both security and availability and should ultimately drive component selection. 3. Content seems very much focused on public IP services provided through multiple networked subsystems. While this is a very popular category of modern systems design, not all systems fall in to this category (eg. embedded), and even if they do many complex systems are internal, and public-facing interfaces are partly shielded/outsourced (Cloudflare, AWS, etc.). 4. Existing depth in areas such as database replication could perhaps be grouped in a generic fashion as examples of fault tolerance and failure / issue-mitigation strategies. 5. Asynchronicity and communication could be grouped together under architectural paradigms (eg. state, consistency and recovery models), since they tend to define at least subsystem-local architectural paradigms. (Ever tried restoring a huge RDBMS backup or performing a backup between major RDBMS versions where downtime is a concern? What about debugging interactions between numerous message queues, or disparate views of a shared database (eg. blockchain, split-capable orchestration systems) with supposed eventual consistency?) 6. Legal and regulatory considerations are often very powerful architectural concerns. In a multinational system with components owned by disparate legal entities in different jurisdictions, potential regulatory ingress (eg. halt/seize/shut down national operations) can become a significant consideration. 7. The new/greenfield systems design perspective is a valid and common one. However, equally commonly, established organizations' subsystems are (re-)designed/upgraded, and in this case system interfaces may be internal or otherwise highly distinct from public service design. Often these sorts of projects are harder because of downtime concerns, migration complexity and organizational/technical inertia.

vacri 10y ago
> Business level constraints (time, human, fiscal and other resources, stakeholders) trump technical constraints every time. Sometimes I feel it's a pity that HN doesn't support bold text and increased font sizes.
donnemartin 10y ago
Thanks for the feedback! I'll see if I can work in some of these suggestions. Pull requests are welcome :)

alkonaut 10y ago

Does "system" here mean "system of internet services"? I'm designing large systems and hope to learn more - but none of my systems have servers. Anywhere.

dwringer 10y ago
Indeed, I think the headline is far too general. I came here hoping to find something from the perspective of systems theory or cybernetics.