7 ms·

System Design Primer: Learn how to design large-scale systems

170 points by jaboutboul 5y ago

dang 5y ago
Past related threads: Learn how to design large-scale systems - https://news.ycombinator.com/item?id=17522362 https://news.ycombinator.com/item?id=17522362 - July 2018 (185 comments) The System Design Primer - https://news.ycombinator.com/item?id=13823979 https://news.ycombinator.com/item?id=13823979 - March 2017 (57 comments)
- sparky4 5y ago
  Thank you
- fivea 5y ago
  HN already provides a link to past threads. Check out the "past" link right next to the submission.
  - nyc111 5y ago
    Thanks, but "past" link does not show past discussions of the posted article, just older states of HN.
    - fivea 5y ago
      > Thanks, but "past" link does not show past discussions of the posted article, just older states of HN. No, not really. It shows a list of past submissions, which of course include the corresponding discussions. Take a look: https://hn.algolia.com/?query=System%20Design%20Primer%3A%20Learn%20how%20to%20design%20large-scale%20systems&type=story&dateRange=all&sort=byDate&storyText=false&prefix&page=0 https://hn.algolia.com/?query=System%20Design%20Primer%3A%20...
      - macintux 5y ago
        There are two completely different “past” links on the page, hence the confusion. One shows previous days, other previous posts.
        agumonkey 5y ago
        I never realized there was an article scoped 'past' link.
      - nyc111 5y ago
        That's the search box at the bottom of the page. I get that "hn.algolia.com" link when I search for the title of the article. Otherwise I don't see a "past" link. Maybe there is, I don't know.
  - smarx007 5y ago
    You understand that you just lectured the moderator of HN on how to use the website? HN already provides a link to profile bios. Check out the %username% link on top of the comment. /s
    - mynegation 5y ago
      TBH Dang’s bio has only implicit connection to The fact that he is a moderator here, but most people of course know.
  - dang 5y ago
    Yup and I'm glad you know that because I wish it got more attention! (I'm a mod here, sorry that wasn't clear.) The reason I post those links to past related threads about specific stories is that many readers like to look up old comments on the same topics. Often there are surprises and gems in there. Oh and there's no implication that the story shouldn't be reposted—reposts are fine after a year or so (https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html). Btw, the 'past' link in the top bar shows you (more or less) what the front page looked like on a particular day. To find threads about specific past stories, it's best to use HN Search at the bottom of the page.
beaconstudios 5y ago
I thought this was going to be about system engineering, but it's just about web scale distributed computing. Disappointing, because I'm digging into systems engineering at the moment and got a bunch of books on it for Christmas, plus thumbing through the SEBoK wiki [1] and associated content. 1: https://www.sebokwiki.org/wiki/Guide_to_the_Systems_Engineering_Body_of_Knowledge_(SEBoK) https://www.sebokwiki.org/wiki/Guide_to_the_Systems_Engineer...
- stewvsshark 5y ago
  Any books you’d recommend on the subject?
  - beaconstudios 5y ago
    Not yet, though I'd love to do recommend some if I could - as I say, I've only just started looking into it fairly recently! If you're interested, drop me an email (in my bio) and I'll let you know, once I've read my new books, if they're any good.
    - MadSudaca 5y ago
      Could you list the books you bought? Thanks!
      - beaconstudios 5y ago
        Sure thing (though it was my lovely wife that bought them, not me!): - Systems Engineering Demystified by Jon Holt - Essential Architecture and Principles of Systems Engineering by C. E. Dickerson & Siyuan Ji - System Design & Management: An Introduction to Systems Engineering by Bob Parkinson On a somewhat related note I also got Difference & Repetition by Deleuze and Process and Reality by Alfred North Whitehead, but they're more philosophical than practical.
        MadSudaca 5y ago
louwrentius 5y ago
I think this is very valuable and helpful. What really matters I think is when you really need to go this route? Wat what kind of traffic / transactions/s level do you need to (re)design for this? Or even, by what metric would you opt for this solution? I mean, just buying a bigger instance will get you a long way and if you offset that cost to development time / effort / lost opportunity, how will the scale be tipped?
- rmbyrro 5y ago
  I think it depends a lot on the application use cases and requirements. Things like: Will there be spikes in demand 50x higher than baseline usage? I.e. how fast and efficiently the system needs to scale up? What level of reliability is needed? I.e. is it really a big problem if the system goes down for an hour a handful of times throughout a given time period?
- whoevercares 5y ago
  Nah people need to go this route just to pass the interview and get the $600k job by answering the question made up for the interview - it always need to scale globally with at least 99.99 availability and development time /other non-tech dimensions almost always not in scope
zerr 5y ago
It seems like all such resources are about web-scale distributed systems and the content/methods are quite standardized. What about other large-scale systems? E.g. desktop applications - for instance, how to design a large-scale desktop system such as Adobe Photoshop or MS Office.
- beaconstudios 5y ago
  Systems engineering is the discipline that got us to the moon, and builds advanced technology like military hardware. For other large scale software projects, I think they tend to use enterprise software architecture patterns like service-oriented architecture (basically microservices) and ESB (basically kafka).
- codr7 5y ago
  I doubt you could find two that look the same on the inside, too many talented coders are needed to get to that point; but there's some good stuff by Sean Parent of Adobe on YT.
- volkk 5y ago
  this resource exists purely for people to practice for systems design interviews. or at least, that's 99% of the use cases of this repo
- theflork 5y ago
  Architecture of open source applications, which has a free book version available on their site, has that info on some other large desktop applications .
decebalus1 5y ago
More like 'learn how to convince someone that you know how to design large-scale systems'

170 points by jaboutboul 5y ago

dang 5y ago

Past related threads: Learn how to design large-scale systems - https://news.ycombinator.com/item?id=17522362 https://news.ycombinator.com/item?id=17522362 - July 2018 (185 comments) The System Design Primer - https://news.ycombinator.com/item?id=13823979 https://news.ycombinator.com/item?id=13823979 - March 2017 (57 comments)

sparky4 5y ago
Thank you
fivea 5y ago
HN already provides a link to past threads. Check out the "past" link right next to the submission.
- nyc111 5y ago
  Thanks, but "past" link does not show past discussions of the posted article, just older states of HN.
  - fivea 5y ago
    > Thanks, but "past" link does not show past discussions of the posted article, just older states of HN. No, not really. It shows a list of past submissions, which of course include the corresponding discussions. Take a look: https://hn.algolia.com/?query=System%20Design%20Primer%3A%20Learn%20how%20to%20design%20large-scale%20systems&type=story&dateRange=all&sort=byDate&storyText=false&prefix&page=0 https://hn.algolia.com/?query=System%20Design%20Primer%3A%20...
    - macintux 5y ago
      There are two completely different “past” links on the page, hence the confusion. One shows previous days, other previous posts.
      - agumonkey 5y ago
        I never realized there was an article scoped 'past' link.
    - nyc111 5y ago
      That's the search box at the bottom of the page. I get that "hn.algolia.com" link when I search for the title of the article. Otherwise I don't see a "past" link. Maybe there is, I don't know.
- smarx007 5y ago
  You understand that you just lectured the moderator of HN on how to use the website? HN already provides a link to profile bios. Check out the %username% link on top of the comment. /s
  - mynegation 5y ago
    TBH Dang’s bio has only implicit connection to The fact that he is a moderator here, but most people of course know.
- dang 5y ago
  Yup and I'm glad you know that because I wish it got more attention! (I'm a mod here, sorry that wasn't clear.) The reason I post those links to past related threads about specific stories is that many readers like to look up old comments on the same topics. Often there are surprises and gems in there. Oh and there's no implication that the story shouldn't be reposted—reposts are fine after a year or so (https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html). Btw, the 'past' link in the top bar shows you (more or less) what the front page looked like on a particular day. To find threads about specific past stories, it's best to use HN Search at the bottom of the page.

beaconstudios 5y ago

I thought this was going to be about system engineering, but it's just about web scale distributed computing. Disappointing, because I'm digging into systems engineering at the moment and got a bunch of books on it for Christmas, plus thumbing through the SEBoK wiki [1] and associated content. 1: https://www.sebokwiki.org/wiki/Guide_to_the_Systems_Engineering_Body_of_Knowledge_(SEBoK) https://www.sebokwiki.org/wiki/Guide_to_the_Systems_Engineer...

stewvsshark 5y ago
Any books you’d recommend on the subject?
- beaconstudios 5y ago
  Not yet, though I'd love to do recommend some if I could - as I say, I've only just started looking into it fairly recently! If you're interested, drop me an email (in my bio) and I'll let you know, once I've read my new books, if they're any good.
  - MadSudaca 5y ago
    Could you list the books you bought? Thanks!
    - beaconstudios 5y ago
      Sure thing (though it was my lovely wife that bought them, not me!): - Systems Engineering Demystified by Jon Holt - Essential Architecture and Principles of Systems Engineering by C. E. Dickerson & Siyuan Ji - System Design & Management: An Introduction to Systems Engineering by Bob Parkinson On a somewhat related note I also got Difference & Repetition by Deleuze and Process and Reality by Alfred North Whitehead, but they're more philosophical than practical.
      - MadSudaca 5y ago

louwrentius 5y ago

I think this is very valuable and helpful. What really matters I think is when you really need to go this route? Wat what kind of traffic / transactions/s level do you need to (re)design for this? Or even, by what metric would you opt for this solution? I mean, just buying a bigger instance will get you a long way and if you offset that cost to development time / effort / lost opportunity, how will the scale be tipped?

rmbyrro 5y ago
I think it depends a lot on the application use cases and requirements. Things like: Will there be spikes in demand 50x higher than baseline usage? I.e. how fast and efficiently the system needs to scale up? What level of reliability is needed? I.e. is it really a big problem if the system goes down for an hour a handful of times throughout a given time period?
whoevercares 5y ago
Nah people need to go this route just to pass the interview and get the $600k job by answering the question made up for the interview - it always need to scale globally with at least 99.99 availability and development time /other non-tech dimensions almost always not in scope

zerr 5y ago

It seems like all such resources are about web-scale distributed systems and the content/methods are quite standardized. What about other large-scale systems? E.g. desktop applications - for instance, how to design a large-scale desktop system such as Adobe Photoshop or MS Office.

beaconstudios 5y ago
Systems engineering is the discipline that got us to the moon, and builds advanced technology like military hardware. For other large scale software projects, I think they tend to use enterprise software architecture patterns like service-oriented architecture (basically microservices) and ESB (basically kafka).
codr7 5y ago
I doubt you could find two that look the same on the inside, too many talented coders are needed to get to that point; but there's some good stuff by Sean Parent of Adobe on YT.
volkk 5y ago
this resource exists purely for people to practice for systems design interviews. or at least, that's 99% of the use cases of this repo
theflork 5y ago
Architecture of open source applications, which has a free book version available on their site, has that info on some other large desktop applications .

decebalus1 5y ago

More like 'learn how to convince someone that you know how to design large-scale systems'