16 ms·

Notes on Distributed Systems for Young Bloods

einpoklum 2y ago
> # If you can fit your problem in memory, it’s probably trivial. A popular fallacy among some distributed systems engineers. It's not at all trivial, it's just in a complementary domain of problems to be tackled. The fallacy easily leads to a situation where you need a 100-cluster machine to do the work that proper optimization would let you do on a single machine.
sillywalk 2y ago
Previous discussions from way back: https://news.ycombinator.com/item?id=5055371 https://news.ycombinator.com/item?id=5055371 346 points|jcdavis|12 years ago|42 comments https://news.ycombinator.com/item?id=12245909 https://news.ycombinator.com/item?id=12245909 386 points|kiyanwang|8 years ago|133 comments
turtledragonfly 2y ago
Excellent list; I like the pragmatic and down-to-earth explanations. No buzzwords, no "microservices" (: I'd say that a good amount of this advice also applies to one-box systems. There can be lots of kinda/sorta distributed sub-components to consider — could be IPC between programs, or even coordination amongst threads in one process. Even the notion of unified memory on one box is a bit of a lie, but at least the hardware can provide some better guarantees than you get in "real" distributed cases. A lot of the advice where they compare "distributed" to "single-machine" could pretty well apply to "multi-threaded" vs "single-threaded," too. And on another axis, once you make a program and give it to various people to run, it becomes sort of a "distributed" situation, too — now you have to worry about different versions of that program existing in the wild, compatibility between them and upgrade issues, etc. So things like feature flags, mentioned in the article, can be relevant there, as well. It's perhaps more of a spectrum of distributedness: from single-CPU to multi-CPU, to multi-computer-tightly-connected, to multi-computer-globally-distributed, with various points in between. And multiple dimensions.
- chipdart 2y ago
  > I'd say that a good amount of this advice also applies to one-box systems. Nothing in "distributed systems" implies any constraint on deployment. The only trait that's critical to the definition is having different flows of control communicating over a network through message-passing. One very famous example of distributed systems is multiple processes running on the same box communicating over localhost, which happens to be where some people cut their distributed system's teeth.
- bee_rider 2y ago
  The neighboring universe, so tantalizingly close, where AMD gave us different memory spaces for each chiplet, is something I think about often. Imagine, we could all be writing all our code as beautiful distributed memory MPI programs. No more false sharing, we all get to think hard and explicitly about our communication patterns.
  - immibis 2y ago
    This is already here. It's called NUMA. You can access all memory from any CPU, but accessing memory that's connected to your CPU makes the access faster. NUMA-aware operating systems can limit your process to a CPU cluster and allocate memory from the same cluster, then replicate this on the other clusters, so they all run fast and only transfer data between clusters when they need to.
sulam 2y ago
One that is not mentioned here but that I like as a general principle is that you cannot have exactly once delivery. At most once or at least once are both possible, but you have to pick your failure poison and architect for it.
- Spivak 2y ago
  Yes but in practice this is not a problem because the bits that are impossible are so narrow that turning at-least-once into exactly-once is so easy it's a service offered by cloud vendors https://cloud.google.com/pubsub/docs/exactly-once-delivery https://cloud.google.com/pubsub/docs/exactly-once-delivery
  - sulam 2y ago
    Those only work because they have retries built in at a layer that runs inside your service. You should understand this because it can have implications for the performance of your system during failures.
    - devoutsalsa 2y ago
      For example, you can have an exactly once implementation that is essentially implemented as at least once with repeated idempotent calls until a confirmation is received. Idempotency handling has a cost, confirmation reply has a cost, retry on call that didn’t have confirmation has a cost, etc.
  - jclulow 2y ago
    "Redelivery versus duplicate" is doing quite a lot of work in there. This is an "at least once" delivery system providing building blocks that you can use to cope with the fact that it's physically impossible to prevent redelivery under some circumstances, which are not actually that rare because some of those circumstances are your fault, not Google's, etc.
  - fiddlerwoaroof 2y ago
    My experience is building streaming systems using “exactly-once delivery” primitives is much more awkward than designing your system around at least once primitives and explicitly de-duplicating using last-write wins. For one thing, LWW gives you an obvious recovery strategy if you have outages of the primitive your system is built on: a lot of the exactly once modes for tools make failure recovery harder than necessary
languagehacker 2y ago
I share this doc with the most promising people I get to work with. When I worked at Lookout, Jeff Hodges shared this essay as a presentation, and ended it with a corollary: don't pretend that engineering isn't political. People that think that the code speaks for itself are missing out on important aspects of how to influence the way things are built and how to truly get results. Ten years later, and there are few people who still so concisely understand the intersection of engineering leadership and those table-stakes capabilities we normally associate with SRE / DevOps.
- decasia 2y ago
  > there are few people who still so concisely understand the intersection of engineering leadership and those table-stakes capabilities we normally associate with SRE / DevOps. I'm curious what else is good to read about this topic, if anything comes to your mind?
abatilo 2y ago
I had the pleasure to briefly work with the author of this post within the last few years. Jeff was one of the most enlightening and positive people I've ever learned from. He was refreshingly honest about what challenges he was having, and delightfully accessible for mentorship and advice.
Maro 2y ago
I think a lot has changed since 2013 when this article was written. Back then cloud services were less mature and there were more legitimate cases when you had to care about the theoretical distributed aspects of your backend architecture.. although even then these were quickly disappearing, unless you worked at a few select bigtech companies like the FAANGs. But today, in 2024, if you just standardize on AWS, you can pretty much use one of the AWS services for pretty much anything. And that AWS service will be already distributed in the backend, for free, in terms of you not having to worry about it. Additionaly, it will be run by AWS engineers for you, with all sorts of failovers, monitoring, alerting, etc, behind the scenes that will be much better than what you can build. So these days, for 99% of people it doesn't really make sense to worry too much about this theoretical stuff, like Paxos, Raft, consistency, vector clocks, byzantine failures, CAP, distributed locks, distributed transactions, etc. And that's good progress, it has been abstracted away behind API calls. I think it's pretty rational to just build on top of AWS (or similar) services, and accept that it's a black box distributed system that may still go down sometimes, but it'll still be 10-100x more reliable then if you try to build your own distributed system. Of course, even for the 99%, there are still important practical things to keep in mind, like logging, debugging, backpressure, etc. Another thing I learned is that some concepts, such as availability, are less important, and less achievable, then they seem on paper. On paper it sounds like a worthy exercise to design systems that will fail over and come back automatically if a component fails, with only a few seconds downtime. Magically, with everything working like a well oiled machine. In practice this is pretty much never works out, because there are componenets of the system that the designed didn't think of, and it's those that will fail and bring the system down. Eg. see the recent Crowdstrike incident. And, with respect to importance of availability, of the ~10 companies I worked at in the past ~20 years there's wasn't a single one that couldn't tolerate a few hours of downtime with zero to minimal business and PR impact (people are used to things going down a couple of times a year). I remember having outages at a SaaS company I worked for 10 years ago, no revenue was coming in for a few days, but then people would just spend more in the following day. Durability is more important, but even that is less important in practice then we'd like to think [1]. The remaining 1% of engineers, who get to worry about the beuatiful theoretical AND practical aspects of distributed computing [because they work at Google or Facebook or AWS] should count themselves lucky! I think it's one of the most interesting fields in Computer Science. I say this as somebody who deeply cares/cared about theoretical distributed computing, I wrote distributed databases [2] and papers in the past [3]. But working in industry (also managing a Platform Eng team), I cannot recall the last time I had to worry about such things. [1] PostgreSQL used fsync incorrectly for 20 years - https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991 [2] https://github.com/scalien/scaliendb https://github.com/scalien/scaliendb [3] https://arxiv.org/abs/1209.4187 https://arxiv.org/abs/1209.4187
j-pb 2y ago
The article should really mention CALM (Consistency as Logical Monotonicity)[1], it's much easier to understand and a more fundamental result than CAP. It is also much more applicable and enables people with little experience to build extremely robust distributed systems. Idempotence, CRDTs, WALs, Raft, they are all special cases of the CALM principle. [1]: https://arxiv.org/pdf/1901.01930 https://arxiv.org/pdf/1901.01930
- baq 2y ago
  The article predates your paper by 6 years.
  - j-pb 2y ago
    Fair, we should add [2013] to the title though.
- foota 2y ago
  I looked at the bloom repo and it seems somewhat stale, do you know if this is something still being worked on?
tapanjk 2y ago
(2013)
ramon156 2y ago
Good article. I did notice it's 8 years old. Some stuff never changes I guess
- baq 2y ago
  8 is the new 11 I see. (Article dates to 2013 ;)
  - gautamsomani 2y ago
    lol
  - bee_rider 2y ago
    Covid years don’t count.
kuharich 2y ago
Past comments: https://news.ycombinator.com/item?id=23365402 https://news.ycombinator.com/item?id=23365402
vitus 2y ago
> If you can fit your problem in memory, it’s probably trivial. A corollary: "in-memory is much bigger than you probably think it is." I thought I knew what a large amount of RAM was, and then all the major clouds started offering 12TB VMs for SAP HANA. edit: this seems like it's touched on very briefly with "Computers can do more than you think they can." but even that only talks about 24GB machines (admittedly in 2012, but still, I'm sure there were plenty of machines with 10x that amount of RAM back then)
- gen220 2y ago
  Even comparatively senior engineers make this mistake relatively often. If you're a SaaS dealing with at most 100GB of analytical data per customer, (eventually, sharded) postgres is all you need.
sillyfluke 2y ago
I guess most of it's still highly relevant since not enough people have clamored for the (2013) tag.
LandR 2y ago
Is Young Bloods a common term for beginner / newbie ?
- Bnjoroge 2y ago
  yup
- Cthulhu_ 2y ago
  https://www.merriam-webster.com/dictionary/youngblood https://www.merriam-webster.com/dictionary/youngblood young· blood 1: a young inexperienced person especially : one who is newly prominent in a field of endeavor 2: a young African American male The first known use of youngblood was in 1602 So it's a bit archaic but not abnormal. Has been used as a surname, too.
pradn 2y ago
> Distributed systems are different because they fail often The key here is not just the rate of failure, but the rate of failure in a system of multiple nodes. And - "distributed systems problems" don't only arise with several servers connected by a network. Any set of nodes with relations between them - files on disk linked logically, buffers on different IO devices - these are also going to face similar problems.
- roryirvine 2y ago
  Absolutely. In fact, it's a class of problems that can - and do - arise on any software system comprising more than a sole single-threaded process that's been locked in memory. Some old-timers love to scoff at the inordinate amount of complexity that comes from mitigating these issues, and will complain that it would all be so much simpler if you would just run your software on a single server. In reality, that was barely true even back in the AS/400 or VAXft days - and even then it didn't apply to the rather more chaotic multi-user, multi-process Unix world.
jupp0r 2y ago
(2013)

> # If you can fit your problem in memory, it’s probably trivial. A popular fallacy among some distributed systems engineers. It's not at all trivial, it's just in a complementary domain of problems to be tackled. The fallacy easily leads to a situation where you need a 100-cluster machine to do the work that proper optimization would let you do on a single machine.

sillywalk 2y ago

Previous discussions from way back: https://news.ycombinator.com/item?id=5055371 https://news.ycombinator.com/item?id=5055371 346 points|jcdavis|12 years ago|42 comments https://news.ycombinator.com/item?id=12245909 https://news.ycombinator.com/item?id=12245909 386 points|kiyanwang|8 years ago|133 comments

turtledragonfly 2y ago

Excellent list; I like the pragmatic and down-to-earth explanations. No buzzwords, no "microservices" (: I'd say that a good amount of this advice also applies to one-box systems. There can be lots of kinda/sorta distributed sub-components to consider — could be IPC between programs, or even coordination amongst threads in one process. Even the notion of unified memory on one box is a bit of a lie, but at least the hardware can provide some better guarantees than you get in "real" distributed cases. A lot of the advice where they compare "distributed" to "single-machine" could pretty well apply to "multi-threaded" vs "single-threaded," too. And on another axis, once you make a program and give it to various people to run, it becomes sort of a "distributed" situation, too — now you have to worry about different versions of that program existing in the wild, compatibility between them and upgrade issues, etc. So things like feature flags, mentioned in the article, can be relevant there, as well. It's perhaps more of a spectrum of distributedness: from single-CPU to multi-CPU, to multi-computer-tightly-connected, to multi-computer-globally-distributed, with various points in between. And multiple dimensions.

chipdart 2y ago
> I'd say that a good amount of this advice also applies to one-box systems. Nothing in "distributed systems" implies any constraint on deployment. The only trait that's critical to the definition is having different flows of control communicating over a network through message-passing. One very famous example of distributed systems is multiple processes running on the same box communicating over localhost, which happens to be where some people cut their distributed system's teeth.
bee_rider 2y ago
The neighboring universe, so tantalizingly close, where AMD gave us different memory spaces for each chiplet, is something I think about often. Imagine, we could all be writing all our code as beautiful distributed memory MPI programs. No more false sharing, we all get to think hard and explicitly about our communication patterns.
- immibis 2y ago
  This is already here. It's called NUMA. You can access all memory from any CPU, but accessing memory that's connected to your CPU makes the access faster. NUMA-aware operating systems can limit your process to a CPU cluster and allocate memory from the same cluster, then replicate this on the other clusters, so they all run fast and only transfer data between clusters when they need to.

sulam 2y ago

One that is not mentioned here but that I like as a general principle is that you cannot have exactly once delivery. At most once or at least once are both possible, but you have to pick your failure poison and architect for it.

Spivak 2y ago
Yes but in practice this is not a problem because the bits that are impossible are so narrow that turning at-least-once into exactly-once is so easy it's a service offered by cloud vendors https://cloud.google.com/pubsub/docs/exactly-once-delivery https://cloud.google.com/pubsub/docs/exactly-once-delivery
- sulam 2y ago
  Those only work because they have retries built in at a layer that runs inside your service. You should understand this because it can have implications for the performance of your system during failures.
  - devoutsalsa 2y ago
    For example, you can have an exactly once implementation that is essentially implemented as at least once with repeated idempotent calls until a confirmation is received. Idempotency handling has a cost, confirmation reply has a cost, retry on call that didn’t have confirmation has a cost, etc.
- jclulow 2y ago
  "Redelivery versus duplicate" is doing quite a lot of work in there. This is an "at least once" delivery system providing building blocks that you can use to cope with the fact that it's physically impossible to prevent redelivery under some circumstances, which are not actually that rare because some of those circumstances are your fault, not Google's, etc.
- fiddlerwoaroof 2y ago
  My experience is building streaming systems using “exactly-once delivery” primitives is much more awkward than designing your system around at least once primitives and explicitly de-duplicating using last-write wins. For one thing, LWW gives you an obvious recovery strategy if you have outages of the primitive your system is built on: a lot of the exactly once modes for tools make failure recovery harder than necessary

languagehacker 2y ago

I share this doc with the most promising people I get to work with. When I worked at Lookout, Jeff Hodges shared this essay as a presentation, and ended it with a corollary: don't pretend that engineering isn't political. People that think that the code speaks for itself are missing out on important aspects of how to influence the way things are built and how to truly get results. Ten years later, and there are few people who still so concisely understand the intersection of engineering leadership and those table-stakes capabilities we normally associate with SRE / DevOps.

decasia 2y ago
> there are few people who still so concisely understand the intersection of engineering leadership and those table-stakes capabilities we normally associate with SRE / DevOps. I'm curious what else is good to read about this topic, if anything comes to your mind?

abatilo 2y ago

I had the pleasure to briefly work with the author of this post within the last few years. Jeff was one of the most enlightening and positive people I've ever learned from. He was refreshingly honest about what challenges he was having, and delightfully accessible for mentorship and advice.

Maro 2y ago

I think a lot has changed since 2013 when this article was written. Back then cloud services were less mature and there were more legitimate cases when you had to care about the theoretical distributed aspects of your backend architecture.. although even then these were quickly disappearing, unless you worked at a few select bigtech companies like the FAANGs. But today, in 2024, if you just standardize on AWS, you can pretty much use one of the AWS services for pretty much anything. And that AWS service will be already distributed in the backend, for free, in terms of you not having to worry about it. Additionaly, it will be run by AWS engineers for you, with all sorts of failovers, monitoring, alerting, etc, behind the scenes that will be much better than what you can build. So these days, for 99% of people it doesn't really make sense to worry too much about this theoretical stuff, like Paxos, Raft, consistency, vector clocks, byzantine failures, CAP, distributed locks, distributed transactions, etc. And that's good progress, it has been abstracted away behind API calls. I think it's pretty rational to just build on top of AWS (or similar) services, and accept that it's a black box distributed system that may still go down sometimes, but it'll still be 10-100x more reliable then if you try to build your own distributed system. Of course, even for the 99%, there are still important practical things to keep in mind, like logging, debugging, backpressure, etc. Another thing I learned is that some concepts, such as availability, are less important, and less achievable, then they seem on paper. On paper it sounds like a worthy exercise to design systems that will fail over and come back automatically if a component fails, with only a few seconds downtime. Magically, with everything working like a well oiled machine. In practice this is pretty much never works out, because there are componenets of the system that the designed didn't think of, and it's those that will fail and bring the system down. Eg. see the recent Crowdstrike incident. And, with respect to importance of availability, of the ~10 companies I worked at in the past ~20 years there's wasn't a single one that couldn't tolerate a few hours of downtime with zero to minimal business and PR impact (people are used to things going down a couple of times a year). I remember having outages at a SaaS company I worked for 10 years ago, no revenue was coming in for a few days, but then people would just spend more in the following day. Durability is more important, but even that is less important in practice then we'd like to think [1]. The remaining 1% of engineers, who get to worry about the beuatiful theoretical AND practical aspects of distributed computing [because they work at Google or Facebook or AWS] should count themselves lucky! I think it's one of the most interesting fields in Computer Science. I say this as somebody who deeply cares/cared about theoretical distributed computing, I wrote distributed databases [2] and papers in the past [3]. But working in industry (also managing a Platform Eng team), I cannot recall the last time I had to worry about such things. [1] PostgreSQL used fsync incorrectly for 20 years - https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991 [2] https://github.com/scalien/scaliendb https://github.com/scalien/scaliendb [3] https://arxiv.org/abs/1209.4187 https://arxiv.org/abs/1209.4187

j-pb 2y ago

The article should really mention CALM (Consistency as Logical Monotonicity)[1], it's much easier to understand and a more fundamental result than CAP. It is also much more applicable and enables people with little experience to build extremely robust distributed systems. Idempotence, CRDTs, WALs, Raft, they are all special cases of the CALM principle. [1]: https://arxiv.org/pdf/1901.01930 https://arxiv.org/pdf/1901.01930

baq 2y ago
The article predates your paper by 6 years.
- j-pb 2y ago
  Fair, we should add [2013] to the title though.
foota 2y ago
I looked at the bloom repo and it seems somewhat stale, do you know if this is something still being worked on?

tapanjk 2y ago

(2013)

ramon156 2y ago

Good article. I did notice it's 8 years old. Some stuff never changes I guess

baq 2y ago
8 is the new 11 I see. (Article dates to 2013 ;)
- gautamsomani 2y ago
  lol
- bee_rider 2y ago
  Covid years don’t count.

kuharich 2y ago

Past comments: https://news.ycombinator.com/item?id=23365402 https://news.ycombinator.com/item?id=23365402

vitus 2y ago

> If you can fit your problem in memory, it’s probably trivial. A corollary: "in-memory is much bigger than you probably think it is." I thought I knew what a large amount of RAM was, and then all the major clouds started offering 12TB VMs for SAP HANA. edit: this seems like it's touched on very briefly with "Computers can do more than you think they can." but even that only talks about 24GB machines (admittedly in 2012, but still, I'm sure there were plenty of machines with 10x that amount of RAM back then)

gen220 2y ago
Even comparatively senior engineers make this mistake relatively often. If you're a SaaS dealing with at most 100GB of analytical data per customer, (eventually, sharded) postgres is all you need.

sillyfluke 2y ago

I guess most of it's still highly relevant since not enough people have clamored for the (2013) tag.

LandR 2y ago

Is Young Bloods a common term for beginner / newbie ?

Bnjoroge 2y ago
yup
Cthulhu_ 2y ago
https://www.merriam-webster.com/dictionary/youngblood https://www.merriam-webster.com/dictionary/youngblood young· blood 1: a young inexperienced person especially : one who is newly prominent in a field of endeavor 2: a young African American male The first known use of youngblood was in 1602 So it's a bit archaic but not abnormal. Has been used as a surname, too.

pradn 2y ago

> Distributed systems are different because they fail often The key here is not just the rate of failure, but the rate of failure in a system of multiple nodes. And - "distributed systems problems" don't only arise with several servers connected by a network. Any set of nodes with relations between them - files on disk linked logically, buffers on different IO devices - these are also going to face similar problems.

roryirvine 2y ago
Absolutely. In fact, it's a class of problems that can - and do - arise on any software system comprising more than a sole single-threaded process that's been locked in memory. Some old-timers love to scoff at the inordinate amount of complexity that comes from mitigating these issues, and will complain that it would all be so much simpler if you would just run your software on a single server. In reality, that was barely true even back in the AS/400 or VAXft days - and even then it didn't apply to the rather more chaotic multi-user, multi-process Unix world.

jupp0r 2y ago

(2013)