5 ms·

Reliability, constant work, and a good cup of coffee

122 points by luu 4y ago

compumike 4y ago
My three-word summary would be "Polling is good", though I don't see the word "polling" mentioned once.
- ignoramous 4y ago
  Not really. It is about eliminating modes and variance from hyperscaled systems. The aritcle itself summarises itself: > Both our Network Load Balancer configuration system, and our Route 53 health check system are actually doing many thousands of operations for every "tick" or "cycle" that they iterate. But those operations don't change because the health check statuses did, or because of customer configurations. That's the point. They're like coffee urns, which hold hundreds of cups of coffee at a time no matter how many customers are looking for a cup. On health checks: > Even when there are only a few health checks active, the health checkers send a set of results to the aggregators that is sized to the maximum. For example, if only 10 health checks are configured on a particular health checker, it's still constantly sending out a set of (for example) 10,000 results, if that's how many health checks it could ultimately support. The other 9,990 entries are dummies. However, this ensures that the network load, as well as the work the aggregators are doing, won't increase as customers configure more health checks. That's a significant source of variance ... gone. On updates: > Every few seconds, the health check aggregators send a fixed-size table of health check statuses to the Route 53 DNS servers. When the DNS servers receive it, they store the table in memory, pretty much as-is. That’s a constant work pattern. Every few seconds, receive a table, store it in memory. Why does Route 53 push the data to the DNS servers... because there are more DNS severs than there are health check aggregators... check out Joe Magerramov’s article on putting the smaller service in control. > Then, at query time... even if the first answer it tried is healthy and eligible, the server checks the other potential answers anyway. This approach ensures that even if a status changes, the DNS server is still performing the same work that it was before. There's no increase in scan or retrieval time. On configuration: > Rather than generate events, AWS Hyperplane integrates customer changes into a configuration file that's stored in Amazon S3. This happens right when the customer makes the change. Then, rather than respond to a workflow, AWS Hyperplane nodes fetch this configuration from Amazon S3 every few seconds. The AWS Hyperplane nodes then process and load this configuration file... Even if the configuration is completely identical to what it was the last time, the nodes process and load the latest copy anyway. Effectively, the system is always processing and loading the maximum number of configuration changes. Whether one load balancer changed or hundreds, it behaves the same.
  - HyperSane 4y ago
    The idea is to make the worse case scenario the only scenario. Very clever.
    - vlovich123 4y ago
      It’s an idea but I’m not sure I’m convinced. There’s no evidence provided that doing this actually improves availability / reduces cascading failures / improves time to recover. And AWS doesn’t really have the best track record in terms of uptime…
      - jaggederest 4y ago
        What do you mean, obviously if the status dashboard is green, nothing is wrong. /s Who are you going to believe, me, or your own lying eyes?
  - donavanm 4y ago
    Exactly this. Eliminating variable modes and especially additional work or slower paths during a state change like a failure. Another classic is debug logs. Turning on additional logging when your system is under stress is a great way to increase the impact. Instead always record/emit the telemetry youll likely need and discard later if its unnecessary to retain.
  - barbazoo 4y ago
    > Even when there are only a few health checks active, the health checkers send a set of results to the aggregators that is sized to the maximum. For example, if only 10 health checks are configured on a particular health checker, it's still constantly sending out a set of (for example) 10,000 results, if that's how many health checks it could ultimately support. The other 9,990 entries are dummies. However, this ensures that the network load, as well as the work the aggregators are doing, won't increase as customers configure more health checks Won't the 10 health checks be less work, i.e. CPU cycles, than 10,000 checks? Sure you can iterate through 10k instead of 10 but performing a health check, parsing the result, etc, won't that still scale with the numbers of health checks configured? How could that be a constant amount of work in this example?
emptybits 4y ago
s/a good cup/many cups/ As the article itself points out, unlike computing at scale, nothing about enormous warming urns of coffee ensures a good cup. Reliable and constant.
- taneq 4y ago
  The best cup of coffee is the one I got right now.
  - taeric 4y ago
    Definitely not true. Is a mix of, "hunger is the best spice," and, "a bird in hand..." But those don't compose, as it were. I've had many cups of coffee that I just poured out, as they were gross.
- danw1979 4y ago
  As long as you like your coffee brewed and served in the way _we_ decide, then we’ve got gallons of coffee for you.
- HPsquared 4y ago
  "Quantity has a quality of its own"
  - helloworld11 4y ago
    Stalin (who knew a thing or two about quantities taking on their own quality), in case anyone wonders. (though some have blamed Napoleon)
michaelteter 4y ago
Here I was expecting a pragmatic nerd approach to producing reliably good cups of coffee :(.
- PartiallyTyped 4y ago
  Check out James Hoffmann on youtube [1]! [1] https://www.youtube.com/@jameshoffmann https://www.youtube.com/@jameshoffmann
  - blamazon 4y ago
    Or you can check out Hames Joffman for an unhelpful summary: [2] https://youtube.com/@hamesjoffmann https://youtube.com/@hamesjoffmann
foobarbecue 4y ago
> Second, many coffee urns contain heating elements and thermostats, so as you take more coffee out of them, they actually perform a bit less work. There’s just less coffee left to keep warm. I know this is just an analogy... but I don't think that's how heat transfer works. Heat loss here is primarily a function of temperature difference across the urn walls, and and thermal conductivity of the urn walls. I don't see how replacing coffee with air would slow down the heat loss.
- knubie 4y ago
  I think he means it takes less energy to heat a smaller volume of coffee.
  - foobarbecue 4y ago
    From cold, sure. But he's talking about an urn that started full of hot coffee, and then saying energy input required to maintain that temperature will decrease as the coffee volume decreases. What's the mechanism for that supposed to be? The more I think about it, the more I think he's just wrong about the coffee thing. Contents of the urn have no effect on energy required to maintain its temperature.
    - Zanovis 4y ago
      I think OP is correct. Think about the extremes of the urn, full vs 1mL in it. It will require much less energy to heat the 1mL state than the full state. Regarding the contents, the thermal conductivity of liquid (Water mostly for coffee) is higher than that of air, so it transfers heat out of the urn faster than an urn full of air. It also has higher heat capacity which requires more heat energy per degree change of temperature. Heat capacity is given by (C) = heat absorbed (Q) / temperature change (ΔT); heat capacity increases with volume
mmmmmbop 4y ago
Is the title a reference to the movie 'Stutz'?
mark_l_watson 4y ago
I more or less buy into the author’s analogy. I have two work modes: building things that have to work so I avoid unnecessary complexity and services I don’t have experience with, and, the second mode is where I am experimenting and learning. BTW, in the opening “”” One of my favorite paintings is “Nighthawks” by Edward Hopper. A few years ago, I was lucky enough to see it in person at the Art Institute of Chicago.””” My wife and I saw the same art piece - that is what hooked me into reading the entire article. I wish I was better at writing introductory ’hooks.’
- onos 4y ago
  FWIW I thought the “lede” was far too buried and stopped reading before I got to it… so this style doesn’t work for everyone.
nijave 4y ago
This reminds me a lot of the controller/control loop pattern used in Kubernetes with the additional constraint that the amount of work done in a loop is fixed. https://kubernetes.io/docs/concepts/architecture/controller/ https://kubernetes.io/docs/concepts/architecture/controller/
k__ 4y ago
I only knew the painting from the meme with the badger. Thought it to be a random scene.
rajman187 4y ago
The most offensive element in the article was general dismissal of espresso as actual coffee rather than the brown sludge that fills most large vats at conference centers and apparently Amazon offices. Sure, it makes for a good analogy, but by the same token if you ever find yourself in Italy the post-lunch rush is never delayed with long waits because pulling a shot is near-instantaneous. In fact espresso was the method derived earlier in the century to refuel factory workers quickly, hence had whilst standing, ready to get back to labor.
taeric 4y ago
I confess I assumed those were percolators. Used to be a rather common way to make coffee, as I understand it. Ironically to the article's point, my understanding is those fell out of favor for being bad coffee.
achrono 4y ago
Can someone explain why anti-fragility is the correct term to use here? It feels like this is a highly resilient and fault-correcting system, but the definition of anti-fragility is the ability to increase in capacity to thrive in the face of stressors -- how exactly is that the case here?
deterministic 4y ago
What a great read. The author truly understands what it takes to design reliable high-performance systems. It reminds me of how well designed hard real-time systems and high-performance games are implemented.

122 points by luu 4y ago

compumike 4y ago

My three-word summary would be "Polling is good", though I don't see the word "polling" mentioned once.

ignoramous 4y ago
Not really. It is about eliminating modes and variance from hyperscaled systems. The aritcle itself summarises itself: > Both our Network Load Balancer configuration system, and our Route 53 health check system are actually doing many thousands of operations for every "tick" or "cycle" that they iterate. But those operations don't change because the health check statuses did, or because of customer configurations. That's the point. They're like coffee urns, which hold hundreds of cups of coffee at a time no matter how many customers are looking for a cup. On health checks: > Even when there are only a few health checks active, the health checkers send a set of results to the aggregators that is sized to the maximum. For example, if only 10 health checks are configured on a particular health checker, it's still constantly sending out a set of (for example) 10,000 results, if that's how many health checks it could ultimately support. The other 9,990 entries are dummies. However, this ensures that the network load, as well as the work the aggregators are doing, won't increase as customers configure more health checks. That's a significant source of variance ... gone. On updates: > Every few seconds, the health check aggregators send a fixed-size table of health check statuses to the Route 53 DNS servers. When the DNS servers receive it, they store the table in memory, pretty much as-is. That’s a constant work pattern. Every few seconds, receive a table, store it in memory. Why does Route 53 push the data to the DNS servers... because there are more DNS severs than there are health check aggregators... check out Joe Magerramov’s article on putting the smaller service in control. > Then, at query time... even if the first answer it tried is healthy and eligible, the server checks the other potential answers anyway. This approach ensures that even if a status changes, the DNS server is still performing the same work that it was before. There's no increase in scan or retrieval time. On configuration: > Rather than generate events, AWS Hyperplane integrates customer changes into a configuration file that's stored in Amazon S3. This happens right when the customer makes the change. Then, rather than respond to a workflow, AWS Hyperplane nodes fetch this configuration from Amazon S3 every few seconds. The AWS Hyperplane nodes then process and load this configuration file... Even if the configuration is completely identical to what it was the last time, the nodes process and load the latest copy anyway. Effectively, the system is always processing and loading the maximum number of configuration changes. Whether one load balancer changed or hundreds, it behaves the same.
- HyperSane 4y ago
  The idea is to make the worse case scenario the only scenario. Very clever.
  - vlovich123 4y ago
    It’s an idea but I’m not sure I’m convinced. There’s no evidence provided that doing this actually improves availability / reduces cascading failures / improves time to recover. And AWS doesn’t really have the best track record in terms of uptime…
    - jaggederest 4y ago
      What do you mean, obviously if the status dashboard is green, nothing is wrong. /s Who are you going to believe, me, or your own lying eyes?
- donavanm 4y ago
  Exactly this. Eliminating variable modes and especially additional work or slower paths during a state change like a failure. Another classic is debug logs. Turning on additional logging when your system is under stress is a great way to increase the impact. Instead always record/emit the telemetry youll likely need and discard later if its unnecessary to retain.
- barbazoo 4y ago
  > Even when there are only a few health checks active, the health checkers send a set of results to the aggregators that is sized to the maximum. For example, if only 10 health checks are configured on a particular health checker, it's still constantly sending out a set of (for example) 10,000 results, if that's how many health checks it could ultimately support. The other 9,990 entries are dummies. However, this ensures that the network load, as well as the work the aggregators are doing, won't increase as customers configure more health checks Won't the 10 health checks be less work, i.e. CPU cycles, than 10,000 checks? Sure you can iterate through 10k instead of 10 but performing a health check, parsing the result, etc, won't that still scale with the numbers of health checks configured? How could that be a constant amount of work in this example?

emptybits 4y ago

s/a good cup/many cups/ As the article itself points out, unlike computing at scale, nothing about enormous warming urns of coffee ensures a good cup. Reliable and constant.

taneq 4y ago
The best cup of coffee is the one I got right now.
- taeric 4y ago
  Definitely not true. Is a mix of, "hunger is the best spice," and, "a bird in hand..." But those don't compose, as it were. I've had many cups of coffee that I just poured out, as they were gross.
danw1979 4y ago
As long as you like your coffee brewed and served in the way _we_ decide, then we’ve got gallons of coffee for you.
HPsquared 4y ago
"Quantity has a quality of its own"
- helloworld11 4y ago
  Stalin (who knew a thing or two about quantities taking on their own quality), in case anyone wonders. (though some have blamed Napoleon)

michaelteter 4y ago

Here I was expecting a pragmatic nerd approach to producing reliably good cups of coffee :(.

PartiallyTyped 4y ago
Check out James Hoffmann on youtube [1]! [1] https://www.youtube.com/@jameshoffmann https://www.youtube.com/@jameshoffmann
- blamazon 4y ago
  Or you can check out Hames Joffman for an unhelpful summary: [2] https://youtube.com/@hamesjoffmann https://youtube.com/@hamesjoffmann

foobarbecue 4y ago

> Second, many coffee urns contain heating elements and thermostats, so as you take more coffee out of them, they actually perform a bit less work. There’s just less coffee left to keep warm. I know this is just an analogy... but I don't think that's how heat transfer works. Heat loss here is primarily a function of temperature difference across the urn walls, and and thermal conductivity of the urn walls. I don't see how replacing coffee with air would slow down the heat loss.

knubie 4y ago
I think he means it takes less energy to heat a smaller volume of coffee.
- foobarbecue 4y ago
  From cold, sure. But he's talking about an urn that started full of hot coffee, and then saying energy input required to maintain that temperature will decrease as the coffee volume decreases. What's the mechanism for that supposed to be? The more I think about it, the more I think he's just wrong about the coffee thing. Contents of the urn have no effect on energy required to maintain its temperature.
  - Zanovis 4y ago
    I think OP is correct. Think about the extremes of the urn, full vs 1mL in it. It will require much less energy to heat the 1mL state than the full state. Regarding the contents, the thermal conductivity of liquid (Water mostly for coffee) is higher than that of air, so it transfers heat out of the urn faster than an urn full of air. It also has higher heat capacity which requires more heat energy per degree change of temperature. Heat capacity is given by (C) = heat absorbed (Q) / temperature change (ΔT); heat capacity increases with volume

mmmmmbop 4y ago

Is the title a reference to the movie 'Stutz'?

mark_l_watson 4y ago

I more or less buy into the author’s analogy. I have two work modes: building things that have to work so I avoid unnecessary complexity and services I don’t have experience with, and, the second mode is where I am experimenting and learning. BTW, in the opening “”” One of my favorite paintings is “Nighthawks” by Edward Hopper. A few years ago, I was lucky enough to see it in person at the Art Institute of Chicago.””” My wife and I saw the same art piece - that is what hooked me into reading the entire article. I wish I was better at writing introductory ’hooks.’

onos 4y ago
FWIW I thought the “lede” was far too buried and stopped reading before I got to it… so this style doesn’t work for everyone.

nijave 4y ago

This reminds me a lot of the controller/control loop pattern used in Kubernetes with the additional constraint that the amount of work done in a loop is fixed. https://kubernetes.io/docs/concepts/architecture/controller/ https://kubernetes.io/docs/concepts/architecture/controller/

k__ 4y ago

I only knew the painting from the meme with the badger. Thought it to be a random scene.

rajman187 4y ago

The most offensive element in the article was general dismissal of espresso as actual coffee rather than the brown sludge that fills most large vats at conference centers and apparently Amazon offices. Sure, it makes for a good analogy, but by the same token if you ever find yourself in Italy the post-lunch rush is never delayed with long waits because pulling a shot is near-instantaneous. In fact espresso was the method derived earlier in the century to refuel factory workers quickly, hence had whilst standing, ready to get back to labor.

taeric 4y ago

I confess I assumed those were percolators. Used to be a rather common way to make coffee, as I understand it. Ironically to the article's point, my understanding is those fell out of favor for being bad coffee.

achrono 4y ago

Can someone explain why anti-fragility is the correct term to use here? It feels like this is a highly resilient and fault-correcting system, but the definition of anti-fragility is the ability to increase in capacity to thrive in the face of stressors -- how exactly is that the case here?

deterministic 4y ago

What a great read. The author truly understands what it takes to design reliable high-performance systems. It reminds me of how well designed hard real-time systems and high-performance games are implemented.