60 ms·

> Consider a system with 1000 nodes where each node sends heartbeats to a central monitor every 500 milliseconds. This results in 2000 heartbeat messages per se

by toast0 1y ago

> Consider a system with 1000 nodes where each node sends heartbeats to a central monitor every 500 milliseconds. This results in 2000 heartbeat messages per second just for health monitoring. In a busy production environment, this overhead can interfere with actual application traffic. If your 1000-node busy production environment is run so close to the edge that 2000 heartbeat messages per second, push it into overload, that's impressive resource scheduling. Really, setting the interval balances speed of detection/cost of slow detection vs cost of reacting to a momentary interruption. If the node actually dies, you'd like to react as soon as possible; but if it's something like a link flap or system pause (GC or otherwise), most applications would prefer to wait and not transition state; some applications like live broadcast are better served by moving very rapidly and 500 ms might be too long. Re: network partitioning, the author left out the really fun splits. Say you have servers in DC, TX, and CA. If there's a damaged (but not severed) link between TX and CA, there's a good chance that DC can talk to everyone, but TX and CA can't communicate. You can have that inside a datacenter too, maybe each node can only reach 75% of the other nodes, but A can reach B and B can reach C does not indicate A can reach C. Lots of fun times there.

macintux 1y ago
Kingsbury & Bailis's paper on the topic of network partitions: https://github.com/aphyr/partitions-post https://github.com/aphyr/partitions-post
karmakaze 1y ago
I've dealt with exactly this. We had a couple thousand webapp server instances that had connections to a MySQL database. Each one only polled its connection for liveliness once per second, but those were little interruptions that were poking at the servers and showed up on top time consuming request charts.
hinkley 1y ago
When systems were smaller I tried to push for the realization that I don’t need a heartbeat from a machine that is currently returning status 200 messages from 60 req/s. The evidence of work is already there, and more meaningful than the status check. We end up adding real work to the status checks often enough anyway, to make sure the database is still visible and other services. So inference has a lot of power that a heartbeat does not.
- taeric 1y ago
  I'm amused that you got pushback. Just going on the metaphor, the very way that we check heartbeats in people is by a proof of work in some other part of the body. Isn't like we are directly hooked into the heart. It isn't like the heart sends out an otherwise useless signal.
zbentley 1y ago
> If your 1000-node busy production environment is run so close to the edge that 2000 heartbeat messages per second, push it into overload, that's impressive resource scheduling. Eeeh I’m not so sure. The overhead of handling a hello-world heartbeat request is negligible, sure, but what about the overhead of having the connections open (file descriptors, maybe >1 per request), client tracking metadata for what are necessarily all different client location identifiers, and so on? That’s still cheap stuff, but at 2krps there are totally realistic scenarios where a system with decent capacity budgeting could still be adversely affected by heartbeats. And what if a heartbeat client’s network link is degraded and there’s a super long time between first byte and last? Whether or not that client gets evicted from the cluster, if it’s basically slowloris-ing the server that can cause issues too.
- toast0 1y ago
  File descriptors are a limited resource, but the limits are huge. My little 2GB instances on GCP claim a limit of 1M; FreeBSD autotunes my 16GB servers to 0.5M (but I could increase it if I needed). I just don't know how you have a 1000 node system and you can't manage to heartbeat everything 2x a second; I don't think you need that many heartbeats in most systems, but it's just not that much work. The only way I can see it being a lot of work is if your nodes are very small; but do you really need a 1000 node esp8266 cluster and you can't get anything bigger for the management node? > And what if a heartbeat client’s network link is degraded and there’s a super long time between first byte and last? Whether or not that client gets evicted from the cluster, if it’s basically slowloris-ing the server that can cause issues too. How big is your heartbeat, is the response really going to be in multiple packets? With a 500ms heartbeat and a typical 3-5x no response => dead, you're going to hold onto the partial heartbeat for like 2 seconds.
  - zbentley 1y ago
    > The only way I can see it being a lot of work is if your nodes are very small Which is a totally valid use case for a huge-nodecount cluster! Geo distribution or similar use cases employ this pattern often. > you're going to hold onto the partial heartbeat for like 2 seconds And if a region goes down such that a large number of nodes are tarpit-heartbeating at the same time? Your cluster can fail to accept real traffic because it’s clogged with heartbeats. Like sure, generally 2krps should be fine. But in clusters this large it is totally reasonable to tune heartbeat behavior. I’ve seen plenty of real distributed system failures along these same exact lines.
schmichael 1y ago
> Really, setting the interval balances speed of detection/cost of slow detection vs cost of reacting to a momentary interruption. Another option is dynamically adjusting heartbeat interval based on cluster-size to ensure processing heartbeats has a fixed cost. That's what Nomad does and in my 10 year fuzzy memory heartbeating has never caused resource constraints on the schedulers: https://developer.hashicorp.com/nomad/docs/configuration/server#max_heartbeats_per_second https://developer.hashicorp.com/nomad/docs/configuration/ser... For reference clusters are commonly over 10k nodes and to my knowledge peak between 20k-30k. At least if anyone is running Nomad larger than that I'd love to hear from them! That being said the default of 50/s is probably too low, and the liveness tradeoff we force on users is probably not articulated clearly enough. As an off-the-shelf scheduler we can't encode liveness costs for our users unfortunately, but we try to offer the right knobs to adjust it including per-workload parameters for what to do when heartbeats fail: https://developer.hashicorp.com/nomad/docs/job-specification/disconnect https://developer.hashicorp.com/nomad/docs/job-specification... (Disclaimer: I'm on the Nomad team)
tstack 1y ago
> ... push it into overload ... Oh, oh, I get to talk about my favorite bug! I was working on network-booting servers with iPXE and we got a bug saying that things were working fine until the cluster size went over 4/5 machines. In a larger cluster, machines would not come up from a reboot. I thought QA was just being silly, why would the size of the cluster matter? I took a closer look and, sure enough, was able to reproduce the bug. Basically, the machine would sit there stuck trying to download the boot image over TCP from the server. After some investigation, it turned out to be related to the heartbeats sent between machines (they were ICMP pings). Since iPXE is a very nice and fancy bootloader, it will happily respond to ICMP pings. Note that, in order to do this, it would do an ARP to find address to send the response to. Unfortunately, the size of the ARP cache was pretty small since this was "embedded" software (take a guess how big the cache was...). Essentially, while iPXE was downloading the image, the address of the image server would get pushed out of the ARP cache by all these heartbeats. Thus, the download would suffer since it had to constantly pause to redo the ARP request. So, things would work with a smaller cluster size since the ARP cache was big enough to keep track of the download server and the peers in the cluster. I think I "fixed" it by responding to the ICMP using the source MAC address (making sure it wasn't broadcast) rather than doing an ARP.
- happyPersonR 1y ago
  Yeah broadcast with iPxe commonly has this issue, I’ve also run into it in my career more than once.
ignoramous 1y ago
> You can have that inside a datacenter too, maybe each node can only reach 75% of the other nodes, but A can reach B and B can reach C does not indicate A can reach C. Lots of fun times there At BigCloud in the early days, things went berserk with a gossip system when A could reach B but B couldn't reach A. Cloudflare hit something similar though they misclassified the failure mode: https://blog.cloudflare.com/a-byzantine-failure-in-the-real-world/ https://blog.cloudflare.com/a-byzantine-failure-in-the-real-...

by toast0 1y ago

macintux 1y ago

Kingsbury & Bailis's paper on the topic of network partitions: https://github.com/aphyr/partitions-post https://github.com/aphyr/partitions-post

karmakaze 1y ago

I've dealt with exactly this. We had a couple thousand webapp server instances that had connections to a MySQL database. Each one only polled its connection for liveliness once per second, but those were little interruptions that were poking at the servers and showed up on top time consuming request charts.

hinkley 1y ago

When systems were smaller I tried to push for the realization that I don’t need a heartbeat from a machine that is currently returning status 200 messages from 60 req/s. The evidence of work is already there, and more meaningful than the status check. We end up adding real work to the status checks often enough anyway, to make sure the database is still visible and other services. So inference has a lot of power that a heartbeat does not.

taeric 1y ago
I'm amused that you got pushback. Just going on the metaphor, the very way that we check heartbeats in people is by a proof of work in some other part of the body. Isn't like we are directly hooked into the heart. It isn't like the heart sends out an otherwise useless signal.

zbentley 1y ago

> If your 1000-node busy production environment is run so close to the edge that 2000 heartbeat messages per second, push it into overload, that's impressive resource scheduling. Eeeh I’m not so sure. The overhead of handling a hello-world heartbeat request is negligible, sure, but what about the overhead of having the connections open (file descriptors, maybe >1 per request), client tracking metadata for what are necessarily all different client location identifiers, and so on? That’s still cheap stuff, but at 2krps there are totally realistic scenarios where a system with decent capacity budgeting could still be adversely affected by heartbeats. And what if a heartbeat client’s network link is degraded and there’s a super long time between first byte and last? Whether or not that client gets evicted from the cluster, if it’s basically slowloris-ing the server that can cause issues too.

toast0 1y ago
File descriptors are a limited resource, but the limits are huge. My little 2GB instances on GCP claim a limit of 1M; FreeBSD autotunes my 16GB servers to 0.5M (but I could increase it if I needed). I just don't know how you have a 1000 node system and you can't manage to heartbeat everything 2x a second; I don't think you need that many heartbeats in most systems, but it's just not that much work. The only way I can see it being a lot of work is if your nodes are very small; but do you really need a 1000 node esp8266 cluster and you can't get anything bigger for the management node? > And what if a heartbeat client’s network link is degraded and there’s a super long time between first byte and last? Whether or not that client gets evicted from the cluster, if it’s basically slowloris-ing the server that can cause issues too. How big is your heartbeat, is the response really going to be in multiple packets? With a 500ms heartbeat and a typical 3-5x no response => dead, you're going to hold onto the partial heartbeat for like 2 seconds.
- zbentley 1y ago
  > The only way I can see it being a lot of work is if your nodes are very small Which is a totally valid use case for a huge-nodecount cluster! Geo distribution or similar use cases employ this pattern often. > you're going to hold onto the partial heartbeat for like 2 seconds And if a region goes down such that a large number of nodes are tarpit-heartbeating at the same time? Your cluster can fail to accept real traffic because it’s clogged with heartbeats. Like sure, generally 2krps should be fine. But in clusters this large it is totally reasonable to tune heartbeat behavior. I’ve seen plenty of real distributed system failures along these same exact lines.

schmichael 1y ago

> Really, setting the interval balances speed of detection/cost of slow detection vs cost of reacting to a momentary interruption. Another option is dynamically adjusting heartbeat interval based on cluster-size to ensure processing heartbeats has a fixed cost. That's what Nomad does and in my 10 year fuzzy memory heartbeating has never caused resource constraints on the schedulers: https://developer.hashicorp.com/nomad/docs/configuration/server#max_heartbeats_per_second https://developer.hashicorp.com/nomad/docs/configuration/ser... For reference clusters are commonly over 10k nodes and to my knowledge peak between 20k-30k. At least if anyone is running Nomad larger than that I'd love to hear from them! That being said the default of 50/s is probably too low, and the liveness tradeoff we force on users is probably not articulated clearly enough. As an off-the-shelf scheduler we can't encode liveness costs for our users unfortunately, but we try to offer the right knobs to adjust it including per-workload parameters for what to do when heartbeats fail: https://developer.hashicorp.com/nomad/docs/job-specification/disconnect https://developer.hashicorp.com/nomad/docs/job-specification... (Disclaimer: I'm on the Nomad team)

tstack 1y ago

> ... push it into overload ... Oh, oh, I get to talk about my favorite bug! I was working on network-booting servers with iPXE and we got a bug saying that things were working fine until the cluster size went over 4/5 machines. In a larger cluster, machines would not come up from a reboot. I thought QA was just being silly, why would the size of the cluster matter? I took a closer look and, sure enough, was able to reproduce the bug. Basically, the machine would sit there stuck trying to download the boot image over TCP from the server. After some investigation, it turned out to be related to the heartbeats sent between machines (they were ICMP pings). Since iPXE is a very nice and fancy bootloader, it will happily respond to ICMP pings. Note that, in order to do this, it would do an ARP to find address to send the response to. Unfortunately, the size of the ARP cache was pretty small since this was "embedded" software (take a guess how big the cache was...). Essentially, while iPXE was downloading the image, the address of the image server would get pushed out of the ARP cache by all these heartbeats. Thus, the download would suffer since it had to constantly pause to redo the ARP request. So, things would work with a smaller cluster size since the ARP cache was big enough to keep track of the download server and the peers in the cluster. I think I "fixed" it by responding to the ICMP using the source MAC address (making sure it wasn't broadcast) rather than doing an ARP.

happyPersonR 1y ago
Yeah broadcast with iPxe commonly has this issue, I’ve also run into it in my career more than once.

ignoramous 1y ago

> You can have that inside a datacenter too, maybe each node can only reach 75% of the other nodes, but A can reach B and B can reach C does not indicate A can reach C. Lots of fun times there At BigCloud in the early days, things went berserk with a gossip system when A could reach B but B couldn't reach A. Cloudflare hit something similar though they misclassified the failure mode: https://blog.cloudflare.com/a-byzantine-failure-in-the-real-world/ https://blog.cloudflare.com/a-byzantine-failure-in-the-real-...