3 ms·

> If your 1000-node busy production environment is run so close to the edge that 2000 heartbeat messages per second, push it into overload, that's impressive re

> If your 1000-node busy production environment is run so close to the edge that 2000 heartbeat messages per second, push it into overload, that's impressive resource scheduling. Eeeh I’m not so sure. The overhead of handling a hello-world heartbeat request is negligible, sure, but what about the overhead of having the connections open (file descriptors, maybe >1 per request), client tracking metadata for what are necessarily all different client location identifiers, and so on? That’s still cheap stuff, but at 2krps there are totally realistic scenarios where a system with decent capacity budgeting could still be adversely affected by heartbeats. And what if a heartbeat client’s network link is degraded and there’s a super long time between first byte and last? Whether or not that client gets evicted from the cluster, if it’s basically slowloris-ing the server that can cause issues too.

toast0 1y ago
File descriptors are a limited resource, but the limits are huge. My little 2GB instances on GCP claim a limit of 1M; FreeBSD autotunes my 16GB servers to 0.5M (but I could increase it if I needed). I just don't know how you have a 1000 node system and you can't manage to heartbeat everything 2x a second; I don't think you need that many heartbeats in most systems, but it's just not that much work. The only way I can see it being a lot of work is if your nodes are very small; but do you really need a 1000 node esp8266 cluster and you can't get anything bigger for the management node? > And what if a heartbeat client’s network link is degraded and there’s a super long time between first byte and last? Whether or not that client gets evicted from the cluster, if it’s basically slowloris-ing the server that can cause issues too. How big is your heartbeat, is the response really going to be in multiple packets? With a 500ms heartbeat and a typical 3-5x no response => dead, you're going to hold onto the partial heartbeat for like 2 seconds.
- zbentley 1y ago
  > The only way I can see it being a lot of work is if your nodes are very small Which is a totally valid use case for a huge-nodecount cluster! Geo distribution or similar use cases employ this pattern often. > you're going to hold onto the partial heartbeat for like 2 seconds And if a region goes down such that a large number of nodes are tarpit-heartbeating at the same time? Your cluster can fail to accept real traffic because it’s clogged with heartbeats. Like sure, generally 2krps should be fine. But in clusters this large it is totally reasonable to tune heartbeat behavior. I’ve seen plenty of real distributed system failures along these same exact lines.
- lokar 1y ago
  For Linux, the limit on FDs is just memory.
- MaulingMonkey 1y ago
  > File descriptors are a limited resource, but the limits are huge. Maybe in the linuxy servery context we're talking about, but I'll note I've encountered EMFILE and ENFILE errors attempting to open a mere couple hundred files on iOS simultaniously. And while my own experience is over a decade old, here's someone hitting those limits this year: https://github.com/bevyengine/bevy/pull/17377 https://github.com/bevyengine/bevy/pull/17377 (Context for my own encounter: gamedev, with massively parallel asset streaming and multiple layers of direct blocking dependencies in deserialization. One of my hackarounds involved temporarily closing the least recently read files, and then reopening them on demand in our VFS abstraction. Of course, for non-dev builds, you can just use a pack file and `pread` a single fd from multiple threads, but loose files were handy for incremental dev updates...)

toast0 1y ago

File descriptors are a limited resource, but the limits are huge. My little 2GB instances on GCP claim a limit of 1M; FreeBSD autotunes my 16GB servers to 0.5M (but I could increase it if I needed). I just don't know how you have a 1000 node system and you can't manage to heartbeat everything 2x a second; I don't think you need that many heartbeats in most systems, but it's just not that much work. The only way I can see it being a lot of work is if your nodes are very small; but do you really need a 1000 node esp8266 cluster and you can't get anything bigger for the management node? > And what if a heartbeat client’s network link is degraded and there’s a super long time between first byte and last? Whether or not that client gets evicted from the cluster, if it’s basically slowloris-ing the server that can cause issues too. How big is your heartbeat, is the response really going to be in multiple packets? With a 500ms heartbeat and a typical 3-5x no response => dead, you're going to hold onto the partial heartbeat for like 2 seconds.

zbentley 1y ago
> The only way I can see it being a lot of work is if your nodes are very small Which is a totally valid use case for a huge-nodecount cluster! Geo distribution or similar use cases employ this pattern often. > you're going to hold onto the partial heartbeat for like 2 seconds And if a region goes down such that a large number of nodes are tarpit-heartbeating at the same time? Your cluster can fail to accept real traffic because it’s clogged with heartbeats. Like sure, generally 2krps should be fine. But in clusters this large it is totally reasonable to tune heartbeat behavior. I’ve seen plenty of real distributed system failures along these same exact lines.
lokar 1y ago
For Linux, the limit on FDs is just memory.
MaulingMonkey 1y ago
> File descriptors are a limited resource, but the limits are huge. Maybe in the linuxy servery context we're talking about, but I'll note I've encountered EMFILE and ENFILE errors attempting to open a mere couple hundred files on iOS simultaniously. And while my own experience is over a decade old, here's someone hitting those limits this year: https://github.com/bevyengine/bevy/pull/17377 https://github.com/bevyengine/bevy/pull/17377 (Context for my own encounter: gamedev, with massively parallel asset streaming and multiple layers of direct blocking dependencies in deserialization. One of my hackarounds involved temporarily closing the least recently read files, and then reopening them on demand in our VFS abstraction. Of course, for non-dev builds, you can just use a pack file and `pread` a single fd from multiple threads, but loose files were handy for incremental dev updates...)