3 ms·
> The only way I can see it being a lot of work is if your nodes are very small Which is a totally valid use case for a huge-nodecount cluster! Geo distributio
by zbentley 1y ago
> The only way I can see it being a lot of work is if your nodes are very small
Which is a totally valid use case for a huge-nodecount cluster! Geo distribution or similar use cases employ this pattern often.
> you're going to hold onto the partial heartbeat for like 2 seconds
And if a region goes down such that a large number of nodes are tarpit-heartbeating at the same time? Your cluster can fail to accept real traffic because it’s clogged with heartbeats.
Like sure, generally 2krps should be fine. But in clusters this large it is totally reasonable to tune heartbeat behavior. I’ve seen plenty of real distributed system failures along these same exact lines.