3 ms·

I was watching the old 2014 video: https://youtu.be/4fFDFbi3toc?t=1081 https://youtu.be/4fFDFbi3toc?t=1081 I've included the timestamp of the relevant bit. He

I was watching the old 2014 video: https://youtu.be/4fFDFbi3toc?t=1081 https://youtu.be/4fFDFbi3toc?t=1081 I've included the timestamp of the relevant bit. He is talking about swizzling, which is like clogging++. He says they take a subset of the network connections, and have them timeout, but then bring them all back in reverse order, and that this somehow exposes far more bugs than other methods. This seems like a very interesting "well, that's funny!" moment that might lead to understanding something foundational about how distributed systems fail. Does anyone have more insight on this?

hinkley 5y ago
That sounds reasonable to me. In a cascading failure where network congestion ramps up (eg, old school NFS), the machines with the worst network bandwidth or latency would time out first, and be the last to recover. In a heterogeneous cluster, I might expect the fastest machines to die last, and either a human operator concentrating on getting the most capacity back online first, or just automation getting the fastest one online first, since a faster machine can go from hard down to green health check sooner than the slow machine. Not precisely reverse chronological order, but certainly with large statistical clusters. And on top of that, exponential back-off would cause the same phenomenon. But if your question is "why more bugs that way?" your guess is as good as mine. Mine is 'consensus is hard.'.