3 ms·
While at WhatsApp, I learnt the importance of jitter the hard way. One day one of our backend services became temporarily unavailable and caused millions (prob
by psadri 5y ago
While at WhatsApp, I learnt the importance of jitter the hard way.
One day one of our backend services became temporarily unavailable and caused millions (probably in 10s of millions at this point) of connected web clients to go into jitter-less retry loops. The retry requests were arriving in coordinated waves and their amplitude was so high that we had trouble bringing the service back up, causing further disruptions. Once we had dug ourselves out, the first thing we did was to make sure all retry loops had jitter.
- nivertech 5y agoI "re-invented" jitter back in 2010, when I developed an Erlang/OTP-based Pub/Sub server supporting up to 1M concurrent long polling connections per single topic. After you published a message to that topic all 1M of them were closing a long polling HTTP request and immediately reconnecting. Which was causing a thundering herd problem [1], but I only learned this term much much later. The obvious solution was to add randomized delays to reconnects. Of course that was probably the easiest part of scaling that system. [1] https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem