4 ms·

The thing that blew my mind is learning that retry loops paradoxically reduce reliability by introducing bi-stable scenarios where a working system at just over

• mr_tristan 4y ago
  That's an excellent example. Mostly because I've probably had to replace simple retry loops more than once after they triggered various kinds of death spirals. Even recently, an engineer on my team spotted a retry technique, where the system tried to check if the DB's replication mechanism was slowing down, then would just re-enqueue the message for later. This ended up doing a bonkers amount of wasted effort because when DB replication started getting overwhelmed, messages would just constantly be re-enqueued and not ever processed. So the queue system ended up just having a bonkers amount of messages that were just popping off the queue, doing nothing, and hopping right back on.
• derangedHorse 4y ago
  Working at scale is truly a different ball game. A 1% yearly failure rate from a service serving around 1m users (a sizable but not huge number in today's age) will end up failing for 1000 different users on average. If the service is responsible for something like payments then 1000 different customers probably will end up calling support, support will have to identify and handle this anticipated failure mode, etc.. Sure error handling strategies like exponential backoff and microservice paradigms like "sagas" try to teach us more error-avoidant approaches, but sometimes it can be a lot to handle.