3 ms·

Retries aren't the only option if you control (or can at least influence) the client. Fallback sources and sane (or customer preferential) defaults are some add

Retries aren't the only option if you control (or can at least influence) the client. Fallback sources and sane (or customer preferential) defaults are some additional options when the primary source of truth is unavailable. Both of these bypass thundering heard situations where recovery can be difficult even if the application returns to full health. As a concrete example, when I worked at a large online retailer, we had to check certain customer properties during checkout. In the optimal case, we went to a service which fronted a consistent, transactional database. If that service failed or timed out, we had the option to hit a fast cache which generally was only a few milliseconds out of date. Finally, we could use less authoritative data that came from the application earlier, and if present, would always be in the customer's favor. That made it unlikely that our service ever appeared "down" regardless of the state of the dependencies.

robbles 9y ago
Why wouldn't you start with the cache? Isn't the consistent database only useful if you require it to be used, and don't fall back to a less consistent alternative?
- jonhohle 9y ago
  Based on previous decisions made by customers that were processed within the lifetime of the same request, there was some probability that the cache could be out of date. Typically, there was enough context within the request to make assumptions about the state of the authoritative store if data hadn't propagated to the cache in time. We eventually moved to using that initially. edit: I should add, though, the order of the datasources wasn't critical to the strategy: have a primary data source, backup data source, use any data available in the request context, and finally err in the customer's favor. That's 4 chances to get things right, none of which include retrying.

robbles 9y ago

Why wouldn't you start with the cache? Isn't the consistent database only useful if you require it to be used, and don't fall back to a less consistent alternative?

jonhohle 9y ago
Based on previous decisions made by customers that were processed within the lifetime of the same request, there was some probability that the cache could be out of date. Typically, there was enough context within the request to make assumptions about the state of the authoritative store if data hadn't propagated to the cache in time. We eventually moved to using that initially. edit: I should add, though, the order of the datasources wasn't critical to the strategy: have a primary data source, backup data source, use any data available in the request context, and finally err in the customer's favor. That's 4 chances to get things right, none of which include retrying.