3 ms·

The answer is to "kick retries up the stack"; when you fail to reach a service, you return a 503 and have your clients retry, to avoid a case where every servic

The answer is to "kick retries up the stack"; when you fail to reach a service, you return a 503 and have your clients retry, to avoid a case where every service in the stack starts retrying all at once and causes a massive increase in traffic. IMHO you should only add retries if it proves necessary in practice to reach your SLA, if you're building something that isn't itself triggered by an RPC, or if you're performing an operation that can never be made idempotent.

dmux 4y ago
I like the idea of pushing the retry up to the caller, but in a lot of apps I've seen they've built up an in-memory object-graph from the results of calling out to other services. Wouldn't failing due to one bad service and asking the caller to retry result in every other service being unnecessarily hit?
- CBLT 4y ago
  I had a tech lead who vehemently agreed with the parent commenter (retry from the top), but I ended up learning different lessons. * Differentiate retryable and non-retryable errors. If the service can't return success because the DB it queries is borked, it should send a non-retryable error. Then it won't get overwhelmed by retries from upstream. * Retry configuration should have sane defaults. Even "retry once" is too often; many services aren't overprovisioned for 100% increase in traffic. What ended up working here for us was having the retry module collect req/sec statistics, then only allowing 20% of that number in retries/second. Individual requests can be retried twice. That was small enough to not push over any services, but enough retries to compensate for garden-variety unavailability. * Services shouldn't serve requests first-come-first-served under load. When SHTF, fairness means everyone has to suffer long delays, often much longer than rpc timeout. Instead, serve them in an unfair order - the most recent requests to come in are the most likely to have a caller still interested in them. Answer those! * Use headless services in kubernetes. By exposing the replicaset to the client, the client can load balance itself more intelligently. Retries should go to different replicas than the failing request. Furthermore, you can perform request hedging to different replicas than the lagging request. * Define a degraded form of your in-memory object-graph. If a feature is optional to the core business flow, it shouldn't take down your whole product. This one is a lot more involved. We needed custom monitoring for degraded responses, in-memory collection and storage of "guesses" to substitute for degraded portions of the object graph, as well as some other work I can't think of right now. This does enable an organization to compartmentalize better, having faster, less fearful deploys of newer initiatives.
  - sokoloff 4y ago
    It seems like knowing the difference between a retryable and non-retryable error is itself difficult (perhaps impossible in practice). DB is unreachable? Is it the DB or my host? If it’s the DB, maybe it’s not retryable, but if it’s me (and I’m load-balanced), it is probably retryable.
    - savy91 4y ago
      IMHO even if db is not reachable, it should be retriable.
electroly 4y ago
Sometimes you want the exact opposite, though. Consider an endpoint that makes 100 behind-the-scenes requests (say, to S3). You absolutely want to retry at the lowest level, not the highest level. You could fail on the 99th request. If you kick it up, the caller will retry and you'll do those 99 requests again, instead of just retrying the one that failed. With enough requests, there's a point where you're unlikely ever to succeed without one of the calls failing if you restart from the beginning every time. I don't think you can "one size fits all" this, and that's one of the reasons retries are hard. I use S3 as an example because it has a comparatively high random failure rate. You MUST implement timeouts and retries if you're using S3 as a backend.
- qu4z-2 4y ago
  Lisp's conditions and restarts are great for this, so you can make the policy decision at a high level while not fully unrolling the stack and allowing retries to be done in the original context if that's the verdict.
- deleted 4y ago
  [deleted]
- jasonhansel 4y ago
  That's why I said "except if your SLA requires it." If you've agreed--with other teams, with outside clients, or just as a project goal--that your service should work on 99.9% of calls, and you find in practice that you need to retry S3 calls to meet that target, then adding retries is reasonable. If the problem is just that the 99 calls could overload downstream services on retries, the ideal solution is to add rate-limiting, though admittedly that is an imprecise science.
  - benlivengood 4y ago
    There's never a reason that a human needs to read an error message caused merely by Kubernetes restarting a pod in the normal course of operations. Passing every error up the stack causes people to be confused by 500 status errors for trivially retry-able calls. Browsers (and TCP) natively implement retries despite there being no SLAs in place.
mcguire 4y ago
That's a variation of ye olde end-to-end argument. In fact, this subthread seems to be recreating many of the network protocol design ideas from the ancient days.

dmux 4y ago

I like the idea of pushing the retry up to the caller, but in a lot of apps I've seen they've built up an in-memory object-graph from the results of calling out to other services. Wouldn't failing due to one bad service and asking the caller to retry result in every other service being unnecessarily hit?

CBLT 4y ago
I had a tech lead who vehemently agreed with the parent commenter (retry from the top), but I ended up learning different lessons. * Differentiate retryable and non-retryable errors. If the service can't return success because the DB it queries is borked, it should send a non-retryable error. Then it won't get overwhelmed by retries from upstream. * Retry configuration should have sane defaults. Even "retry once" is too often; many services aren't overprovisioned for 100% increase in traffic. What ended up working here for us was having the retry module collect req/sec statistics, then only allowing 20% of that number in retries/second. Individual requests can be retried twice. That was small enough to not push over any services, but enough retries to compensate for garden-variety unavailability. * Services shouldn't serve requests first-come-first-served under load. When SHTF, fairness means everyone has to suffer long delays, often much longer than rpc timeout. Instead, serve them in an unfair order - the most recent requests to come in are the most likely to have a caller still interested in them. Answer those! * Use headless services in kubernetes. By exposing the replicaset to the client, the client can load balance itself more intelligently. Retries should go to different replicas than the failing request. Furthermore, you can perform request hedging to different replicas than the lagging request. * Define a degraded form of your in-memory object-graph. If a feature is optional to the core business flow, it shouldn't take down your whole product. This one is a lot more involved. We needed custom monitoring for degraded responses, in-memory collection and storage of "guesses" to substitute for degraded portions of the object graph, as well as some other work I can't think of right now. This does enable an organization to compartmentalize better, having faster, less fearful deploys of newer initiatives.
- sokoloff 4y ago
  It seems like knowing the difference between a retryable and non-retryable error is itself difficult (perhaps impossible in practice). DB is unreachable? Is it the DB or my host? If it’s the DB, maybe it’s not retryable, but if it’s me (and I’m load-balanced), it is probably retryable.
  - savy91 4y ago
    IMHO even if db is not reachable, it should be retriable.

electroly 4y ago

Sometimes you want the exact opposite, though. Consider an endpoint that makes 100 behind-the-scenes requests (say, to S3). You absolutely want to retry at the lowest level, not the highest level. You could fail on the 99th request. If you kick it up, the caller will retry and you'll do those 99 requests again, instead of just retrying the one that failed. With enough requests, there's a point where you're unlikely ever to succeed without one of the calls failing if you restart from the beginning every time. I don't think you can "one size fits all" this, and that's one of the reasons retries are hard. I use S3 as an example because it has a comparatively high random failure rate. You MUST implement timeouts and retries if you're using S3 as a backend.

qu4z-2 4y ago
Lisp's conditions and restarts are great for this, so you can make the policy decision at a high level while not fully unrolling the stack and allowing retries to be done in the original context if that's the verdict.
deleted 4y ago
[deleted]
jasonhansel 4y ago
That's why I said "except if your SLA requires it." If you've agreed--with other teams, with outside clients, or just as a project goal--that your service should work on 99.9% of calls, and you find in practice that you need to retry S3 calls to meet that target, then adding retries is reasonable. If the problem is just that the 99 calls could overload downstream services on retries, the ideal solution is to add rate-limiting, though admittedly that is an imprecise science.
- benlivengood 4y ago
  There's never a reason that a human needs to read an error message caused merely by Kubernetes restarting a pod in the normal course of operations. Passing every error up the stack causes people to be confused by 500 status errors for trivially retry-able calls. Browsers (and TCP) natively implement retries despite there being no SLAs in place.

mcguire 4y ago

That's a variation of ye olde end-to-end argument. In fact, this subthread seems to be recreating many of the network protocol design ideas from the ancient days.