6 ms·

> Hell, the author of this exact blog post has picked 10 seconds in all of his examples. That a FUCKING BAD timeout. It's far, FAR too short for many use cases.

> Hell, the author of this exact blog post has picked 10 seconds in all of his examples. That a FUCKING BAD timeout. It's far, FAR too short for many use cases. It isn't necessarily. It all depends on the use-case. If most of the operations are finishing in 5ms then the probability of something finishing after 10s are rather low - and timing out and retrying early is probably the way to go. Someone else in this thread recommended setting the timeout to around the P99 time that operations take. I think that's a reasonable starting point, even I might move it towards P99.9. I worked (and am still working) on adjusting timeouts for systems doing billions of requests/s. One takeaway from that is that the actual value of timeouts is often not too important if you look at one system in isolation. The latency distribution will be rather logarithmic. Most requests might e.g. finish in 20ms. Then you get a P99 at maybe 3 digit ms, and a P99.9 at 10s (example numbers). From there on it will make a minor difference in availability if you now set your actual timeout to 5s or to 120s - it might just be noise along your other error sources. However it makes sense to align the absolute timeout with timeouts of dependencies. E.g. if you have a chain of client -> service A -> service B and service A always times out first, then the client will get an error that service A is broken - but nobody can easily diagnose whether it was service A or service Bs fault. If service B times out first, the client can get an error message that indicates that. Therefore it makes sense if upstream service timeouts are shorter (even if only by a second). In the same model if the client times out first the services actually do only observe the client dropping the connection. They don't know whether the client timed out or cancelled the operation for other reasons. And therefore they might also not record that something in the service is actually not ideal. For that reason I would recommend setting client timeouts higher than service timeouts (if you are aware of them). However there is yet another exception to this thing, which are TCP connection timeouts. If you can configure them separately, it makes sense to have those rather low and performing multiple retries. That can improve overall latency, since dropped SYN packets will only be retried by the OS after 1s.

user5994461 6y ago
TCP takes 3-4 seconds to detect a dropped packet and retransmit. That puts an absolute minimum baseline of 5 seconds on timeout just to be able to send or receive a packet. If you're doing anything across datacenters, you have to take a minimum baseline of 10 - 15 seconds to account for extra latency on top. If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! There is a ton of issues caused by having too low timeouts set by developers (anything below 30 seconds). I had to reconfigure a ton of systems and libraries to have higher timeouts and ignore configuration passed by developers.
- doomslice 6y ago
  This doesn’t address much of your comment, but I work on systems with millions of req/s, but even with billions you can still sample to do logging and monitoring. But you’re right, we don’t care that every request works, just that 99.9% do.
- Matthias247 6y ago
  > If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. That's an assumption. Our customers care a lot. And we have sufficient monitoring in place. All recommendations I provided above where about maximizing availability, and provided based on the experiencing of improving the experience for lots of users. > I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! Without knowing much more about your system: If you are losing that amount of money for failed requests (which can e.g. happen due to random network blips) you are doing something wrong. You should invest into different strategies than increasing timeouts.
  - user5994461 6y ago
    It's inherent to payment systems to "lose" money on a failed request. A transaction involves two sides, either both sides agree that the transaction is completed or money is lost/duplicated. If the client decides to drop (timeout) and consider the transaction cancelled, while the server is processing it and will consider it done. That's a catastrophic issue that needs to be addressed. It is one of the most common bugs I've seen in the wild (root cause: too short timeouts). How to make highly critical systems reliable enough in the face of hardware and software issues is a complex topic. At this level this involves a holistic approach to get every component to cooperate together (timeout is a minor example). A HUGE amount of work is to detect errors, and more importantly to propagate errors across diverse stacks (software should be aware of database errors, services should detect other services failing).
    - 6y ago
jefftk 6y ago
> recommended setting the timeout to around the P99 time that operations take. I think that's a reasonable starting point, even I might move it towards P99.9. Why would you intentionally drop 1% or even 0.1% of requests?
- afiori 6y ago
  depending on the distribution of processing times[1] dropping slow request and retrying can improve latency. [1] https://en.wikipedia.org/wiki/Fat-tailed_distribution https://en.wikipedia.org/wiki/Fat-tailed_distribution
rrauenza 6y ago
What's the process of finding P99? Is it taking a bunch of samples, getting the standard deviation, and then calculating the value at which 99% of all possible samples would fall? This also assumes O(1) I assume... (I'm thinking of how to apply reasonable timeouts to background celery tasks)

user5994461 6y ago

TCP takes 3-4 seconds to detect a dropped packet and retransmit. That puts an absolute minimum baseline of 5 seconds on timeout just to be able to send or receive a packet. If you're doing anything across datacenters, you have to take a minimum baseline of 10 - 15 seconds to account for extra latency on top. If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! There is a ton of issues caused by having too low timeouts set by developers (anything below 30 seconds). I had to reconfigure a ton of systems and libraries to have higher timeouts and ignore configuration passed by developers.

doomslice 6y ago
This doesn’t address much of your comment, but I work on systems with millions of req/s, but even with billions you can still sample to do logging and monitoring. But you’re right, we don’t care that every request works, just that 99.9% do.
Matthias247 6y ago
> If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. That's an assumption. Our customers care a lot. And we have sufficient monitoring in place. All recommendations I provided above where about maximizing availability, and provided based on the experiencing of improving the experience for lots of users. > I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! Without knowing much more about your system: If you are losing that amount of money for failed requests (which can e.g. happen due to random network blips) you are doing something wrong. You should invest into different strategies than increasing timeouts.
- user5994461 6y ago
  It's inherent to payment systems to "lose" money on a failed request. A transaction involves two sides, either both sides agree that the transaction is completed or money is lost/duplicated. If the client decides to drop (timeout) and consider the transaction cancelled, while the server is processing it and will consider it done. That's a catastrophic issue that needs to be addressed. It is one of the most common bugs I've seen in the wild (root cause: too short timeouts). How to make highly critical systems reliable enough in the face of hardware and software issues is a complex topic. At this level this involves a holistic approach to get every component to cooperate together (timeout is a minor example). A HUGE amount of work is to detect errors, and more importantly to propagate errors across diverse stacks (software should be aware of database errors, services should detect other services failing).
  - 6y ago

jefftk 6y ago

> recommended setting the timeout to around the P99 time that operations take. I think that's a reasonable starting point, even I might move it towards P99.9. Why would you intentionally drop 1% or even 0.1% of requests?

afiori 6y ago
depending on the distribution of processing times[1] dropping slow request and retrying can improve latency. [1] https://en.wikipedia.org/wiki/Fat-tailed_distribution https://en.wikipedia.org/wiki/Fat-tailed_distribution

rrauenza 6y ago

What's the process of finding P99? Is it taking a bunch of samples, getting the standard deviation, and then calculating the value at which 99% of all possible samples would fall? This also assumes O(1) I assume... (I'm thinking of how to apply reasonable timeouts to background celery tasks)