5 ms·

No! The danger in forcing programmers to pick a timeout is that they will pick the wrong value, most often a too short timeout, because they have been testing t

No! The danger in forcing programmers to pick a timeout is that they will pick the wrong value, most often a too short timeout, because they have been testing their software on a super-fast internal network and haven't considered the poor users in the real world. Case in point: Google's Waze. If I have a slow mobile connection (e.g. edge or even 3g), Waze will repeatedly fail to load a driving route. It will think for a few seconds at most, then timeout and tell me there was a problem. If it only would wait a few more seconds to load, then the app would be useful. Instead, due to their crappy choice of timeouts, the app becomes useless.

giancarlostoro 6y ago
This is probably it. Production timeouts vs developer timeouts. What a shame about Waze. I wanted to try it out but then realized its just owned by Google now so its kinda pointless for me to bother.
horsawlarway 6y ago
I strongly second this "No!" for both of the JS examples he makes. There was no way to set a timeout in Fetch because the browser, acting as the user's agent, has a sane default (~75 seconds on average, but it varies by browser and platform). Developers often pick TERRIBLE values for timeouts when left to their own devices. Hell, the author of this exact blog post has picked 10 seconds in all of his examples. That a FUCKING BAD timeout. It's far, FAR too short for many use cases.
- Matthias247 6y ago
  > Hell, the author of this exact blog post has picked 10 seconds in all of his examples. That a FUCKING BAD timeout. It's far, FAR too short for many use cases. It isn't necessarily. It all depends on the use-case. If most of the operations are finishing in 5ms then the probability of something finishing after 10s are rather low - and timing out and retrying early is probably the way to go. Someone else in this thread recommended setting the timeout to around the P99 time that operations take. I think that's a reasonable starting point, even I might move it towards P99.9. I worked (and am still working) on adjusting timeouts for systems doing billions of requests/s. One takeaway from that is that the actual value of timeouts is often not too important if you look at one system in isolation. The latency distribution will be rather logarithmic. Most requests might e.g. finish in 20ms. Then you get a P99 at maybe 3 digit ms, and a P99.9 at 10s (example numbers). From there on it will make a minor difference in availability if you now set your actual timeout to 5s or to 120s - it might just be noise along your other error sources. However it makes sense to align the absolute timeout with timeouts of dependencies. E.g. if you have a chain of client -> service A -> service B and service A always times out first, then the client will get an error that service A is broken - but nobody can easily diagnose whether it was service A or service Bs fault. If service B times out first, the client can get an error message that indicates that. Therefore it makes sense if upstream service timeouts are shorter (even if only by a second). In the same model if the client times out first the services actually do only observe the client dropping the connection. They don't know whether the client timed out or cancelled the operation for other reasons. And therefore they might also not record that something in the service is actually not ideal. For that reason I would recommend setting client timeouts higher than service timeouts (if you are aware of them). However there is yet another exception to this thing, which are TCP connection timeouts. If you can configure them separately, it makes sense to have those rather low and performing multiple retries. That can improve overall latency, since dropped SYN packets will only be retried by the OS after 1s.
  - user5994461 6y ago
    TCP takes 3-4 seconds to detect a dropped packet and retransmit. That puts an absolute minimum baseline of 5 seconds on timeout just to be able to send or receive a packet. If you're doing anything across datacenters, you have to take a minimum baseline of 10 - 15 seconds to account for extra latency on top. If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! There is a ton of issues caused by having too low timeouts set by developers (anything below 30 seconds). I had to reconfigure a ton of systems and libraries to have higher timeouts and ignore configuration passed by developers.
    - doomslice 6y ago
      This doesn’t address much of your comment, but I work on systems with millions of req/s, but even with billions you can still sample to do logging and monitoring. But you’re right, we don’t care that every request works, just that 99.9% do.
    - Matthias247 6y ago
      > If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. That's an assumption. Our customers care a lot. And we have sufficient monitoring in place. All recommendations I provided above where about maximizing availability, and provided based on the experiencing of improving the experience for lots of users. > I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! Without knowing much more about your system: If you are losing that amount of money for failed requests (which can e.g. happen due to random network blips) you are doing something wrong. You should invest into different strategies than increasing timeouts.
      - user5994461 6y ago
hrktb 6y ago
Is the argument that a call that could get stuck for a few minutes is better than a “wrong“ dozens of second value ? Even as a user I feel it’s a waste of precious resource (including my time). It’s like waiting at the register until the shop closes down because the employee had to go somewhere, instead of giving up and trying the next open register. I’d think infinity is not a valid state. For waze’s case, I supposed their priority is not on salvaging the 1% longest request (though critical to you), and instead preserve server resources for the 99% faster clients. That’s not a “wrong” value on their side, and probably have been carefully tailored to get the right tradeoff.
- Dylan16807 6y ago
  > For waze’s case, I supposed their priority is not on salvaging the 1% longest request (though critical to you), and instead preserve server resources for the 99% faster clients. What resources? Buffering a response takes a minuscule amount, and if even a tiny fraction of people try again it will waste far more. And even if it did take more in total, it would not be by much. This justification for saying it's not a wrong value is very weak.
- user5994461 6y ago
  A too short timeout is more problematic than no timeout because it breaks the application. Let's say 10 seconds, typical intuitive but bad timeout. This will cause requests to fail for no reason other than users are in Asia or Africa, high latency. This will break the application when it's used or deployed across datacenters because high latency. This will cause requests to fail when the server is a bit busy (couple seconds more to process requests). Worse, it will cause chain reactions under load, creating more retries and even more load, causing other services/servers to timeout too. Better go for a long timeout. A long timeout doesn't break the application.
  - rfoo 6y ago
    > A long timeout doesn't break the application. I'm pretty sure infinite timeout also breaks the application, in a way people rarely realize that it is because of the timeout. People would rather think it "just didn't work, don't know why" instead of being very clever and realized "it must be low timeouts!!!"
acdha 6y ago
When I implement this, I typically use separate thresholds for the entire request and time since last progress or some rolling average transfer rates. Letting a slow transfer complete is useful both for what you mentioned and also reducing server congestion but you do need to detect failures where the remote end goes silent (server failure, network roaming, etc.) without tearing down the connection.
DangitBobby 6y ago
I straight up couldn't use terraform on DSL while visiting family over Christmas because of this. It would be chugging along at what appeared to me a reasonable speed, but either one of Google's services or terraform itself would decide I was taking too long and stop. I was unable to work that week.
- spockz 6y ago
  Maybe it is a good idea for next time to provision a vm/machine closer to where you are deploying? It would also prevent loss of progress when the connection got dropped or something.
dheera 6y ago
I've seen many cases on mobile web pages where it tries to connect, hangs on a progress bar, my phone loses and regains signal during the process, it still doesn't finish loading, but a pull-to-refresh makes it load instantly. That's an example where a timeout and retry would have fixed the problem. If it had been an API call behind an app, it would have hung indefinitely. Some libraries sadly have their default timeouts set to infinite.

giancarlostoro 6y ago

This is probably it. Production timeouts vs developer timeouts. What a shame about Waze. I wanted to try it out but then realized its just owned by Google now so its kinda pointless for me to bother.

horsawlarway 6y ago

I strongly second this "No!" for both of the JS examples he makes. There was no way to set a timeout in Fetch because the browser, acting as the user's agent, has a sane default (~75 seconds on average, but it varies by browser and platform). Developers often pick TERRIBLE values for timeouts when left to their own devices. Hell, the author of this exact blog post has picked 10 seconds in all of his examples. That a FUCKING BAD timeout. It's far, FAR too short for many use cases.

Matthias247 6y ago
> Hell, the author of this exact blog post has picked 10 seconds in all of his examples. That a FUCKING BAD timeout. It's far, FAR too short for many use cases. It isn't necessarily. It all depends on the use-case. If most of the operations are finishing in 5ms then the probability of something finishing after 10s are rather low - and timing out and retrying early is probably the way to go. Someone else in this thread recommended setting the timeout to around the P99 time that operations take. I think that's a reasonable starting point, even I might move it towards P99.9. I worked (and am still working) on adjusting timeouts for systems doing billions of requests/s. One takeaway from that is that the actual value of timeouts is often not too important if you look at one system in isolation. The latency distribution will be rather logarithmic. Most requests might e.g. finish in 20ms. Then you get a P99 at maybe 3 digit ms, and a P99.9 at 10s (example numbers). From there on it will make a minor difference in availability if you now set your actual timeout to 5s or to 120s - it might just be noise along your other error sources. However it makes sense to align the absolute timeout with timeouts of dependencies. E.g. if you have a chain of client -> service A -> service B and service A always times out first, then the client will get an error that service A is broken - but nobody can easily diagnose whether it was service A or service Bs fault. If service B times out first, the client can get an error message that indicates that. Therefore it makes sense if upstream service timeouts are shorter (even if only by a second). In the same model if the client times out first the services actually do only observe the client dropping the connection. They don't know whether the client timed out or cancelled the operation for other reasons. And therefore they might also not record that something in the service is actually not ideal. For that reason I would recommend setting client timeouts higher than service timeouts (if you are aware of them). However there is yet another exception to this thing, which are TCP connection timeouts. If you can configure them separately, it makes sense to have those rather low and performing multiple retries. That can improve overall latency, since dropped SYN packets will only be retried by the OS after 1s.
- user5994461 6y ago
  TCP takes 3-4 seconds to detect a dropped packet and retransmit. That puts an absolute minimum baseline of 5 seconds on timeout just to be able to send or receive a packet. If you're doing anything across datacenters, you have to take a minimum baseline of 10 - 15 seconds to account for extra latency on top. If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! There is a ton of issues caused by having too low timeouts set by developers (anything below 30 seconds). I had to reconfigure a ton of systems and libraries to have higher timeouts and ignore configuration passed by developers.
  - doomslice 6y ago
    This doesn’t address much of your comment, but I work on systems with millions of req/s, but even with billions you can still sample to do logging and monitoring. But you’re right, we don’t care that every request works, just that 99.9% do.
  - Matthias247 6y ago
    > If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. That's an assumption. Our customers care a lot. And we have sufficient monitoring in place. All recommendations I provided above where about maximizing availability, and provided based on the experiencing of improving the experience for lots of users. > I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! Without knowing much more about your system: If you are losing that amount of money for failed requests (which can e.g. happen due to random network blips) you are doing something wrong. You should invest into different strategies than increasing timeouts.
    - user5994461 6y ago

hrktb 6y ago

Is the argument that a call that could get stuck for a few minutes is better than a “wrong“ dozens of second value ? Even as a user I feel it’s a waste of precious resource (including my time). It’s like waiting at the register until the shop closes down because the employee had to go somewhere, instead of giving up and trying the next open register. I’d think infinity is not a valid state. For waze’s case, I supposed their priority is not on salvaging the 1% longest request (though critical to you), and instead preserve server resources for the 99% faster clients. That’s not a “wrong” value on their side, and probably have been carefully tailored to get the right tradeoff.

Dylan16807 6y ago
> For waze’s case, I supposed their priority is not on salvaging the 1% longest request (though critical to you), and instead preserve server resources for the 99% faster clients. What resources? Buffering a response takes a minuscule amount, and if even a tiny fraction of people try again it will waste far more. And even if it did take more in total, it would not be by much. This justification for saying it's not a wrong value is very weak.
user5994461 6y ago
A too short timeout is more problematic than no timeout because it breaks the application. Let's say 10 seconds, typical intuitive but bad timeout. This will cause requests to fail for no reason other than users are in Asia or Africa, high latency. This will break the application when it's used or deployed across datacenters because high latency. This will cause requests to fail when the server is a bit busy (couple seconds more to process requests). Worse, it will cause chain reactions under load, creating more retries and even more load, causing other services/servers to timeout too. Better go for a long timeout. A long timeout doesn't break the application.
- rfoo 6y ago
  > A long timeout doesn't break the application. I'm pretty sure infinite timeout also breaks the application, in a way people rarely realize that it is because of the timeout. People would rather think it "just didn't work, don't know why" instead of being very clever and realized "it must be low timeouts!!!"

acdha 6y ago

When I implement this, I typically use separate thresholds for the entire request and time since last progress or some rolling average transfer rates. Letting a slow transfer complete is useful both for what you mentioned and also reducing server congestion but you do need to detect failures where the remote end goes silent (server failure, network roaming, etc.) without tearing down the connection.

DangitBobby 6y ago

I straight up couldn't use terraform on DSL while visiting family over Christmas because of this. It would be chugging along at what appeared to me a reasonable speed, but either one of Google's services or terraform itself would decide I was taking too long and stop. I was unable to work that week.

spockz 6y ago
Maybe it is a good idea for next time to provision a vm/machine closer to where you are deploying? It would also prevent loss of progress when the connection got dropped or something.

dheera 6y ago

I've seen many cases on mobile web pages where it tries to connect, hangs on a progress bar, my phone loses and regains signal during the process, it still doesn't finish loading, but a pull-to-refresh makes it load instantly. That's an example where a timeout and retry would have fixed the problem. If it had been an API call behind an app, it would have hung indefinitely. Some libraries sadly have their default timeouts set to infinite.