4 ms·

TCP takes 3-4 seconds to detect a dropped packet and retransmit. That puts an absolute minimum baseline of 5 seconds on timeout just to be able to send or recei

TCP takes 3-4 seconds to detect a dropped packet and retransmit. That puts an absolute minimum baseline of 5 seconds on timeout just to be able to send or receive a packet. If you're doing anything across datacenters, you have to take a minimum baseline of 10 - 15 seconds to account for extra latency on top. If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! There is a ton of issues caused by having too low timeouts set by developers (anything below 30 seconds). I had to reconfigure a ton of systems and libraries to have higher timeouts and ignore configuration passed by developers.

doomslice 6y ago
This doesn’t address much of your comment, but I work on systems with millions of req/s, but even with billions you can still sample to do logging and monitoring. But you’re right, we don’t care that every request works, just that 99.9% do.
Matthias247 6y ago
> If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. That's an assumption. Our customers care a lot. And we have sufficient monitoring in place. All recommendations I provided above where about maximizing availability, and provided based on the experiencing of improving the experience for lots of users. > I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! Without knowing much more about your system: If you are losing that amount of money for failed requests (which can e.g. happen due to random network blips) you are doing something wrong. You should invest into different strategies than increasing timeouts.
- user5994461 6y ago
  It's inherent to payment systems to "lose" money on a failed request. A transaction involves two sides, either both sides agree that the transaction is completed or money is lost/duplicated. If the client decides to drop (timeout) and consider the transaction cancelled, while the server is processing it and will consider it done. That's a catastrophic issue that needs to be addressed. It is one of the most common bugs I've seen in the wild (root cause: too short timeouts). How to make highly critical systems reliable enough in the face of hardware and software issues is a complex topic. At this level this involves a holistic approach to get every component to cooperate together (timeout is a minor example). A HUGE amount of work is to detect errors, and more importantly to propagate errors across diverse stacks (software should be aware of database errors, services should detect other services failing).
  - KallDrexx 6y ago
    This doesn't make sense to me. If there is a real risk that a timeout can happen (and there always is) then the payment system should be implementing a two phase commit.
    - earthboundkid 6y ago
      Doesn’t that just create a Byzantine generals problem? My understanding is that real payment systems solve the problem by just taking a day to finalize transactions…
      - KallDrexx 6y ago
        I don't know what the Byzantine generals problem is. Two stage commit is important because it has: 1) Predefined transaction id prior to final submission that allows you to validate the status, so if your request to commit gets 503'ed or you get a timeout you can reliably query to know if it was processed or not 2) Unlimited resubmissions of the final commit. It doesn't matter if I perform the final commit api request 1 time or 100 times, it will never cause a duplicated transaction to occur. So if I get a timeout or a 503 I can resubmit knowing that if my original commit request went in my new submit will be a no-op, and if my last commit request didn't get processed then this time it hopefully will be processed. This pattern isn't just a payments pattern thing either. This is heavily used in distributed systems where failures can occur. UPS' API used to use this as well so you could be sure that you don't pay for duplicate shipping labels or cause duplicate shippments.

doomslice 6y ago

This doesn’t address much of your comment, but I work on systems with millions of req/s, but even with billions you can still sample to do logging and monitoring. But you’re right, we don’t care that every request works, just that 99.9% do.

Matthias247 6y ago

> If you do billions of requests/s I bet you don't care that requests fail? You probably can't even see that requests are failing because you'd have no logging, too expensive at this scale. That's an assumption. Our customers care a lot. And we have sufficient monitoring in place. All recommendations I provided above where about maximizing availability, and provided based on the experiencing of improving the experience for lots of users. > I do financial systems, most load typically doesn't go above 1k/s, but every request matters because a dropped request is a dropped payment, possibly tens of millions of dollars lost! Without knowing much more about your system: If you are losing that amount of money for failed requests (which can e.g. happen due to random network blips) you are doing something wrong. You should invest into different strategies than increasing timeouts.

user5994461 6y ago
It's inherent to payment systems to "lose" money on a failed request. A transaction involves two sides, either both sides agree that the transaction is completed or money is lost/duplicated. If the client decides to drop (timeout) and consider the transaction cancelled, while the server is processing it and will consider it done. That's a catastrophic issue that needs to be addressed. It is one of the most common bugs I've seen in the wild (root cause: too short timeouts). How to make highly critical systems reliable enough in the face of hardware and software issues is a complex topic. At this level this involves a holistic approach to get every component to cooperate together (timeout is a minor example). A HUGE amount of work is to detect errors, and more importantly to propagate errors across diverse stacks (software should be aware of database errors, services should detect other services failing).
- KallDrexx 6y ago
  This doesn't make sense to me. If there is a real risk that a timeout can happen (and there always is) then the payment system should be implementing a two phase commit.
  - earthboundkid 6y ago
    Doesn’t that just create a Byzantine generals problem? My understanding is that real payment systems solve the problem by just taking a day to finalize transactions…
    - KallDrexx 6y ago
      I don't know what the Byzantine generals problem is. Two stage commit is important because it has: 1) Predefined transaction id prior to final submission that allows you to validate the status, so if your request to commit gets 503'ed or you get a timeout you can reliably query to know if it was processed or not 2) Unlimited resubmissions of the final commit. It doesn't matter if I perform the final commit api request 1 time or 100 times, it will never cause a duplicated transaction to occur. So if I get a timeout or a 503 I can resubmit knowing that if my original commit request went in my new submit will be a no-op, and if my last commit request didn't get processed then this time it hopefully will be processed. This pattern isn't just a payments pattern thing either. This is heavily used in distributed systems where failures can occur. UPS' API used to use this as well so you could be sure that you don't pay for duplicate shipping labels or cause duplicate shippments.