7 ms·

If you can only handle 100 events/second and you are getting 101 events/second continuously the queue depth is going to go up by 1 event/second up to infinity,

If you can only handle 100 events/second and you are getting 101 events/second continuously the queue depth is going to go up by 1 event/second up to infinity, causing your latency to go to infinity.

gizmo686 5y ago
The moral of that story is don't let your queue go up to infinity. Drop 1/101 events and serve the other 99% with a typical latency. Instead of latency going to infinity to everyone, your service gracefully-ish degrades by serving as many requests as it is able to and quickly reporting an error to those it is not. Once load drops below capacity, you are immedietly back to a fully functioning state. Granted, depending on how well behaved your clients are, the number of events each second may as the failed requests are retried.
- btilly 5y ago
  So we should solve the problem at the port of Long Beach by randomly dropping cargo into the ocean until we no longer get container ships piling up off the coast?
  - vishnugupta 5y ago
    Or build sufficient back pressure mechanisms so that the cargo doesn’t get loaded in the first place or better still the goods doesn’t get manufactured and so on. Send the feedback as far up the source as possible.
    - kqr 5y ago
      Right, and now you're using the argument of the top level comment. Look at the port. Is ingress consistently larger than egress? Then limit ingress. Now look at the ocean. Is ingress consistently larger than egress? Then limit ingress. Now look at the exporting ports. Is ingress consistently larger than egress? Limit ingress. Now look at trucks going to the port. Is ingress consistently larger than egress? Limit ingress. Now look at the factories that load trucks. Is ingress consistently larger than egress? Limit ingress. ---- Back pressure builds on looking at one queue and asking if arrivals come faster than departures go -- exactly what the top-level comment suggested.
    - simonh 5y ago
      The problem there is signal lag. When the backlog at the port is cleared and you signal the factory to start production again, it might take 3 months for the products to start arriving at the port again. If the port gets congested again and you stop production it might take a month for the goods already made and on their way to stop arriving. But you can also redirect cargo to different ports, or increase the price you charge for servicing cargo. Even if you dump containers you don’t have to do so randomly. Customers could even mark containers as rejectable, or non critical and therefore first to go to a backlog for a discount. I found the article really interesting. For most IT use cases jobs are generic but thanks for raising the port example.
      - btilly 5y ago
        The signal lag is a huge problem. When you go a few stages through a supply chain, a gradual variation in consumer demand turns into huge whiplashes in demand. This is called "the bullwhip effect" and is a huge problem in supply chain management. See https://www.cips.org/knowledge/procurement-topics-and-skills/operations-management/bullwhip-effect-in-supply-chain/ https://www.cips.org/knowledge/procurement-topics-and-skills... for more.
    - gpderetta 5y ago
      AKA "there are no truly unbounded queues, only queues you do not know the bounds yet". Back Pressure is good (but also hard).
  - ivalm 5y ago
    I know you mean it sarcastically, but I am guessing this could solve the problem… the throughput of the ports would increase and the demand shock ameliorated.
tharkun__ 5y ago
I am not talking about infinity. I am not talking about a scenario where you have a constant event stream, which will never vary. You are correct that if you have such a system, then at 101 events/second "you're dead". I am talking about a bazillion other use cases in which there is variance in the number of events produced at any given time and where the immediate processing of those events is not of utmost importance. "Slap a queue in between" and suddenly you are no longer bound by what your systems can handle, you are only bound by what the queuing system can handle and that is usually much easier to scale (or can already handle this kind of load). E.g. think about a system needing to synchronize with another external system. Back in the old days (and well I'm sure those systems are still out there today) synchronizing two systems may well have been an overnight job. A nightly sync job. Nowadays many such systems probably try near realtime synchronization but you don't want to make it part of the regular flow either and block the actual operation. But sometimes there's too many synchronizations that need to happen at once, so a queue builds up over some period and goes down again a bit later. It's still better user experience than a nightly job. E.g. your regular load might be at 10 e/s. During some particularly busy time or specific bulk operations being executed in parallel by multiple people, you might have an influx of 200 e/s for a minute or so.

If you can only handle 100 events/second and you are getting 101 events/second continuously the queue depth is going to go up by 1 event/second up to infinity, causing your latency to go to infinity.

gizmo686 5y ago

The moral of that story is don't let your queue go up to infinity. Drop 1/101 events and serve the other 99% with a typical latency. Instead of latency going to infinity to everyone, your service gracefully-ish degrades by serving as many requests as it is able to and quickly reporting an error to those it is not. Once load drops below capacity, you are immedietly back to a fully functioning state. Granted, depending on how well behaved your clients are, the number of events each second may as the failed requests are retried.

btilly 5y ago
So we should solve the problem at the port of Long Beach by randomly dropping cargo into the ocean until we no longer get container ships piling up off the coast?
- vishnugupta 5y ago
  Or build sufficient back pressure mechanisms so that the cargo doesn’t get loaded in the first place or better still the goods doesn’t get manufactured and so on. Send the feedback as far up the source as possible.
  - kqr 5y ago
    Right, and now you're using the argument of the top level comment. Look at the port. Is ingress consistently larger than egress? Then limit ingress. Now look at the ocean. Is ingress consistently larger than egress? Then limit ingress. Now look at the exporting ports. Is ingress consistently larger than egress? Limit ingress. Now look at trucks going to the port. Is ingress consistently larger than egress? Limit ingress. Now look at the factories that load trucks. Is ingress consistently larger than egress? Limit ingress. ---- Back pressure builds on looking at one queue and asking if arrivals come faster than departures go -- exactly what the top-level comment suggested.
  - simonh 5y ago
    The problem there is signal lag. When the backlog at the port is cleared and you signal the factory to start production again, it might take 3 months for the products to start arriving at the port again. If the port gets congested again and you stop production it might take a month for the goods already made and on their way to stop arriving. But you can also redirect cargo to different ports, or increase the price you charge for servicing cargo. Even if you dump containers you don’t have to do so randomly. Customers could even mark containers as rejectable, or non critical and therefore first to go to a backlog for a discount. I found the article really interesting. For most IT use cases jobs are generic but thanks for raising the port example.
    - btilly 5y ago
      The signal lag is a huge problem. When you go a few stages through a supply chain, a gradual variation in consumer demand turns into huge whiplashes in demand. This is called "the bullwhip effect" and is a huge problem in supply chain management. See https://www.cips.org/knowledge/procurement-topics-and-skills/operations-management/bullwhip-effect-in-supply-chain/ https://www.cips.org/knowledge/procurement-topics-and-skills... for more.
  - gpderetta 5y ago
    AKA "there are no truly unbounded queues, only queues you do not know the bounds yet". Back Pressure is good (but also hard).
- ivalm 5y ago
  I know you mean it sarcastically, but I am guessing this could solve the problem… the throughput of the ports would increase and the demand shock ameliorated.

tharkun__ 5y ago

I am not talking about infinity. I am not talking about a scenario where you have a constant event stream, which will never vary. You are correct that if you have such a system, then at 101 events/second "you're dead". I am talking about a bazillion other use cases in which there is variance in the number of events produced at any given time and where the immediate processing of those events is not of utmost importance. "Slap a queue in between" and suddenly you are no longer bound by what your systems can handle, you are only bound by what the queuing system can handle and that is usually much easier to scale (or can already handle this kind of load). E.g. think about a system needing to synchronize with another external system. Back in the old days (and well I'm sure those systems are still out there today) synchronizing two systems may well have been an overnight job. A nightly sync job. Nowadays many such systems probably try near realtime synchronization but you don't want to make it part of the regular flow either and block the actual operation. But sometimes there's too many synchronizations that need to happen at once, so a queue builds up over some period and goes down again a bit later. It's still better user experience than a nightly job. E.g. your regular load might be at 10 e/s. During some particularly busy time or specific bulk operations being executed in parallel by multiple people, you might have an influx of 200 e/s for a minute or so.