4 ms·

Why is 101 events per second == system down? Of course there are systems where not being able to process everything right when it comes in means that you're "d

Why is 101 events per second == system down? Of course there are systems where not being able to process everything right when it comes in means that you're "down". It's not a given though and queuing systems are actually perfect for use cases where not being able to process at the speed of incoming requests is completely fine. Eventual consistency is a thing. I have the same experience though that it seems to be hard for folks to accept that yes, 100k messages in _that_ queue is totally fine. Some process dumped 100k messages to then be processed over the next hour and that's totally fine. That way we don't have to actually spend a small fortune to process them at the speed at which they can be generated. Of course it depends on the problem at hand. If that particular flow in the system is one that is highly user interactive, then yes, this means 'system down' for your users. Never mind that the system will eventually process it all. I always smile when I see another system than ours and I can spot distributed processing w/ queues in between in how the system behaves in certain circumstances.

dinosaurdynasty 5y ago
If you can only handle 100 events/second and you are getting 101 events/second continuously the queue depth is going to go up by 1 event/second up to infinity, causing your latency to go to infinity.
- gizmo686 5y ago
  The moral of that story is don't let your queue go up to infinity. Drop 1/101 events and serve the other 99% with a typical latency. Instead of latency going to infinity to everyone, your service gracefully-ish degrades by serving as many requests as it is able to and quickly reporting an error to those it is not. Once load drops below capacity, you are immedietly back to a fully functioning state. Granted, depending on how well behaved your clients are, the number of events each second may as the failed requests are retried.
  - btilly 5y ago
    So we should solve the problem at the port of Long Beach by randomly dropping cargo into the ocean until we no longer get container ships piling up off the coast?
    - vishnugupta 5y ago
      Or build sufficient back pressure mechanisms so that the cargo doesn’t get loaded in the first place or better still the goods doesn’t get manufactured and so on. Send the feedback as far up the source as possible.
      - kqr 5y ago
        Right, and now you're using the argument of the top level comment. Look at the port. Is ingress consistently larger than egress? Then limit ingress. Now look at the ocean. Is ingress consistently larger than egress? Then limit ingress. Now look at the exporting ports. Is ingress consistently larger than egress? Limit ingress. Now look at trucks going to the port. Is ingress consistently larger than egress? Limit ingress. Now look at the factories that load trucks. Is ingress consistently larger than egress? Limit ingress. ---- Back pressure builds on looking at one queue and asking if arrivals come faster than departures go -- exactly what the top-level comment suggested.
      - simonh 5y ago
        The problem there is signal lag. When the backlog at the port is cleared and you signal the factory to start production again, it might take 3 months for the products to start arriving at the port again. If the port gets congested again and you stop production it might take a month for the goods already made and on their way to stop arriving. But you can also redirect cargo to different ports, or increase the price you charge for servicing cargo. Even if you dump containers you don’t have to do so randomly. Customers could even mark containers as rejectable, or non critical and therefore first to go to a backlog for a discount. I found the article really interesting. For most IT use cases jobs are generic but thanks for raising the port example.
        5y ago
- tharkun__ 5y ago
  I am not talking about infinity. I am not talking about a scenario where you have a constant event stream, which will never vary. You are correct that if you have such a system, then at 101 events/second "you're dead". I am talking about a bazillion other use cases in which there is variance in the number of events produced at any given time and where the immediate processing of those events is not of utmost importance. "Slap a queue in between" and suddenly you are no longer bound by what your systems can handle, you are only bound by what the queuing system can handle and that is usually much easier to scale (or can already handle this kind of load). E.g. think about a system needing to synchronize with another external system. Back in the old days (and well I'm sure those systems are still out there today) synchronizing two systems may well have been an overnight job. A nightly sync job. Nowadays many such systems probably try near realtime synchronization but you don't want to make it part of the regular flow either and block the actual operation. But sometimes there's too many synchronizations that need to happen at once, so a queue builds up over some period and goes down again a bit later. It's still better user experience than a nightly job. E.g. your regular load might be at 10 e/s. During some particularly busy time or specific bulk operations being executed in parallel by multiple people, you might have an influx of 200 e/s for a minute or so.
NegativeLatency 5y ago
Assuming you have queues and stuff that can hold the requests in the intermediary time. Some systems will just start dropping the data or crashing and loose data. If you have a queue or something in the middle then the failure is potentially recoverable or never happens in the first place.
- kqr 5y ago
  Queues don't fix overload: https://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html
  - NegativeLatency 5y ago
    I don’t think anyone was suggesting that they did. (Seems like a bit of a strawman to me) However they do allow you to smooth or average spikes that could otherwise be problematic. In your articles analogy I am able to drain a pot of pasta down the sink even though I might not be able to pour the same pot down the drain.
  - marius_k 5y ago
    It lowers the load if the consumer is able to process batches of requests efficiently (ex. for vector operations).
- tharkun__ 5y ago
  The term here is literally queues, so yes, there's a queue in there. I.e. HornetQ, RabbitMQ etc. I'm talking about distributed systems that are built with queuing in mind and that analyze the specific use case to understand when it is OK to have a queue build up and when it isn't.

dinosaurdynasty 5y ago

If you can only handle 100 events/second and you are getting 101 events/second continuously the queue depth is going to go up by 1 event/second up to infinity, causing your latency to go to infinity.

gizmo686 5y ago
The moral of that story is don't let your queue go up to infinity. Drop 1/101 events and serve the other 99% with a typical latency. Instead of latency going to infinity to everyone, your service gracefully-ish degrades by serving as many requests as it is able to and quickly reporting an error to those it is not. Once load drops below capacity, you are immedietly back to a fully functioning state. Granted, depending on how well behaved your clients are, the number of events each second may as the failed requests are retried.
- btilly 5y ago
  So we should solve the problem at the port of Long Beach by randomly dropping cargo into the ocean until we no longer get container ships piling up off the coast?
  - vishnugupta 5y ago
    Or build sufficient back pressure mechanisms so that the cargo doesn’t get loaded in the first place or better still the goods doesn’t get manufactured and so on. Send the feedback as far up the source as possible.
    - kqr 5y ago
      Right, and now you're using the argument of the top level comment. Look at the port. Is ingress consistently larger than egress? Then limit ingress. Now look at the ocean. Is ingress consistently larger than egress? Then limit ingress. Now look at the exporting ports. Is ingress consistently larger than egress? Limit ingress. Now look at trucks going to the port. Is ingress consistently larger than egress? Limit ingress. Now look at the factories that load trucks. Is ingress consistently larger than egress? Limit ingress. ---- Back pressure builds on looking at one queue and asking if arrivals come faster than departures go -- exactly what the top-level comment suggested.
    - simonh 5y ago
      The problem there is signal lag. When the backlog at the port is cleared and you signal the factory to start production again, it might take 3 months for the products to start arriving at the port again. If the port gets congested again and you stop production it might take a month for the goods already made and on their way to stop arriving. But you can also redirect cargo to different ports, or increase the price you charge for servicing cargo. Even if you dump containers you don’t have to do so randomly. Customers could even mark containers as rejectable, or non critical and therefore first to go to a backlog for a discount. I found the article really interesting. For most IT use cases jobs are generic but thanks for raising the port example.
      - 5y ago
tharkun__ 5y ago
I am not talking about infinity. I am not talking about a scenario where you have a constant event stream, which will never vary. You are correct that if you have such a system, then at 101 events/second "you're dead". I am talking about a bazillion other use cases in which there is variance in the number of events produced at any given time and where the immediate processing of those events is not of utmost importance. "Slap a queue in between" and suddenly you are no longer bound by what your systems can handle, you are only bound by what the queuing system can handle and that is usually much easier to scale (or can already handle this kind of load). E.g. think about a system needing to synchronize with another external system. Back in the old days (and well I'm sure those systems are still out there today) synchronizing two systems may well have been an overnight job. A nightly sync job. Nowadays many such systems probably try near realtime synchronization but you don't want to make it part of the regular flow either and block the actual operation. But sometimes there's too many synchronizations that need to happen at once, so a queue builds up over some period and goes down again a bit later. It's still better user experience than a nightly job. E.g. your regular load might be at 10 e/s. During some particularly busy time or specific bulk operations being executed in parallel by multiple people, you might have an influx of 200 e/s for a minute or so.

NegativeLatency 5y ago

Assuming you have queues and stuff that can hold the requests in the intermediary time. Some systems will just start dropping the data or crashing and loose data. If you have a queue or something in the middle then the failure is potentially recoverable or never happens in the first place.

kqr 5y ago
Queues don't fix overload: https://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html
- NegativeLatency 5y ago
  I don’t think anyone was suggesting that they did. (Seems like a bit of a strawman to me) However they do allow you to smooth or average spikes that could otherwise be problematic. In your articles analogy I am able to drain a pot of pasta down the sink even though I might not be able to pour the same pot down the drain.
- marius_k 5y ago
  It lowers the load if the consumer is able to process batches of requests efficiently (ex. for vector operations).
tharkun__ 5y ago
The term here is literally queues, so yes, there's a queue in there. I.e. HornetQ, RabbitMQ etc. I'm talking about distributed systems that are built with queuing in mind and that analyze the specific use case to understand when it is OK to have a queue build up and when it isn't.