5 ms·

I like the author's solution (always use bounded queues) because it usually forces you to confront back-pressure up front. It doesn't matter how big your queue

• sdevonoes 5y ago
  > It's annoying but also understandable how often people just dump stuff into an unbounded queue and punt on making sure things work until the system is falling down. It's annoying if it is done by the infrastructure team (I mean, they should know the details of the queue they are managing). It's understandable if it is done by product developers (they are more into "inheritance vs composition" kind of things).
  • dcow 5y ago
    I've seen plenty of product engineers not understand this fundamental aspect of queues and just add them because it "felt right" or "for scale" or something silly...
    • sparsely 5y ago
      There's a lot of "Let's add a queue to deal with scaling issues" type thoughts which don't really work in practice. Like 95% of the time the system works better without one.
      • treis 5y ago
        The joke I like is: You have a problem so you implement a queue. Now you have two problems. It succiently illustrates the problem because you should build your application to account for the queue being down. So you still have your original problem of what do I do if I can't process everything I need to.
        • zmmmmm 5y ago
          > you should build your application to account for the queue being down Maybe. but the whole idea is that the queuing infrastructure is intrinsically more reliable than the rest of the system. So while you may design for it, you can do so with different thresholds for severity of what the conseqeunces might be.
      • joseph8th 5y ago
        I think this is insightful thanks. We did this recently, and combined with the OP article, I'm really reminded of how fundamental and ubiquitous queues are. They aren't always obvious, or even intentional. I generally don't set out to design a queue. It just sort of happens while I solve a problem. So yes, adding an MQ specifically just embeds another queue in your original queue. If your scaling problem is an unbounded, maybe unintentional queue, then the MQ can provide just the throttling you need to keep from overloading your consumer end. Yep, the system just got more complicated, but now it's also more manageable, because we hacked a governor into our unregulated queue. As discussed elsewhere, you still have to deal with the back-pressure on the producers' side.
        • hazz99 5y ago
          Aren’t queues simply a requirement to do asynchronous processing? And MQs are a way to do it while keeping your application stateless, and with features to make it easier to recover from failure (e.g. persistence, DLQs). I love discovering simpler solutions to problems! Could you explain this a bit more - how could you design things that seemingly need a queue, without a queue?
          • dcow 5y ago
            Abstractly, everything has a queue size of 1. Synchronous vs asynchronous just refers to what the producer does while its message is being processed. In synchronous programming, the producer blocks and waits until their message is processed to proceed. In async programming, the producer does other things and optionally receives a notification from the consumer once the task is complete.
            • hazz99 5y ago
              How does this apply to not needing queues? I suppose you can rely on your language runtime to juggle the various jobs and concurrent threads (analogous to workers), but then you lose a lot of the benefits of having an explicit MQ system. If your system goes down, for example, you’ll lose all the in-progress work. Actually, is that the point I was missing? That the benefits of an explicit MQ system are not always required, so it can be simpler to just rely on the async primitives of your language?
  • capn_duck 5y ago
    I've never met an application developer who was unaware of what a "queue" was or the problems they purport to solve. Pretty sure stacks/queues are among the first "data structures" that students create, which inevitably leads to the question of, "what to do when full?" i.e. circular queues, double ended queues, priority queues. I know that the enterprise grade queuing systems we're talking about are a lot more involved than that, but to suggest that developers don't grok queues is pretty disingenuous. And the implications of rate in > rate out is pretty obvious for anyone that's ever had a clogged sink.
    • sdevonoes 5y ago
      I didn't mean queues in general, my bad. I meant, as you pointed out, enterprise grade queuing systems: there are lot of stuff going on there that are not exactly the stuff one learns in the Data Structures 101 course. > And the implications of rate in > rate out is pretty obvious for anyone that's ever had a clogged sink. Well, many developers I know are in their early 20s. I'm not sure they ever had to deal with clogged sinks :)
• konha 5y ago
  > and thus your system must handle the possibility that an event fails to be processed and must be retried/dropped Isn’t a retry just extending the queue to the caller?
  • dcow 5y ago
    That's why I said "retry/drop". Handling back-pressure means you must propagate it backwards to a point in your program where there is enough context to enable decision to be made on how to handle the failure. Also, a retry can wait some amount of time until the system is less busy (and this can be enforced by some other part if needed). Further, you may have many callers and one central thing processing requests so propagating back-pressure to callers also allows you to push the problem back to a point where there are potentially more resources. In a silly example, if someone spamming your queue starts to experience a sluggish UI or run out of RAM because back-pressure has forced the queue to grow on their machine an not your infrastructure, perhaps they'll get the message and cool it (ideally the program handles this gracefully too and tells the user a maximum has been hit, but even without that back-pressure is useful). So, yes. But that's the point.
  • ggrrhh_ta 5y ago
    That's the best scenario; regardless of whether the event reaches the destination or not, in most cases (except continuous monitoring of some parameter which admits losses) the source still will keep the information or the way to generate that information, so the amount of state that the queue at the source requires is still being used. But moreover, the source can throttle the generation of events, or it can just decide that that event is stale and can be dropped, or can pack that event with the next events, or can signal other processes about what is happening, or it can choose other path, or maybe it has some out-of-band signaling capability, or....
  • im3w1l 5y ago
    Yes but that queue is bounded too. And if you follow the systems backward eventually you end up with a real person. That is then forced to prioritize away less important tasks.
  • pornel 5y ago
• mwcampbell 5y ago
  I wonder which combination of HTTP reverse proxy (e.g. nginx, cloud load balancer) and application server (e.g. Python WSGI server) is best at using bounded queues and signaling backpressure (e.g. HTTP 503) rather than just letting requests pile up.
  • slrz 5y ago
    I guess that basically everything you'd consider for production use allows you to configure queue sizes and related behaviour however you fancy.
• lanstin 5y ago
  This philosophy, all queues have explicit tuned limits with a drop log and increment a metric on queue full, was used thoroughly at AOL in the 90s. It worked very well, and let us use hardware at much higher loadings than is popular currently. Our goal was ~90% CPU at peak (systems I have worked with more recently start to die around fifty percent). Also of course there was a meeting each Friday to look at all queue depths and queue latencies and to see where we needed more capacity coming up. We did have so many subscribers that traffic was fairly smooth over all.
• _vvhw 5y ago
  > I like the author's solution (always use bounded queues) because it usually forces you to confront back-pressure up front. I work on a system that can process a million financial transactions a second, and this principle of "always use bounded queues" has definitely made an incredible impact on the design. We use bounded queues everywhere — and everything is also static allocation: memory, network resources, disk too, even all the messages that might possibly be needed to execute different variants of the distributed consensus protocol. It's really nice coding on it, because all the limits are clearly defined. It forces a little more upfront thinking. This thinking can be a little difficult for a day or two, but the net result is a better design. I love Little's law.
• marcosdumay 5y ago
  > Queues only serve to mitigate bursts. Well, they also serve to hold your state while you add processing capacity. If you look at a market queues, when they start to grow, management deviates people from non time sensitive tasks into cashiers. The queues make keeps things working during the change. Equivalent situations happen on software too.
  • dcow 5y ago
    It's two sides of the same coin. In your example management only adds more cashiers if the queues have become unacceptably long (they've tripped a soft limit). The queues are serving to accommodate irregular shopper purchasing patterns because shoppers don't arrive consistently throughout the day. So really they are serving to smooth over bursts. Adding more cashiers is simply a response to queues hitting an unacceptably long length. If management had infinite cashiers and you could add them instantly then you wouldn't need queues. But, and this is especially obvious if you consider the scenario where all registers are staffed, then you still need queues to handle bursts. Costco is a good place to experience this.