5 ms·

Queues invert control flow but require flow control

78 points by alexzeitler 2y ago

taeric 2y ago
I always wish more metaphors were built using conveyor belts in these discussions. It helps me mentally underscore that you have to pay attention to what queue/belt you load and why you need to give that a lot of thought. Granted, I'm probably mostly afraid of diving into factorio again. :D
- jimbokun 2y ago
  Like Lucy in the chocolate factory.
  - Cyphase 2y ago
    What an iconic episode.
- westurner 2y ago
  The Spintronics mechanical circuits game is sort of like conveyor belts. Electrons are not individually identified like things on a conveyor belt. Electrons in conductors, semiconductors, and superconductors do behave like fluids. Turing tapes; https://hackaday.com/2016/08/18/the-turing-tapes/ https://hackaday.com/2016/08/18/the-turing-tapes/ Theory of computation > Models of computation: https://en.wikipedia.org/wiki/Theory_of_computation https://en.wikipedia.org/wiki/Theory_of_computation
- golergka 2y ago
  Playing Factorio has indeed given me a much better understanding of how congestion works in complex systems.
andrewstuart 2y ago
Backpressure matters.
stickfigure 2y ago
This article is weird because it conflates queues with pub/sub systems. Even to the point of comparing Amazon SQS with Google Pub/Sub rather than the closer analogue, Cloud Tasks. I've pushed queues from Amazon and Google to thousands of qps and never found their limits. Flow control is unnecessary; the queue either grows or shrinks, and you adjust processing to compensate (ideally by autoscaling). If the queue depth grows continuously and you can't keep up, you work on the architecture (or budget) problems. If you're storing so much data in queues that you're hitting the storage limits of something like SQS, you are using the wrong tool for the job. It's not a database.
- Joel_Mckay 2y ago
  It also assumes the producer and consumer are monolith-bound data streams. Almost all AMQP back-ends include traffic message-pattern smart-message-routing to pre-sort the data as kind of an implicit sharding operation. This pattern is also particularly performant using static route definitions. Unless one is using Elixir/Erlang channels, than eventually one must glue on the equivalent queue structure to handle concurrency spikes. Note, this is less to do bandwidth peaks, but rather the OS ISO/OSI limitations will become the bottleneck for concurrency. If your data traffic is truly separable, than it is trivial load-balancing across multiple pattern-matching routing consumers. Thus, you get pre-sharded localized data, and simply avoid the stores state-syncing cost on back-channel links. If one has over 40k users, than they should already have re-discovered this phenomena. Simply repeat the mantra "concurrent consumers should never share state". =3
  - pyinstallwoes 2y ago
    How does Elixir/Erlang mitigate the issue? My mental model is that all queueing systems with pub/sub features are OTP-lite or OTP-wannabes. Are you saying that Erlang just does it more efficiently to the extent that scale isn't an issue for the concern/solution?
    - Joel_Mckay 2y ago
      For modern AMQP back ends, people usually implement a quorum queue based on this paper: https://raft.github.io/raft.pdf https://raft.github.io/raft.pdf An overview of why this replaced many traditional non-durable queue use-cases in RabbitMQ: https://www.youtube.com/watch?v=wuZC7m6dCDA https://www.youtube.com/watch?v=wuZC7m6dCDA https://www.rabbitmq.com/docs/quorum-queues https://www.rabbitmq.com/docs/quorum-queues What are Elixir/Erlang channels: https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html What types of problems channels are used to solve: https://felt.com/blog/pheonix-channel-routing-patterns https://felt.com/blog/pheonix-channel-routing-patterns Have a great day, =3
aljarry 2y ago
The article states that "Other event sources like SNS use so-called Event Source Mapping", but (at least to my knowledge) SNS directly triggers the Lambda. SQS and few other "stream and queue-based services" use Event Source Mapping. SNS = topics, push. SQS = queues, poll.

78 points by alexzeitler 2y ago

I always wish more metaphors were built using conveyor belts in these discussions. It helps me mentally underscore that you have to pay attention to what queue/belt you load and why you need to give that a lot of thought. Granted, I'm probably mostly afraid of diving into factorio again. :D

jimbokun 2y ago
Like Lucy in the chocolate factory.
- Cyphase 2y ago
  What an iconic episode.
westurner 2y ago
The Spintronics mechanical circuits game is sort of like conveyor belts. Electrons are not individually identified like things on a conveyor belt. Electrons in conductors, semiconductors, and superconductors do behave like fluids. Turing tapes; https://hackaday.com/2016/08/18/the-turing-tapes/ https://hackaday.com/2016/08/18/the-turing-tapes/ Theory of computation > Models of computation: https://en.wikipedia.org/wiki/Theory_of_computation https://en.wikipedia.org/wiki/Theory_of_computation
golergka 2y ago
Playing Factorio has indeed given me a much better understanding of how congestion works in complex systems.

andrewstuart 2y ago

Backpressure matters.

stickfigure 2y ago

This article is weird because it conflates queues with pub/sub systems. Even to the point of comparing Amazon SQS with Google Pub/Sub rather than the closer analogue, Cloud Tasks. I've pushed queues from Amazon and Google to thousands of qps and never found their limits. Flow control is unnecessary; the queue either grows or shrinks, and you adjust processing to compensate (ideally by autoscaling). If the queue depth grows continuously and you can't keep up, you work on the architecture (or budget) problems. If you're storing so much data in queues that you're hitting the storage limits of something like SQS, you are using the wrong tool for the job. It's not a database.

Joel_Mckay 2y ago
It also assumes the producer and consumer are monolith-bound data streams. Almost all AMQP back-ends include traffic message-pattern smart-message-routing to pre-sort the data as kind of an implicit sharding operation. This pattern is also particularly performant using static route definitions. Unless one is using Elixir/Erlang channels, than eventually one must glue on the equivalent queue structure to handle concurrency spikes. Note, this is less to do bandwidth peaks, but rather the OS ISO/OSI limitations will become the bottleneck for concurrency. If your data traffic is truly separable, than it is trivial load-balancing across multiple pattern-matching routing consumers. Thus, you get pre-sharded localized data, and simply avoid the stores state-syncing cost on back-channel links. If one has over 40k users, than they should already have re-discovered this phenomena. Simply repeat the mantra "concurrent consumers should never share state". =3
- pyinstallwoes 2y ago
  How does Elixir/Erlang mitigate the issue? My mental model is that all queueing systems with pub/sub features are OTP-lite or OTP-wannabes. Are you saying that Erlang just does it more efficiently to the extent that scale isn't an issue for the concern/solution?
  - Joel_Mckay 2y ago
    For modern AMQP back ends, people usually implement a quorum queue based on this paper: https://raft.github.io/raft.pdf https://raft.github.io/raft.pdf An overview of why this replaced many traditional non-durable queue use-cases in RabbitMQ: https://www.youtube.com/watch?v=wuZC7m6dCDA https://www.youtube.com/watch?v=wuZC7m6dCDA https://www.rabbitmq.com/docs/quorum-queues https://www.rabbitmq.com/docs/quorum-queues What are Elixir/Erlang channels: https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html What types of problems channels are used to solve: https://felt.com/blog/pheonix-channel-routing-patterns https://felt.com/blog/pheonix-channel-routing-patterns Have a great day, =3

aljarry 2y ago

The article states that "Other event sources like SNS use so-called Event Source Mapping", but (at least to my knowledge) SNS directly triggers the Lambda. SQS and few other "stream and queue-based services" use Event Source Mapping. SNS = topics, push. SQS = queues, poll.