3 ms·
It also assumes the producer and consumer are monolith-bound data streams. Almost all AMQP back-ends include traffic message-pattern smart-message-routing to pr
by Joel_Mckay 2y ago
It also assumes the producer and consumer are monolith-bound data streams. Almost all AMQP back-ends include traffic message-pattern smart-message-routing to pre-sort the data as kind of an implicit sharding operation. This pattern is also particularly performant using static route definitions.
Unless one is using Elixir/Erlang channels, than eventually one must glue on the equivalent queue structure to handle concurrency spikes. Note, this is less to do bandwidth peaks, but rather the OS ISO/OSI limitations will become the bottleneck for concurrency.
If your data traffic is truly separable, than it is trivial load-balancing across multiple pattern-matching routing consumers. Thus, you get pre-sharded localized data, and simply avoid the stores state-syncing cost on back-channel links.
If one has over 40k users, than they should already have re-discovered this phenomena. Simply repeat the mantra "concurrent consumers should never share state". =3
- pyinstallwoes 2y agoHow does Elixir/Erlang mitigate the issue? My mental model is that all queueing systems with pub/sub features are OTP-lite or OTP-wannabes. Are you saying that Erlang just does it more efficiently to the extent that scale isn't an issue for the concern/solution?
- Joel_Mckay 2y agoFor modern AMQP back ends, people usually implement a quorum queue based on this paper: https://raft.github.io/raft.pdf https://raft.github.io/raft.pdf An overview of why this replaced many traditional non-durable queue use-cases in RabbitMQ: https://www.youtube.com/watch?v=wuZC7m6dCDA https://www.youtube.com/watch?v=wuZC7m6dCDA https://www.rabbitmq.com/docs/quorum-queues https://www.rabbitmq.com/docs/quorum-queues What are Elixir/Erlang channels: https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html What types of problems channels are used to solve: https://felt.com/blog/pheonix-channel-routing-patterns https://felt.com/blog/pheonix-channel-routing-patterns Have a great day, =3