3 ms·
Here's the thing, this isn't really just a PUB/SUB queue. Their forex data example is disingenuous since it's a bad application of their product. For comparison
by alfalfasprout 10y ago
Here's the thing, this isn't really just a PUB/SUB queue. Their forex data example is disingenuous since it's a bad application of their product. For comparison, our market data stream via ZeroMQ using UDP multicast gets around 20 million msg/s (~15 gb/s) with <10 microseconds of latency. But we also buy very high end switches and NICs (that guarantee near-zero packet loss) with built-in redundancies. We use quorum and hardware fencing for fast failover. Even then our total network hardware cost is < $20k for 8 nodes. Our stream aggregator/producer and clients use fast serialization formats like flatbuffers that minimize dynamic memory allocations. JSON is suuuuper slow in comparison. We also use ring buffers designed to fit on a cache-line and bypass the kernel network stack (using DPDK). Fact is, if you're just aggregating logs there's no point in going through all this trouble. Nor will a cloud provider set up custom networking hardware just for you.
These high level messaging frameworks attempt to provide some degree of reliability and easy clustering out of the box while keeping things simple. For specific domains where performance is paramount, you can always find light-weight solutions that outperform them. For instance, with market data it's almost always better to drop a quote if it's out-of-sequence or got dropped somewhere than to add latency waiting for a retransmission. This simplifies things since you no longer need durability.