7 ms·

So, roughly 1.1m messages per second of about 23 bytes. We're handling 4m to 6m 1.5k log lines per second using Apache Kafka on a cluster of around 100 nodes.

stuartaxelowen 10y ago
Do you mind if I ask what size of machine in CPU/Mem?
- jgrahamc 10y ago
  Each node is something like this: 2x Intel Broadwell E5-2630 v4 (10 core, 2.40 GHz) 128GB - 4X DDR4 32G ECC Registered 2133MHz DIMM 6 x 2.5" 240GB Intel SSD 540 x1 S2S 1G LAN/B W/OCP BRKT ASSY(1IN1) x1 Solarflare 10G SFP+ SFN7002 PCIe card TPM Module Dual Power Supply 1100W each (x2 / chassis) We have this stuff built by Quanta for us.
  - beachstartup 10y ago
    for the curious, if quanta pricing is anything similar to supermicro this is about $750k in hardware.
yelnatz 10y ago
1.5k log lines is how many bytes? Would've been helpful if you sticked with the same unit.
- jgrahamc 10y ago
  Sorry, that wasn't clear. Each log line (in JSON) is roughly 1.5kB.
- gberger 10y ago
  I think he means 4 to 6 million log lines with 1.5 kB each.
paulasmuth 10y ago
That's almost 70gbit/s (are those cloudflare http logs by any chance?) on 100 nodes vs ~170mbit/s on 6 nodes. Or, in other terms 700mbit/s per host with your kafka setup versus ~30mbit/s per host in the benchmark. Allthough your machines seem to be quite a bit beefier (I wonder if all that RAM is actually used?).
- jgrahamc 10y ago
  A lot of it is log data from requests passing through CloudFlare. We run them through Kafka and consumers do stuff like attack detection and generate statistics for our customers. We have about 4 million sites on CloudFlare and each customer has access to analytics about their site which are stored in a CitusDB database.
  - paulasmuth 10y ago
    Impressive. Any chance you could tell how much data is stored for the analytics service after pre-aggregation? (In terms of TB/day or so - I guess it can't be the full 70gbit/s?).
    - jgrahamc 10y ago
      No, nothing like that much data. It's aggregates like "requests per second" and "attacks per second" etc. The actual log lines aren't stored. More details: https://blog.cloudflare.com/scaling-out-postgresql-for-cloudflare-analytics-using-citusdb/ https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...
      - paulasmuth 10y ago
        Thanks. I had read that article some time back :) Sadly it does not give a figure/order of magnitude of the amount of data that's stored in citus after aggregation, but I guess it's just not public information. [I'm working on a system that is somewhat similar to CitusDB (eventql.io) and am always really interested in these numbers] EDIT: I can't reply to your other comment for some reason but many thanks for digging that up, it's very interesting info to me!
thefastlane 10y ago
would love to know what field / business domain this is in, sounds fun!
- jgrahamc 10y ago
  I work for https://www.cloudflare.com/ https://www.cloudflare.com/
alfalfasprout 10y ago
Here's the thing, this isn't really just a PUB/SUB queue. Their forex data example is disingenuous since it's a bad application of their product. For comparison, our market data stream via ZeroMQ using UDP multicast gets around 20 million msg/s (~15 gb/s) with <10 microseconds of latency. But we also buy very high end switches and NICs (that guarantee near-zero packet loss) with built-in redundancies. We use quorum and hardware fencing for fast failover. Even then our total network hardware cost is < $20k for 8 nodes. Our stream aggregator/producer and clients use fast serialization formats like flatbuffers that minimize dynamic memory allocations. JSON is suuuuper slow in comparison. We also use ring buffers designed to fit on a cache-line and bypass the kernel network stack (using DPDK). Fact is, if you're just aggregating logs there's no point in going through all this trouble. Nor will a cloud provider set up custom networking hardware just for you. These high level messaging frameworks attempt to provide some degree of reliability and easy clustering out of the box while keeping things simple. For specific domains where performance is paramount, you can always find light-weight solutions that outperform them. For instance, with market data it's almost always better to drop a quote if it's out-of-sequence or got dropped somewhere than to add latency waiting for a retransmission. This simplifies things since you no longer need durability.

stuartaxelowen 10y ago

Do you mind if I ask what size of machine in CPU/Mem?

jgrahamc 10y ago
Each node is something like this: 2x Intel Broadwell E5-2630 v4 (10 core, 2.40 GHz) 128GB - 4X DDR4 32G ECC Registered 2133MHz DIMM 6 x 2.5" 240GB Intel SSD 540 x1 S2S 1G LAN/B W/OCP BRKT ASSY(1IN1) x1 Solarflare 10G SFP+ SFN7002 PCIe card TPM Module Dual Power Supply 1100W each (x2 / chassis) We have this stuff built by Quanta for us.
- beachstartup 10y ago
  for the curious, if quanta pricing is anything similar to supermicro this is about $750k in hardware.

yelnatz 10y ago

1.5k log lines is how many bytes? Would've been helpful if you sticked with the same unit.

jgrahamc 10y ago
Sorry, that wasn't clear. Each log line (in JSON) is roughly 1.5kB.
gberger 10y ago
I think he means 4 to 6 million log lines with 1.5 kB each.

paulasmuth 10y ago

That's almost 70gbit/s (are those cloudflare http logs by any chance?) on 100 nodes vs ~170mbit/s on 6 nodes. Or, in other terms 700mbit/s per host with your kafka setup versus ~30mbit/s per host in the benchmark. Allthough your machines seem to be quite a bit beefier (I wonder if all that RAM is actually used?).

jgrahamc 10y ago
A lot of it is log data from requests passing through CloudFlare. We run them through Kafka and consumers do stuff like attack detection and generate statistics for our customers. We have about 4 million sites on CloudFlare and each customer has access to analytics about their site which are stored in a CitusDB database.
- paulasmuth 10y ago
  Impressive. Any chance you could tell how much data is stored for the analytics service after pre-aggregation? (In terms of TB/day or so - I guess it can't be the full 70gbit/s?).
  - jgrahamc 10y ago
    No, nothing like that much data. It's aggregates like "requests per second" and "attacks per second" etc. The actual log lines aren't stored. More details: https://blog.cloudflare.com/scaling-out-postgresql-for-cloudflare-analytics-using-citusdb/ https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...
    - paulasmuth 10y ago
      Thanks. I had read that article some time back :) Sadly it does not give a figure/order of magnitude of the amount of data that's stored in citus after aggregation, but I guess it's just not public information. [I'm working on a system that is somewhat similar to CitusDB (eventql.io) and am always really interested in these numbers] EDIT: I can't reply to your other comment for some reason but many thanks for digging that up, it's very interesting info to me!

thefastlane 10y ago

would love to know what field / business domain this is in, sounds fun!

jgrahamc 10y ago
I work for https://www.cloudflare.com/ https://www.cloudflare.com/

alfalfasprout 10y ago

Here's the thing, this isn't really just a PUB/SUB queue. Their forex data example is disingenuous since it's a bad application of their product. For comparison, our market data stream via ZeroMQ using UDP multicast gets around 20 million msg/s (~15 gb/s) with <10 microseconds of latency. But we also buy very high end switches and NICs (that guarantee near-zero packet loss) with built-in redundancies. We use quorum and hardware fencing for fast failover. Even then our total network hardware cost is < $20k for 8 nodes. Our stream aggregator/producer and clients use fast serialization formats like flatbuffers that minimize dynamic memory allocations. JSON is suuuuper slow in comparison. We also use ring buffers designed to fit on a cache-line and bypass the kernel network stack (using DPDK). Fact is, if you're just aggregating logs there's no point in going through all this trouble. Nor will a cloud provider set up custom networking hardware just for you. These high level messaging frameworks attempt to provide some degree of reliability and easy clustering out of the box while keeping things simple. For specific domains where performance is paramount, you can always find light-weight solutions that outperform them. For instance, with market data it's almost always better to drop a quote if it's out-of-sequence or got dropped somewhere than to add latency waiting for a retransmission. This simplifies things since you no longer need durability.