3 ms·
Here's some benchmark: Word count of 1.13B messages - Storm: ~16K QPS/node, 100ms per event (P999) - Spark Streaming: 100K QPS/node, 1s batch window - Concord:
by bimil 10y ago
Here's some benchmark:
Word count of 1.13B messages
- Storm: ~16K QPS/node, 100ms per event (P999)
- Spark Streaming: 100K QPS/node, 1s batch window
- Concord: 500K QPS/node, 10ms per event (P999)
Server log processing (29G server log, ~260M msgs)
7 different computations including deduplication, counting, pattern matching, windowing...
4 nodes, 8 vCPU, 32GB RAM each
Concord: 1M – 1.8M QPS / cluster
Spark Streaming: 72K – 2M QPS / cluster
Concord generally performed in the consistent range of 1-1.8M QPS for whereas Spark's throughput varied differently based on window sliding / amount of internal shared state.