3 ms·
At Cloudflare a year ago we said we were doing 6m per second ingest rate on a cluster of 106 brokers with x3 replication factor, 106 partitions: https://blog.cl
by buro9 8y ago
At Cloudflare a year ago we said we were doing 6m per second ingest rate on a cluster of 106 brokers with x3 replication factor, 106 partitions: https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p...
The more interesting blog post IMHO is this one on Kafka compression https://blog.cloudflare.com/squeezing-the-firehose/ https://blog.cloudflare.com/squeezing-the-firehose/
- Rebelgecko 8y agoThe compression article was really useful for me when I was trying to squeeze a bit more performance out of our brokers at work. Any plans to revisit the article now that Kafka has zstd support?
- nosequel 8y agoThey talk about zstd in the post and they were the ones who added support to Sarama.
- halbritt 8y agoI'm using zstd with my kafka implementation and it works great. v2.1 is required. We're using librdkafka, which works, but I believe we had to compile it in to get that and idempotent producer.