3 ms·

Line is doing 150 billion msgs/day on their Kafka infra, about 3million per second. https://www.slideshare.net/linecorp/building-a-companywide-data-pipeline-on

• tudelo 8y ago
  That seems... like a lot. Thanks for the link.
• buro9 8y ago
  At Cloudflare a year ago we said we were doing 6m per second ingest rate on a cluster of 106 brokers with x3 replication factor, 106 partitions: https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... The more interesting blog post IMHO is this one on Kafka compression https://blog.cloudflare.com/squeezing-the-firehose/ https://blog.cloudflare.com/squeezing-the-firehose/
  • Rebelgecko 8y ago
    The compression article was really useful for me when I was trying to squeeze a bit more performance out of our brokers at work. Any plans to revisit the article now that Kafka has zstd support?
    • nosequel 8y ago
      They talk about zstd in the post and they were the ones who added support to Sarama.
    • halbritt 8y ago
      I'm using zstd with my kafka implementation and it works great. v2.1 is required. We're using librdkafka, which works, but I believe we had to compile it in to get that and idempotent producer.