3 ms·
My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular clust
by blindriver 3y ago
My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular cluster that didn't have the capacity to handle 1M qps, but Kafka easily had that capacity years ago.
- _a_a_a_ 3y agoI have to ask, what value is this adding business-wise to store so much?
- sharms 3y agoNot OP but I think it isn't always about storing, but having a log of events which get routed, processed, and aggregated in many cases.
- blindriver 3y agoKafka when used correctly is the like the nervous system for your entire company. You use it like a message bus and dump every single thing into it, and you can extract it out at your leisure, but mostly real-time. It completely transforms how you do services in a company, but it also means you have to invest a lot of money and manpower into maintaining it because it is mission critical.
- kuchenbecker 3y agoIt was quota and hardware, not ability. This is a single service onboarding and they need the hardware. And at that scale, we need to grep the logs so the downstream need the ability to process that volume, which it couldn't until about 2 years ago.