3 ms·
Hi Justin! Thanks for sharing, very interesting stuff. How do you scale Kafka to handle the massive amount of traffic (and storage) that you seem to generate d
by ricardobeat 10y ago
Hi Justin! Thanks for sharing, very interesting stuff.
How do you scale Kafka to handle the massive amount of traffic (and storage) that you seem to generate daily?
With services talking among themselves via HTTP there is a lot of resilience built-in. Do you have anything in place to avoid this becoming a single point of failure? It must have become the most critical piece of your infra.
- poooogles 10y agoScaling Kafka is pretty simple, the operations document contains most things you'll need to get started [1]. We push 500k documents a second through over 10 6 core/24gb ram hosts pretty uneventfully. Only real pointer is to size ZK appropriately and make sure you leave lots of memory for the file system cache. 1 - https://kafka.apache.org/documentation.html#operations https://kafka.apache.org/documentation.html#operations
- justinc-yelp 10y agoI'm not actually a good resource on scaling Kafka. Our distributed systems teams do a great job of providing reliable infrastructure and scaling it up, so on the application side we are mostly able to treat it like a black box that just works. In general, I do think poooogles covered it well. Kafka is designed to scale. The one thing we do that you might not expect is splitting data across clusters, depending on what guarantees we want to provide. We also tend to make sure all data is replicated using geographic distribution to avoid SPOF issues. We do use the min ISR settings and different required ack levels, depending on we want to trade off durability and availability for an application.