5 ms·
I'm not actually a good resource on scaling Kafka. Our distributed systems teams do a great job of providing reliable infrastructure and scaling it up, so on t
by justinc-yelp 10y ago
I'm not actually a good resource on scaling Kafka. Our distributed systems teams do a great job of providing reliable infrastructure and scaling it up, so on the application side we are mostly able to treat it like a black box that just works.
In general, I do think poooogles covered it well. Kafka is designed to scale. The one thing we do that you might not expect is splitting data across clusters, depending on what guarantees we want to provide.
We also tend to make sure all data is replicated using geographic distribution to avoid SPOF issues. We do use the min ISR settings and different required ack levels, depending on we want to trade off durability and availability for an application.