Strange, we've had exactly the opposite time with Kafka. It's never gone down for us unless we've done something dumb (e.g. running it out of disk). We punish K

Strange, we've had exactly the opposite time with Kafka. It's never gone down for us unless we've done something dumb (e.g. running it out of disk). We punish Kafka and regularly publish and consume massive amounts of data to/from it. It's the one thing in our infrastructure I haven't been able to kill. Even when we ran Kafka out of disk it still kept running for quite a while. Then again, I've never tried running stateful services on Docker. Seems like a bad time. > Kafka/Zookeeper combo struggled to rejoin the cluster We've had issues with the broker properly re-registering with Zookeeper, but nothing that wasn't solved by stopping the broker and starting it again after the Zookeeper session timeout elapsed. > Large companies like Netflix may be able to spend some serious engineering resources to address the problem or at least get it under control. We're a company of 70, of which maybe 15 qualify as backend engineers/ops. Never had an issue managing Kafka. We've accidentally deployed Kafka with insufficient heap space and it kept on ticking. > As a result of this difference, despite Kafka being known to be super fast compared to other message brokers (e.g. RabbitMQ), it is necessarily slower than ZeroMQ given the need to go to disk and back. Writes do, certainly. But you don't have to "go back" from disk with Kafka during steady-state operation. Kafka writes segments to disk, this ends up in the pagecache. Reads to this segment are served directly from RAM which is excellent for the fanout consumer case. A healthy Kafka cluster usually has little disk read IO (unless a consumer is catching up or batch jobs are reading older data), but lots of network IO out. Not saying that Kafka was the right solution here, but it seems like guaranteed delivery and archival of logs is vastly more important than "real time" logs, but what do I know? Their design of the Kafka-based system seems sketchy: colocating brokers with workers is awfully strange. If logging went down, do babies die? If so, ZMQ seems like the wrong solution. If they just wanted to publish logs in realtime only, cool. I'm sure ZMQ will serve them well as it's a better fit for realtime non-durable publishing of data. That said, I'd be interested in them publishing more details about their Kafka outages. The fact that we've had such contrasting experiences points to an interesting X factor in their setup that should be avoided by others.