Kafka makes reliable durable messaging not just possible, but solves all the usual attendant problems you try to design around. * Durable messages cause a slow

Kafka makes reliable durable messaging not just possible, but solves all the usual attendant problems you try to design around. * Durable messages cause a slow-down when publishing, but not with Kafka because it uses the Linux kernel's page cache to write. * Durable messages are slow to read if they have to come from disk. Kafka is optimized to load sequential blocks into memory an push them out through a socket with very few copies. This makes for very fast reads. * Slow consumers can bog down the broker. Kafka stores all messages and keeps them on the topic for a time horizon. No back-pressure from slow consumers. * Disconnected but subscribed consumers cause messages to back up on disk and eventually clog the broker. Kafka stores all messages. There's no clogging or backup, that's just how it works. * Brokers must track whether a consumer actually received the message, failures can cause missed messages or clogs. Kafka clients may read from a given point on the topic forward. If they fail during a read, they just back up and read again. The messages will be there for hours/days/weeks as configured. With a rock-steady durable messaging system based on commit logs, all of those problems that arose from attempting to avoid durable messaging go away. Now you build microservices that emit and respond to events. Microservices that can "rehydrate" their state from private checkpoints and topic replays. And all of this with partition tolerance and simple mirroring. Although it isn't usually necessary, if you want, you can make all of that elastic with Mesos, too. Further reading: [1] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... [2] http://mesos.apache.org/ http://mesos.apache.org/