I don't have as much depth with it as the author, but I also felt like using it was kind of a bait and switch, especially coming from having read (and loved) Ja

I don't have as much depth with it as the author, but I also felt like using it was kind of a bait and switch, especially coming from having read (and loved) Jay's book I Heart Logs. We're using it at my work but I'm not really in love with it and will probably be trading it out for AMQP for an event system we're planning. I was working with a junior engineer on this project, and he kept on getting confused about what features were ZooKeeper and which were Kafka. There are two complex technologies here as a first hurdle to using it. This isn't ideal. In the book he describes a scenario where your stream processors record to their own storage where they are in the stream, but Kafka's stock consumers now seem to keep track of that in ZooKeeper instead, which seems like an odd place to make the decision. We initially had a small Node.js server, just to experiment with, but I discovered that it had no error handling at all. I could put fake hostnames in and it would just hang out, as if eventually maybe they would appear and it could connect. This is really the Kafka driver's fault; we switched back to Java and the Java client worked, it's just a little overcomplex. But we also periodically came in to find the server had crashed. I still don't know why. (I'm open to it being our general ignorance and a misconfiguration or something.) In the book, Jay describes this beautiful computing model where you have these log streams and you just process them, and it's high-level and very alluring. The actual APIs that Kafka gives you are not beautiful or intuitive. Rewinding to the beginning is something you can only do after you read, for instance. We were thinking of using it like an external write-ahead-log (as described in the book) but it just doesn't really support that use-case directly through its API. It's kind of a shame, because AMQP doesn't support that use case all that well. I believe you have to decide whether you want your queue to act like a round-robin affair or as a persistent queue. Kafka sort of lets you have both; streams (ostensibly) work like persistent broadcast queues. I don't think I'll be able to use AMQP as a write-ahead-log by itself; probably I'll have to have some kind of mediating service that's just recording events to persistence and have a separate way of getting historical stuff. I spent a year or so unable to work on Kafka but telling everyone to read I Heart Logs, so getting in there a few months ago and seeing how wide the gap is between the beautiful theory and the practice has been disillusioning. Frankly, the actual system and the one in the book are pretty radically divergent. I am still a big fan of the system described in the book. I hope someday I get to use it.