As it stands though, kakfa is no where near as complicated as setting up and managing a hadoop/yarn/spark(etc etc) cluster. Without using something like Horton

As it stands though, kakfa is no where near as complicated as setting up and managing a hadoop/yarn/spark(etc etc) cluster. Without using something like Hortonwork's Ambari, it's a right pain trying to get everything installed, working together and managing it. Then you have to worry about how you are going to upgrade it. Currently you can use this chef cookbook https://github.com/mthssdrbrg/kafka-cookbook https://github.com/mthssdrbrg/kafka-cookbook , along with the yahoo kafka-manager application and you can get a cluster running without much effort or complication. The cookbook has a coordination feature which allows you to use a locking mechanism ( I use consul as we use it for dns/auto registration), which allows you to roll out changes across the cluster, restarting each broker one at a time, it also allows you to upgrade in a similar fashion.