4 ms·

My lazy self always wonder how nice it would be if some of these infrastructure designs were always accompanied with a docker/fig configuration example to be us

My lazy self always wonder how nice it would be if some of these infrastructure designs were always accompanied with a docker/fig configuration example to be used as a start point/proof of concept for people looking for similar solutions. It obviously happens some times [1] [2], but it should be more common... [1] http://alvinhenrick.com/2014/08/18/apache-storm-and-kafka-cluster-with-docker/ http://alvinhenrick.com/2014/08/18/apache-storm-and-kafka-cl... [2] https://registry.hub.docker.com/u/ches/kafka/ https://registry.hub.docker.com/u/ches/kafka/

ewencp 12y ago
Agreed! It's not Docker, but I'm working on getting a decent Vagrant setup included with Kafka: https://issues.apache.org/jira/browse/KAFKA-1173 https://issues.apache.org/jira/browse/KAFKA-1173 That supports pulling up a full cluster locally in VirtualBox or in EC2. Just a first cut, but it already makes testing a lot easier for me. But in it's current state, that patch is a starting point that is really intended more for Kafka developers than for Kafka users. I really like what the Mesosphere folks have done -- great variety of OSes and cloud platforms, plus they do all the heavy lifting of bringing the cluster up for you.
erichmond 12y ago
As you pointed out, this is already pretty common. As someone who's tried this before, it's hard to find the right balance. To setup a dead simple kafka config is about 10 lines of shell, so simple, it almost doesn't make sense to do. A complex kafka setup is pretty involved (relatively speaking) and becomes domain specific pretty quickly, at which point, it probably becomes less usable / understandable to someone who is trying to learn / get interested. On the whole, I agree with you. Some of the open source software that exists now is truly amazing and I think lots of people are defaulting to less-the-optimal solutions because they just don't have exposure to the latest and greatest.

ewencp 12y ago

Agreed! It's not Docker, but I'm working on getting a decent Vagrant setup included with Kafka: https://issues.apache.org/jira/browse/KAFKA-1173 https://issues.apache.org/jira/browse/KAFKA-1173 That supports pulling up a full cluster locally in VirtualBox or in EC2. Just a first cut, but it already makes testing a lot easier for me. But in it's current state, that patch is a starting point that is really intended more for Kafka developers than for Kafka users. I really like what the Mesosphere folks have done -- great variety of OSes and cloud platforms, plus they do all the heavy lifting of bringing the cluster up for you.

erichmond 12y ago

As you pointed out, this is already pretty common. As someone who's tried this before, it's hard to find the right balance. To setup a dead simple kafka config is about 10 lines of shell, so simple, it almost doesn't make sense to do. A complex kafka setup is pretty involved (relatively speaking) and becomes domain specific pretty quickly, at which point, it probably becomes less usable / understandable to someone who is trying to learn / get interested. On the whole, I agree with you. Some of the open source software that exists now is truly amazing and I think lots of people are defaulting to less-the-optimal solutions because they just don't have exposure to the latest and greatest.