4 ms·

Is Kafka expensive because of the hardware requirements, the number of servers needed or just because its relatively niche? Or something else?

WaxProlix 9y ago
For simpler setups, you can just search for "kafka" in the AWS marketplace and snag a bitnami instance. Larger clusters take more administrative overhead; I've had to set a few of these up manually and it's not fun or particularly intuitive.
tomnipotent 9y ago
Common complaint is the reliance on ZooKeeper, which requires three servers to get started.
- WaxProlix 9y ago
  Yeah and it's not obvious that there's any way to get Kafka up and running without Zookeeper, or what their interaction is. And you can get Zookeeper and Kafka running on one machine, but it's such a pain and there's nothing to help really in the docs.
  - poooogles 9y ago
    >And you can get Zookeeper and Kafka running on one machine, but it's such a pain and there's nothing to help really in the docs. What sort of thing are you after? Like a getting started for production without creating footguns style how too?
    - WaxProlix 9y ago
      More like getting started in a dev environment; I want to play with a technology on a laptop or my home compute machine sometimes before venturing into production or recommending it to clients.
      - d_t_w 9y ago
        https://github.com/troy-west/thimble https://github.com/troy-west/thimble
      - miguno 9y ago
        Why don't you run Kafka from a Docker image, like the Docker images provided by Confluent?https://docs.confluent.io/current/installation/docker/docs/ https://docs.confluent.io/current/installation/docker/docs/
      - miguno 9y ago
        Also, since you talked above about the "lack of documentation" to get up and running with a small Kafka deployment of only 1 broker and 1 ZK instance: this is covered in the Kafka quickstart, which is probably what most people will read when starting out with Kafka. See https://kafka.apache.org/documentation/#quickstart https://kafka.apache.org/documentation/#quickstart. Perhaps you had a different issue though?
- miguno 9y ago
  Kafka requires only a single ZooKeeper instance to get started. Kafka's official quickstart actually walks you through running a local deployment of Kafka that has 1x Kafka broker and 1x ZK instance: https://kafka.apache.org/documentation/#quickstart https://kafka.apache.org/documentation/#quickstart

WaxProlix 9y ago

For simpler setups, you can just search for "kafka" in the AWS marketplace and snag a bitnami instance. Larger clusters take more administrative overhead; I've had to set a few of these up manually and it's not fun or particularly intuitive.

tomnipotent 9y ago

Common complaint is the reliance on ZooKeeper, which requires three servers to get started.

WaxProlix 9y ago
Yeah and it's not obvious that there's any way to get Kafka up and running without Zookeeper, or what their interaction is. And you can get Zookeeper and Kafka running on one machine, but it's such a pain and there's nothing to help really in the docs.
- poooogles 9y ago
  >And you can get Zookeeper and Kafka running on one machine, but it's such a pain and there's nothing to help really in the docs. What sort of thing are you after? Like a getting started for production without creating footguns style how too?
  - WaxProlix 9y ago
    More like getting started in a dev environment; I want to play with a technology on a laptop or my home compute machine sometimes before venturing into production or recommending it to clients.
    - d_t_w 9y ago
      https://github.com/troy-west/thimble https://github.com/troy-west/thimble
    - miguno 9y ago
      Why don't you run Kafka from a Docker image, like the Docker images provided by Confluent?https://docs.confluent.io/current/installation/docker/docs/ https://docs.confluent.io/current/installation/docker/docs/
    - miguno 9y ago
      Also, since you talked above about the "lack of documentation" to get up and running with a small Kafka deployment of only 1 broker and 1 ZK instance: this is covered in the Kafka quickstart, which is probably what most people will read when starting out with Kafka. See https://kafka.apache.org/documentation/#quickstart https://kafka.apache.org/documentation/#quickstart. Perhaps you had a different issue though?
miguno 9y ago
Kafka requires only a single ZooKeeper instance to get started. Kafka's official quickstart actually walks you through running a local deployment of Kafka that has 1x Kafka broker and 1x ZK instance: https://kafka.apache.org/documentation/#quickstart https://kafka.apache.org/documentation/#quickstart