7 ms·

I'm currently comparing using Kinesis vs running a small scale Kafka cluster on AWS. The ecosystem around Kafka is great, especially Kafka connect's stuff like

I'm currently comparing using Kinesis vs running a small scale Kafka cluster on AWS. The ecosystem around Kafka is great, especially Kafka connect's stuff like Debezium. But I don't know if it's worth the trouble to deal with the extra operational complexity. Any opinions on administrating Kafka at small scale?

rhodin 9y ago
Go with a hosted solution: CloudKarafka, Aiven, Heroku Kafka, Eventador are all available. First one has a free plan too. Disclaimer: I work with CloudKarafka.
mmrezaie 9y ago
I am not sure if you had seen comparisons like this [1] or not but it has some important points when comparing these two. But I would say it also depends on how you are gonna use the streams. For example, if like us, you are going to digest the streams using Spark I would say Spark's API is more mature with Kafka (unless you have switched to Spark 2.2 and higher). [1] http://go.datapipe.com/whitepaper-kafka-vs-kinesis-download http://go.datapipe.com/whitepaper-kafka-vs-kinesis-download
- whopa 9y ago
  If you're in AWS land you can use Kinesis firehose to S3, which is perfect for Spark. Way more straightforward than any Kafka solution.
jscheel 9y ago
I've enjoyed using Aiven's hosted Kafka. Can't comment on the other services that offer hosted solutions. I was originally going to go with Kinesis, but there were certain latency issues that didn't make sense for our application. I don't remember the specifics, but I seem to recall something about monster latency occasionally cropping up in Kinesis.
- oskari 9y ago
  Thanks for mentioning us! We're currently running the Kafka 1.0.0 final release packages through our QA systems and hope to make it available for all Aiven (https://aiven.io https://aiven.io) users next week.
sidlls 9y ago
Define "small scale". Keep in mind that Kafka was developed by LinkedIn to handle trillions of bytes of message ingest/transfer per day in real time (that's at least tens of MB per second across all boundaries, and probably much more during "bursty" periods). Is that you? If not, unless you just want to use Kafka as a learning experience/resume builder it's (probably) not necessary to even use it. If that isn't you and you are still intent on using it, I'd second the opinion to go with "let someone else (e.g. 'cloud provider') administer it". It's probably not really worth the effort to get configuration settings and installation details right unless you genuinely have a serious interest in it.
- faizshah 9y ago
  I'm looking for a small scale solution (several orders smaller than linkedin) for change data capture into an ordered queue that Flink or Spark can consume continuously. The solution needs to be reliable and highly available, latency needs to be reasonable and relatively consistent to keep aggregates relatively fresh. Do you have a recommendation?
  - Xorlev 9y ago
    We "only" had 20K messages/s at my last job, and Kafka worked incredibly well for us. We ran 3 brokers and always had headroom for jobs which read the last N days of the log. We had a mix of realtime (continuous) consumers and batch consumers that consumed directly from Kafka, but also archived that data to S3 for future batch processing. Operations were quite simple for us, but we were also used to running Zookeeper before, so YMMV. If you can get away with Kinesis, it's simpler operationally, but Kafka isn't difficult compared to most systems.
    - faizshah 9y ago
      Thanks for sharing, what was your experience with node failure at that scale?
      - Xorlev 9y ago
whopa 9y ago
At small scale just go with Kinesis. The base semantics are pretty much the same between the two, and Kafka is terribly complex to run. The hosted Kafka solutions are too expensive for small scale. Kinesis has a real auth story too, plus you can trigger Lambda functions off streams.
- jurre 9y ago
  > Kafka is terribly complex to run I read this quite often, but we run a relatively small kafka cluster on GCP and it's pretty much hassle-free. We also run some ad-hoc clusters in kubernetes from time to time which also works well. What exactly have you found complex about running Kafka?
  - CSDude 9y ago
    Running Kafka is just fine, the issues arise when a node fails, when you need to add data and re-partition a topic. However, it is not that hard once you know what to do, but Kinesis is simpler but it is expensive as shit.
    - knicholes 9y ago
      I get shit for free daily!
    - optimuspaul 9y ago
      At small scale Kinesis is far less expensive. There is definitely a point where Kinesis becomes more expensive, especially if you consider the operational and human costs involved.
  - nemothekid 9y ago
    >What exactly have you found complex about running Kafka? I run small 2-node kafka cluster that processes to 10 million messages/hr - not large at all - it's very stable, for almost a year now. However what was complex was: * Setup. We wanted it managed it by mesos/marathon, and having to figure out BROKER_IDs took a couple hours of trial and error. * Operations. Adding queues and checking on consumers isn't amazing to do from the command line. * Monitoring. It took a while before I settled on a decent monitoring solution that could give insight into kafka's own consumer paradigm. Even still there are more metrics I would like to have about our cluster that I don't care to put the time in to retrieve.
sertorius 9y ago
Redis has just announced a streaming function that will have components inspired by kafka (consumer groups). http://antirez.com/news/114 http://antirez.com/news/114 And there's always RabbitMQ.
opmac 9y ago
I recommend Kinesis. I was in this same position a nearly couple years ago and Kafka is seriously complicated to manage and run. Kinesis is super simple, 100% managed, and has about the same semantics as Kafka. Also the AWS ecosystem around Kinesis (lambda triggers, firehose, etc) is awesome.
- patrickmay 9y ago
  The number of consumers is also important. Kinesis is optimized (and priced) for the many producers / few consumers scenario. If you have many consumers, Kinesis gets expensive quickly.
mpd 9y ago
Kafka needs additional infrastructure (ZK cluster) if you're going to handle it yourselves. Kinesis wins that category, but it has atrocious write performance compared to Kafka, and limited data retention time, if that's important for your project. I'd look at a hosted Kafka solution before going with Kinesis.
deleted 9y ago
[deleted]
nehanarkhede 9y ago
Disclaimer: I'm one of the creators of Kafka and founders of Confluent The best way to get the power, throughput, latency of Kafka without the operations is to use a hosted service. The one created and supported by the Kafka team is Confluent Cloud https://www.confluent.io/confluent-cloud/ https://www.confluent.io/confluent-cloud/
- ralusek 9y ago
  I love companies developing a product also offering a managed implementation as a service, but I cannot find the pricing. Any time it seems like I have to establish some sort of relationship and get an enterprise tailored solution, I no longer feel like I'm part of a public and elastic market.
  - mianos 9y ago
    I beleive the phrase is "If you have to ask you can't afford it”.
    - samsonradu 9y ago
      Hehe, that only applies to luxury goods, which are decooupled from supply/demand rules.
  - vlahmot 9y ago
    Talked with one of the sales reps, starts at $1500/month. From what they said multi-tenant is about a month a way and self service cloud early next year.
- gnud 9y ago
  When you don't show any information about pricing at all on your page, you're obviously not meant for small-scale use. Interesting to know about if I want to introduce Kafka in a bigger project, though.
- faizshah 9y ago
  I'm familiar, I emailed over a week ago with my usecase (change data capture with debezium and postgres at small scale) and asked for some basic pricing information in comparison to Kinesis. I never received a response so I assumed you were looking for bigger enterprise customers right now.
kod 9y ago
It's really not that big a deal to administer Kafka at small scale. If you have to, you can run it on 3 machines, with ZK on the same nodes. I'd still run it myself over Kinesis, even if I was a single-person startup.

rhodin 9y ago

Go with a hosted solution: CloudKarafka, Aiven, Heroku Kafka, Eventador are all available. First one has a free plan too. Disclaimer: I work with CloudKarafka.

mmrezaie 9y ago

I am not sure if you had seen comparisons like this [1] or not but it has some important points when comparing these two. But I would say it also depends on how you are gonna use the streams. For example, if like us, you are going to digest the streams using Spark I would say Spark's API is more mature with Kafka (unless you have switched to Spark 2.2 and higher). [1] http://go.datapipe.com/whitepaper-kafka-vs-kinesis-download http://go.datapipe.com/whitepaper-kafka-vs-kinesis-download

whopa 9y ago
If you're in AWS land you can use Kinesis firehose to S3, which is perfect for Spark. Way more straightforward than any Kafka solution.

jscheel 9y ago

I've enjoyed using Aiven's hosted Kafka. Can't comment on the other services that offer hosted solutions. I was originally going to go with Kinesis, but there were certain latency issues that didn't make sense for our application. I don't remember the specifics, but I seem to recall something about monster latency occasionally cropping up in Kinesis.

oskari 9y ago
Thanks for mentioning us! We're currently running the Kafka 1.0.0 final release packages through our QA systems and hope to make it available for all Aiven (https://aiven.io https://aiven.io) users next week.

sidlls 9y ago

Define "small scale". Keep in mind that Kafka was developed by LinkedIn to handle trillions of bytes of message ingest/transfer per day in real time (that's at least tens of MB per second across all boundaries, and probably much more during "bursty" periods). Is that you? If not, unless you just want to use Kafka as a learning experience/resume builder it's (probably) not necessary to even use it. If that isn't you and you are still intent on using it, I'd second the opinion to go with "let someone else (e.g. 'cloud provider') administer it". It's probably not really worth the effort to get configuration settings and installation details right unless you genuinely have a serious interest in it.

faizshah 9y ago
I'm looking for a small scale solution (several orders smaller than linkedin) for change data capture into an ordered queue that Flink or Spark can consume continuously. The solution needs to be reliable and highly available, latency needs to be reasonable and relatively consistent to keep aggregates relatively fresh. Do you have a recommendation?
- Xorlev 9y ago
  We "only" had 20K messages/s at my last job, and Kafka worked incredibly well for us. We ran 3 brokers and always had headroom for jobs which read the last N days of the log. We had a mix of realtime (continuous) consumers and batch consumers that consumed directly from Kafka, but also archived that data to S3 for future batch processing. Operations were quite simple for us, but we were also used to running Zookeeper before, so YMMV. If you can get away with Kinesis, it's simpler operationally, but Kafka isn't difficult compared to most systems.
  - faizshah 9y ago
    Thanks for sharing, what was your experience with node failure at that scale?
    - Xorlev 9y ago

whopa 9y ago

At small scale just go with Kinesis. The base semantics are pretty much the same between the two, and Kafka is terribly complex to run. The hosted Kafka solutions are too expensive for small scale. Kinesis has a real auth story too, plus you can trigger Lambda functions off streams.

jurre 9y ago
> Kafka is terribly complex to run I read this quite often, but we run a relatively small kafka cluster on GCP and it's pretty much hassle-free. We also run some ad-hoc clusters in kubernetes from time to time which also works well. What exactly have you found complex about running Kafka?
- CSDude 9y ago
  Running Kafka is just fine, the issues arise when a node fails, when you need to add data and re-partition a topic. However, it is not that hard once you know what to do, but Kinesis is simpler but it is expensive as shit.
  - knicholes 9y ago
    I get shit for free daily!
  - optimuspaul 9y ago
    At small scale Kinesis is far less expensive. There is definitely a point where Kinesis becomes more expensive, especially if you consider the operational and human costs involved.
- nemothekid 9y ago
  >What exactly have you found complex about running Kafka? I run small 2-node kafka cluster that processes to 10 million messages/hr - not large at all - it's very stable, for almost a year now. However what was complex was: * Setup. We wanted it managed it by mesos/marathon, and having to figure out BROKER_IDs took a couple hours of trial and error. * Operations. Adding queues and checking on consumers isn't amazing to do from the command line. * Monitoring. It took a while before I settled on a decent monitoring solution that could give insight into kafka's own consumer paradigm. Even still there are more metrics I would like to have about our cluster that I don't care to put the time in to retrieve.

sertorius 9y ago

Redis has just announced a streaming function that will have components inspired by kafka (consumer groups). http://antirez.com/news/114 http://antirez.com/news/114 And there's always RabbitMQ.

opmac 9y ago

I recommend Kinesis. I was in this same position a nearly couple years ago and Kafka is seriously complicated to manage and run. Kinesis is super simple, 100% managed, and has about the same semantics as Kafka. Also the AWS ecosystem around Kinesis (lambda triggers, firehose, etc) is awesome.

patrickmay 9y ago
The number of consumers is also important. Kinesis is optimized (and priced) for the many producers / few consumers scenario. If you have many consumers, Kinesis gets expensive quickly.

mpd 9y ago

Kafka needs additional infrastructure (ZK cluster) if you're going to handle it yourselves. Kinesis wins that category, but it has atrocious write performance compared to Kafka, and limited data retention time, if that's important for your project. I'd look at a hosted Kafka solution before going with Kinesis.

deleted 9y ago

[deleted]

nehanarkhede 9y ago

Disclaimer: I'm one of the creators of Kafka and founders of Confluent The best way to get the power, throughput, latency of Kafka without the operations is to use a hosted service. The one created and supported by the Kafka team is Confluent Cloud https://www.confluent.io/confluent-cloud/ https://www.confluent.io/confluent-cloud/

ralusek 9y ago
I love companies developing a product also offering a managed implementation as a service, but I cannot find the pricing. Any time it seems like I have to establish some sort of relationship and get an enterprise tailored solution, I no longer feel like I'm part of a public and elastic market.
- mianos 9y ago
  I beleive the phrase is "If you have to ask you can't afford it”.
  - samsonradu 9y ago
    Hehe, that only applies to luxury goods, which are decooupled from supply/demand rules.
- vlahmot 9y ago
  Talked with one of the sales reps, starts at $1500/month. From what they said multi-tenant is about a month a way and self service cloud early next year.
gnud 9y ago
When you don't show any information about pricing at all on your page, you're obviously not meant for small-scale use. Interesting to know about if I want to introduce Kafka in a bigger project, though.
faizshah 9y ago
I'm familiar, I emailed over a week ago with my usecase (change data capture with debezium and postgres at small scale) and asked for some basic pricing information in comparison to Kinesis. I never received a response so I assumed you were looking for bigger enterprise customers right now.

kod 9y ago

It's really not that big a deal to administer Kafka at small scale. If you have to, you can run it on 3 machines, with ZK on the same nodes. I'd still run it myself over Kinesis, even if I was a single-person startup.