32 ms·

Apache Kafka Goes 1.0

392 points by pradeepchhetri 9y ago

faizshah 9y ago
I'm currently comparing using Kinesis vs running a small scale Kafka cluster on AWS. The ecosystem around Kafka is great, especially Kafka connect's stuff like Debezium. But I don't know if it's worth the trouble to deal with the extra operational complexity. Any opinions on administrating Kafka at small scale?
- rhodin 9y ago
  Go with a hosted solution: CloudKarafka, Aiven, Heroku Kafka, Eventador are all available. First one has a free plan too. Disclaimer: I work with CloudKarafka.
- mmrezaie 9y ago
  I am not sure if you had seen comparisons like this [1] or not but it has some important points when comparing these two. But I would say it also depends on how you are gonna use the streams. For example, if like us, you are going to digest the streams using Spark I would say Spark's API is more mature with Kafka (unless you have switched to Spark 2.2 and higher). [1] http://go.datapipe.com/whitepaper-kafka-vs-kinesis-download http://go.datapipe.com/whitepaper-kafka-vs-kinesis-download
  - whopa 9y ago
    If you're in AWS land you can use Kinesis firehose to S3, which is perfect for Spark. Way more straightforward than any Kafka solution.
- jscheel 9y ago
  I've enjoyed using Aiven's hosted Kafka. Can't comment on the other services that offer hosted solutions. I was originally going to go with Kinesis, but there were certain latency issues that didn't make sense for our application. I don't remember the specifics, but I seem to recall something about monster latency occasionally cropping up in Kinesis.
  - oskari 9y ago
    Thanks for mentioning us! We're currently running the Kafka 1.0.0 final release packages through our QA systems and hope to make it available for all Aiven (https://aiven.io https://aiven.io) users next week.
- sidlls 9y ago
  Define "small scale". Keep in mind that Kafka was developed by LinkedIn to handle trillions of bytes of message ingest/transfer per day in real time (that's at least tens of MB per second across all boundaries, and probably much more during "bursty" periods). Is that you? If not, unless you just want to use Kafka as a learning experience/resume builder it's (probably) not necessary to even use it. If that isn't you and you are still intent on using it, I'd second the opinion to go with "let someone else (e.g. 'cloud provider') administer it". It's probably not really worth the effort to get configuration settings and installation details right unless you genuinely have a serious interest in it.
  - faizshah 9y ago
    I'm looking for a small scale solution (several orders smaller than linkedin) for change data capture into an ordered queue that Flink or Spark can consume continuously. The solution needs to be reliable and highly available, latency needs to be reasonable and relatively consistent to keep aggregates relatively fresh. Do you have a recommendation?
    - Xorlev 9y ago
      We "only" had 20K messages/s at my last job, and Kafka worked incredibly well for us. We ran 3 brokers and always had headroom for jobs which read the last N days of the log. We had a mix of realtime (continuous) consumers and batch consumers that consumed directly from Kafka, but also archived that data to S3 for future batch processing. Operations were quite simple for us, but we were also used to running Zookeeper before, so YMMV. If you can get away with Kinesis, it's simpler operationally, but Kafka isn't difficult compared to most systems.
      - faizshah 9y ago
        Thanks for sharing, what was your experience with node failure at that scale?
        Xorlev 9y ago
- whopa 9y ago
  At small scale just go with Kinesis. The base semantics are pretty much the same between the two, and Kafka is terribly complex to run. The hosted Kafka solutions are too expensive for small scale. Kinesis has a real auth story too, plus you can trigger Lambda functions off streams.
  - jurre 9y ago
    > Kafka is terribly complex to run I read this quite often, but we run a relatively small kafka cluster on GCP and it's pretty much hassle-free. We also run some ad-hoc clusters in kubernetes from time to time which also works well. What exactly have you found complex about running Kafka?
    - CSDude 9y ago
      Running Kafka is just fine, the issues arise when a node fails, when you need to add data and re-partition a topic. However, it is not that hard once you know what to do, but Kinesis is simpler but it is expensive as shit.
      - knicholes 9y ago
        I get shit for free daily!
      - optimuspaul 9y ago
        At small scale Kinesis is far less expensive. There is definitely a point where Kinesis becomes more expensive, especially if you consider the operational and human costs involved.
    - nemothekid 9y ago
      >What exactly have you found complex about running Kafka? I run small 2-node kafka cluster that processes to 10 million messages/hr - not large at all - it's very stable, for almost a year now. However what was complex was: * Setup. We wanted it managed it by mesos/marathon, and having to figure out BROKER_IDs took a couple hours of trial and error. * Operations. Adding queues and checking on consumers isn't amazing to do from the command line. * Monitoring. It took a while before I settled on a decent monitoring solution that could give insight into kafka's own consumer paradigm. Even still there are more metrics I would like to have about our cluster that I don't care to put the time in to retrieve.
- sertorius 9y ago
  Redis has just announced a streaming function that will have components inspired by kafka (consumer groups). http://antirez.com/news/114 http://antirez.com/news/114 And there's always RabbitMQ.
- opmac 9y ago
  I recommend Kinesis. I was in this same position a nearly couple years ago and Kafka is seriously complicated to manage and run. Kinesis is super simple, 100% managed, and has about the same semantics as Kafka. Also the AWS ecosystem around Kinesis (lambda triggers, firehose, etc) is awesome.
  - patrickmay 9y ago
    The number of consumers is also important. Kinesis is optimized (and priced) for the many producers / few consumers scenario. If you have many consumers, Kinesis gets expensive quickly.
- mpd 9y ago
  Kafka needs additional infrastructure (ZK cluster) if you're going to handle it yourselves. Kinesis wins that category, but it has atrocious write performance compared to Kafka, and limited data retention time, if that's important for your project. I'd look at a hosted Kafka solution before going with Kinesis.
- deleted 9y ago
  [deleted]
- nehanarkhede 9y ago
  Disclaimer: I'm one of the creators of Kafka and founders of Confluent The best way to get the power, throughput, latency of Kafka without the operations is to use a hosted service. The one created and supported by the Kafka team is Confluent Cloud https://www.confluent.io/confluent-cloud/ https://www.confluent.io/confluent-cloud/
  - ralusek 9y ago
    I love companies developing a product also offering a managed implementation as a service, but I cannot find the pricing. Any time it seems like I have to establish some sort of relationship and get an enterprise tailored solution, I no longer feel like I'm part of a public and elastic market.
    - mianos 9y ago
      I beleive the phrase is "If you have to ask you can't afford it”.
      - samsonradu 9y ago
        Hehe, that only applies to luxury goods, which are decooupled from supply/demand rules.
    - vlahmot 9y ago
      Talked with one of the sales reps, starts at $1500/month. From what they said multi-tenant is about a month a way and self service cloud early next year.
  - gnud 9y ago
    When you don't show any information about pricing at all on your page, you're obviously not meant for small-scale use. Interesting to know about if I want to introduce Kafka in a bigger project, though.
  - faizshah 9y ago
    I'm familiar, I emailed over a week ago with my usecase (change data capture with debezium and postgres at small scale) and asked for some basic pricing information in comparison to Kinesis. I never received a response so I assumed you were looking for bigger enterprise customers right now.
- kod 9y ago
  It's really not that big a deal to administer Kafka at small scale. If you have to, you can run it on 3 machines, with ZK on the same nodes. I'd still run it myself over Kinesis, even if I was a single-person startup.
kafkaisthatyou 9y ago
Do people use kafka for situations where data loss is not tolerable(eg: accept credit card receipts).
- travisp 9y ago
  Yes, but the default configuration is not suitable to that situation. You will need to make some adjustments if you cannot tolerate data loss.
  - kafkaisthatyou 9y ago
    What are the alternatives people are using instead of kafka in these situations. Low volume but high reliability.
    - iampims 9y ago
      Write to a replicated database?
    - oskari 9y ago
      For low volume my recommendation would be to just use a traditional relational database configured for high availability. If you want to use Kafka and need disaster recovery capabilities we typically recommend using Kafka Connect or other similar tools to replicate the data to another cluster or persistent storage system such as S3.
      - theptip 9y ago
        +1. For use-cases which impose strict data durability requirements (either for business or regulatory reasons), I think it's unwise to use anything fancy like Kafka unless you've maxed out performance of your SQL database. For example, for credit card receipts, simply by the nature of the type of transaction, you're unlikely to be processing enough of these to put pressure on a SQL database. One $1/transaction per second means you're grossing north of $30m, which is easy to handle in even an unoptimized schema. Citus reckon you can get thousands to tens of thousands of writes per second[1] on Postgres, which would be grossing tens or hundreds of billions of dollars; this tech stack is suitable even when "low volume" becomes quite significant. Of course, Kafka is designed for situations where you need to process millions of writes per second, which is into "GDP of the whole world" territory if those writes are credit card receipts, so I'd contend you're unlikely to ever need something like Kafka scale for your credit card payment handling components. [1]: https://www.citusdata.com/blog/2017/09/29/what-performance-can-you-expect-from-postgres/ https://www.citusdata.com/blog/2017/09/29/what-performance-c...
deleted 9y ago
[deleted]
SEJeff 9y ago
For workloads that aren't as important durability wise, has anyone considered nats? https://nats.io/ https://nats.io/
- pritambaral 9y ago
  I use NATS in production. Kafka and NATS are almost complete opposites. One is a durable, partitioned queue; the other is a no-persistence message queue.
  - no1youknowz 9y ago
    Aren't they fixing that with NATS Streaming Server?
    - vpol 9y ago
      They did. Partially. I use nats-streaming in Production. It still does not have real “clustering” solution, but for my small scale I’m able to manage ~7-10k messages per second. Which is fine for many companies.
    - manigandham 9y ago
      There's nothing to "fix" - NATS is a distributed and fast pub/sub system. NATS Streaming is actually a client, it uses NATS to communicate and basically just subscribes to some topics, saves messages it receives, and will then replay them back to you if asked. Imagine how you would build a persistent logging app on top of NATS, and that's what NATS Streaming is.
      - SEJeff 9y ago
        We built a metrics relay using it (nats) and are pushing > 500k metrics per second in bursts just fine. I've been pretty blown away with the perf of it actually. That said, it isn't durable, so normal here be dragons warning apply.
  - manigandham 9y ago
di4na 9y ago
I really hope they would stop with that exactly once thing. It does not mean what they use it for, and it does generate a lot of confusion.
- chairmanwow 9y ago
  How so?
  - danielblazevski 9y ago
    If you have a consumer on some other node that does nothing more that a println statement, you will see that there will be more than once delivery. Exactly once seems to refer to exactly once write to kafka and processing in kafka streams that write back to kafka and don't do stuff like make external http requests, in which case you'll need your logic of at most or at least once to delivery results to wherever you're sending the http requests. Confluence blogs do seem pretty misleading in terms of hiding these issues.
    - thanatos_dem 9y ago
      I think the term they meant to use was “idempotent”, but that doesn’t get attention the same way that claiming to violate the FLP result does.
      - danielblazevski 9y ago
        Not sure that I agree with that. I just think that they are opaque to what they really mean by exactly-once and definitely hide the fact that they do not offer exactly-once delivery.
        deleted 9y ago
        [deleted]
        thanatos_dem 9y ago
        We can look at the ProduceRequest logic in their design doc - (https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#bookmark=kix.x4k78mufyunt https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7C...) Check the sequence number map to determine if the PID is already present. If it is not, check sequence number. If it is 0, insert the PID, epoch, and sequence number into the PID mapping table and proceed to append the entries. If the sequence is not zero, return the InvalidSequenceNumber error code. If the PID is present in the mapping, check the epoch. If the epoch is older than the current one, return InvalidProducerEpoch error code. If the epoch matches the current epoch, check the sequence number: If it matches the current sequence number, allow the append. If the sequence number is older than the current sequence number, return DuplicateSequenceNumber. If it is newer, return InvalidSequenceNumber. If the epoch is newer, check the sequence number. If it is 0, update the mapping and allow the append. If it is not 0, return InvalidSequenceNumber. That's a pretty straightforward approach of enforcing idempotency; checking producer and sequence IDs and throwing them out if they aren't correctly ordered. The other big addition to the code base was cross-partition transactions, which, while pretty cool, is a far cry from "Exactly once delivery".
manigandham 9y ago
Congrats to the kafka/confluent team. Side note: https://pulsar.apache.org https://pulsar.apache.org also seems to be gaining traction and has a much better story around performance, pub/sub, multi-tenancy, and cross-dc replication. Will be interesting to see the evolution of both going forward.
- delaSoul 9y ago
  Has somebody compared pulsar with pravega(http://pravega.io http://pravega.io)?
- slap_shot 9y ago
  Interesting. Can someone give a quick/high level list what makes it better than Kafka and in what use cases?
  - manigandham 9y ago
    I recommend reading the Pulsar architecture page: https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/ https://pulsar.apache.org/docs/latest/getting-started/Concep... Biggest difference is that Pulsar splits compute from storage. Pulsar brokers are stateless services that manage topics and handle clients (publishers and consumers) while reading/writing data from a lower tier of bookies, which is the Apache BookKeeper project for low-latency real-time storage workloads. Better performance and easier scaling compared to Kafka's partition rebalancing. It also natively supports data center replication and multiple clusters working together globally. Pulsar addressing is property/cluster/namespace/topic so multi-tenant isolation is built-in. Anywhere you use Kafka can use Pulsar and also consolidate pub/sub only systems that need lower latency than Kafka.
    - lmsp 9y ago
      I think it is beyond partition rebalancing. There is a fact that people didn't realize of making message broker `stateless`. It is actually much better on reacting to failures or shifting traffic, which is critical when running a messaging bus for online services. because it doesn't have to wait for copying the data of a whole partition when error occurs.
truth_seeker 9y ago
Kafka has come a long way. Kudos to confluent team. I came across some projects which claim to be full or partial alternative to Kafka with better scalability and less operational overhead. Jocko - https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko SMF - https://senior7515.github.io/smf/ https://senior7515.github.io/smf/ MapR-ES - https://mapr.com/blog/kafka-vs-mapr-streams-why-mapr/ https://mapr.com/blog/kafka-vs-mapr-streams-why-mapr/ Has anyone tried it in their project ?
- manigandham 9y ago
  None of those are alternatives. Jocko's creator now works for Confluent and it's not production ready or supported, more of a hobby project to recreate kafka in go without zookeeper. SMF is a prototype RPC system and libraries based on the C++ Seastar framework, not a messaging system in itself. MapR-ES is part of the MapR hadoop/data platform and also not standalone. If you want alternatives then look at NATS + NATS Streaming [1] or Apache Pulsar [2]. 1. http://nats.io http://nats.io 2. https://pulsar.apache.org/ https://pulsar.apache.org/
  - truth_seeker 9y ago
    Thanks for sharing the Jocko's creators story. Yes. MapR-ES comes with its own File system and tool chain but its more closer alternative to Kafka than NATS as it follows Kafka 0.9 API and can act as drop in replacement. Although it does not support exactly once semantics yet.
    - manigandham 9y ago
      Usually alternatives are about providing similar functionality, not the exact interface. If you just want a Kafka API then there are multiple adapters available for most other messaging systems, including pulsar. MapR is an entire data platform and suite of services, one of which is the ES (event streams) offering. It's not drop-in for Kafka unless you're already running the MapR platform which is much more involved than just running Kafka.
    - 9y ago
pwdisswordfish 9y ago
So can it be used comfortably now with anything else than Java?
- frankmcsherry 9y ago
  I've been using it in Rust, but librdkafka had (has?) a bug that's been fixed in tree that cut the read throughput to about 10MB/s using loopback (edit: cuts throughput to 10MB/s in many settings, which hurts especially in loopback). With that fixed, it seemed pretty pleasant for my use cases (framed bytestreams at line rates, nothing fancy or web-scale). Edit: crate I was using, and enjoy (which brings in librdkafka): https://github.com/fede1024/rust-rdkafka https://github.com/fede1024/rust-rdkafka
qntmfred 9y ago
For the folks using kafka or kinesis or other products with streaming event architectures - do you have a replay capability? For example, I am using kinesis and I have a lambda that processes events and writes to a postgres database. so say 3 months from now I want to also create a lambda that firehoses to redshift. How do I get the first 3 months of my data into redshift? Right now I have a lambda that writes all events to S3, so I can replay them for other stream consumers (or even for debugging existing consumers). This seems like a reasonable (if naive) solution, but I don't see a lot of talk out there in the first place about ensuring this capability exists, and definitely haven't seen any more robust or thoughtful options discussed. Are people just not doing this generally?
- breischl 9y ago
  IDK about Kinesis, but for Kafka there are two major cases here: - If your Kafka retention period is long enough, then the data will still be sitting in Kafka. You don't need to replay, just have the new consumer start from the beginning. - If the retention period is not long enough, then you'll have to replay data. There's not any provision for that in Kafka itself - it's really just a somewhat-smart logfile. But whatever you're using to write into Kafka may have some way to do it. This is similar to most message queuing or eventing systems. eg, RabbitMQ.
- manigandham 9y ago
  There's nothing special about replay when it comes to a log. The only issue is retention and many managed services have limited retention based on data size or time frame. S3 archiving is common to keep the operational dataset small so reading it back and refilling Kafka for another consumer is actually pretty normal. If you run Kafka yourself, you can set an infinite retention and keep all the messages, and also use the compaction feature to keep the latest value for every key to reduce some of the bloat if that applies to your data, but if you don't need it instantly accessible then it's probably worth archiving to cheaper storage.
- wenc 9y ago
  This replayability is discussed in Designing Data Intensive Applications (DDIA), a book by Martin Kleppmann. Essentially you can use the Change Data Capture (CDC) information in your primary Postgres database, and pipe it through Kafka and replay it on any other data store.[0] This is also the basis of traditional database replication technology, where the change logs are replayed on other databases. Is this architecture common? Well, I suspect it is overkill for most smaller organizations due to increased infrastructure complexity. I wouldn't do this just for the sake of doing it -- you may find yourself saddled with an increased maintenance workload just keeping the infrastructure running. But if you truly have this use case, this is a well-known method for syncing data across various types of datastores (ie. so-called polyglot persistence). [0] https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://www.confluent.io/blog/bottled-water-real-time-integr...
zekrioca 9y ago
Is Kafka good for processing hundreads of GB size messages per second? Can also a Kafka Stream (output) be input to another Kafka stream dynamically?
- spiffytech 9y ago
  Kafka expects messages to be under 1MiB. If you need larger messages, they recommend you store the payload somewhere else and use Kafka to pass around pointers to the payload. So e.g., pass around an S3 URL.
banq 9y ago
kafka transactional producer not working on windows 7, this bug not yet fixed.https://issues.apache.org/jira/browse/KAFKA-6052 https://issues.apache.org/jira/browse/KAFKA-6052
brootstrap 9y ago
ugh... i dispise this crap.. " the “it just works” experience of using Kafka. " Blog post says how it just works, then shows a crazy diagram stringing together 8 different pieces of technology (which themselves are insanely deep). Is there any easy way to understand WTF this crap does without knowing about RDS, hadoop, monitoring, real time BS, streaming etc? Does it "just work" once you've spent weeks and months understanding wtf it is?
- manigandham 9y ago
  Nothing in that diagram has anything to do with Kafka, it's just showing that you can connect different systems together using a single message bus. If you are new to that concept then I recommend reading the seminal blog post about using logs as the data backbone: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... After that, you can choose whichever implementation you want, of which Kafka is just one option. And yes it does require some work to setup and run, as most software does, but it's no more or less than any other distributed system at this point.

I'm currently comparing using Kinesis vs running a small scale Kafka cluster on AWS. The ecosystem around Kafka is great, especially Kafka connect's stuff like Debezium. But I don't know if it's worth the trouble to deal with the extra operational complexity. Any opinions on administrating Kafka at small scale?

rhodin 9y ago
Go with a hosted solution: CloudKarafka, Aiven, Heroku Kafka, Eventador are all available. First one has a free plan too. Disclaimer: I work with CloudKarafka.
mmrezaie 9y ago
I am not sure if you had seen comparisons like this [1] or not but it has some important points when comparing these two. But I would say it also depends on how you are gonna use the streams. For example, if like us, you are going to digest the streams using Spark I would say Spark's API is more mature with Kafka (unless you have switched to Spark 2.2 and higher). [1] http://go.datapipe.com/whitepaper-kafka-vs-kinesis-download http://go.datapipe.com/whitepaper-kafka-vs-kinesis-download
- whopa 9y ago
  If you're in AWS land you can use Kinesis firehose to S3, which is perfect for Spark. Way more straightforward than any Kafka solution.
jscheel 9y ago
I've enjoyed using Aiven's hosted Kafka. Can't comment on the other services that offer hosted solutions. I was originally going to go with Kinesis, but there were certain latency issues that didn't make sense for our application. I don't remember the specifics, but I seem to recall something about monster latency occasionally cropping up in Kinesis.
- oskari 9y ago
  Thanks for mentioning us! We're currently running the Kafka 1.0.0 final release packages through our QA systems and hope to make it available for all Aiven (https://aiven.io https://aiven.io) users next week.
sidlls 9y ago
Define "small scale". Keep in mind that Kafka was developed by LinkedIn to handle trillions of bytes of message ingest/transfer per day in real time (that's at least tens of MB per second across all boundaries, and probably much more during "bursty" periods). Is that you? If not, unless you just want to use Kafka as a learning experience/resume builder it's (probably) not necessary to even use it. If that isn't you and you are still intent on using it, I'd second the opinion to go with "let someone else (e.g. 'cloud provider') administer it". It's probably not really worth the effort to get configuration settings and installation details right unless you genuinely have a serious interest in it.
- faizshah 9y ago
  I'm looking for a small scale solution (several orders smaller than linkedin) for change data capture into an ordered queue that Flink or Spark can consume continuously. The solution needs to be reliable and highly available, latency needs to be reasonable and relatively consistent to keep aggregates relatively fresh. Do you have a recommendation?
  - Xorlev 9y ago
    We "only" had 20K messages/s at my last job, and Kafka worked incredibly well for us. We ran 3 brokers and always had headroom for jobs which read the last N days of the log. We had a mix of realtime (continuous) consumers and batch consumers that consumed directly from Kafka, but also archived that data to S3 for future batch processing. Operations were quite simple for us, but we were also used to running Zookeeper before, so YMMV. If you can get away with Kinesis, it's simpler operationally, but Kafka isn't difficult compared to most systems.
    - faizshah 9y ago
      Thanks for sharing, what was your experience with node failure at that scale?
      - Xorlev 9y ago
whopa 9y ago
At small scale just go with Kinesis. The base semantics are pretty much the same between the two, and Kafka is terribly complex to run. The hosted Kafka solutions are too expensive for small scale. Kinesis has a real auth story too, plus you can trigger Lambda functions off streams.
- jurre 9y ago
  > Kafka is terribly complex to run I read this quite often, but we run a relatively small kafka cluster on GCP and it's pretty much hassle-free. We also run some ad-hoc clusters in kubernetes from time to time which also works well. What exactly have you found complex about running Kafka?
  - CSDude 9y ago
    Running Kafka is just fine, the issues arise when a node fails, when you need to add data and re-partition a topic. However, it is not that hard once you know what to do, but Kinesis is simpler but it is expensive as shit.
    - knicholes 9y ago
      I get shit for free daily!
    - optimuspaul 9y ago
      At small scale Kinesis is far less expensive. There is definitely a point where Kinesis becomes more expensive, especially if you consider the operational and human costs involved.
  - nemothekid 9y ago
    >What exactly have you found complex about running Kafka? I run small 2-node kafka cluster that processes to 10 million messages/hr - not large at all - it's very stable, for almost a year now. However what was complex was: * Setup. We wanted it managed it by mesos/marathon, and having to figure out BROKER_IDs took a couple hours of trial and error. * Operations. Adding queues and checking on consumers isn't amazing to do from the command line. * Monitoring. It took a while before I settled on a decent monitoring solution that could give insight into kafka's own consumer paradigm. Even still there are more metrics I would like to have about our cluster that I don't care to put the time in to retrieve.
sertorius 9y ago
Redis has just announced a streaming function that will have components inspired by kafka (consumer groups). http://antirez.com/news/114 http://antirez.com/news/114 And there's always RabbitMQ.
opmac 9y ago
I recommend Kinesis. I was in this same position a nearly couple years ago and Kafka is seriously complicated to manage and run. Kinesis is super simple, 100% managed, and has about the same semantics as Kafka. Also the AWS ecosystem around Kinesis (lambda triggers, firehose, etc) is awesome.
- patrickmay 9y ago
  The number of consumers is also important. Kinesis is optimized (and priced) for the many producers / few consumers scenario. If you have many consumers, Kinesis gets expensive quickly.
mpd 9y ago
Kafka needs additional infrastructure (ZK cluster) if you're going to handle it yourselves. Kinesis wins that category, but it has atrocious write performance compared to Kafka, and limited data retention time, if that's important for your project. I'd look at a hosted Kafka solution before going with Kinesis.
deleted 9y ago
[deleted]
nehanarkhede 9y ago
Disclaimer: I'm one of the creators of Kafka and founders of Confluent The best way to get the power, throughput, latency of Kafka without the operations is to use a hosted service. The one created and supported by the Kafka team is Confluent Cloud https://www.confluent.io/confluent-cloud/ https://www.confluent.io/confluent-cloud/
- ralusek 9y ago
  I love companies developing a product also offering a managed implementation as a service, but I cannot find the pricing. Any time it seems like I have to establish some sort of relationship and get an enterprise tailored solution, I no longer feel like I'm part of a public and elastic market.
  - mianos 9y ago
    I beleive the phrase is "If you have to ask you can't afford it”.
    - samsonradu 9y ago
      Hehe, that only applies to luxury goods, which are decooupled from supply/demand rules.
  - vlahmot 9y ago
    Talked with one of the sales reps, starts at $1500/month. From what they said multi-tenant is about a month a way and self service cloud early next year.
- gnud 9y ago
  When you don't show any information about pricing at all on your page, you're obviously not meant for small-scale use. Interesting to know about if I want to introduce Kafka in a bigger project, though.
- faizshah 9y ago
  I'm familiar, I emailed over a week ago with my usecase (change data capture with debezium and postgres at small scale) and asked for some basic pricing information in comparison to Kinesis. I never received a response so I assumed you were looking for bigger enterprise customers right now.
kod 9y ago
It's really not that big a deal to administer Kafka at small scale. If you have to, you can run it on 3 machines, with ZK on the same nodes. I'd still run it myself over Kinesis, even if I was a single-person startup.

kafkaisthatyou 9y ago

Do people use kafka for situations where data loss is not tolerable(eg: accept credit card receipts).

travisp 9y ago
Yes, but the default configuration is not suitable to that situation. You will need to make some adjustments if you cannot tolerate data loss.
- kafkaisthatyou 9y ago
  What are the alternatives people are using instead of kafka in these situations. Low volume but high reliability.
  - iampims 9y ago
    Write to a replicated database?
  - oskari 9y ago
    For low volume my recommendation would be to just use a traditional relational database configured for high availability. If you want to use Kafka and need disaster recovery capabilities we typically recommend using Kafka Connect or other similar tools to replicate the data to another cluster or persistent storage system such as S3.
    - theptip 9y ago
      +1. For use-cases which impose strict data durability requirements (either for business or regulatory reasons), I think it's unwise to use anything fancy like Kafka unless you've maxed out performance of your SQL database. For example, for credit card receipts, simply by the nature of the type of transaction, you're unlikely to be processing enough of these to put pressure on a SQL database. One $1/transaction per second means you're grossing north of $30m, which is easy to handle in even an unoptimized schema. Citus reckon you can get thousands to tens of thousands of writes per second[1] on Postgres, which would be grossing tens or hundreds of billions of dollars; this tech stack is suitable even when "low volume" becomes quite significant. Of course, Kafka is designed for situations where you need to process millions of writes per second, which is into "GDP of the whole world" territory if those writes are credit card receipts, so I'd contend you're unlikely to ever need something like Kafka scale for your credit card payment handling components. [1]: https://www.citusdata.com/blog/2017/09/29/what-performance-can-you-expect-from-postgres/ https://www.citusdata.com/blog/2017/09/29/what-performance-c...

deleted 9y ago

[deleted]

SEJeff 9y ago

For workloads that aren't as important durability wise, has anyone considered nats? https://nats.io/ https://nats.io/

pritambaral 9y ago
I use NATS in production. Kafka and NATS are almost complete opposites. One is a durable, partitioned queue; the other is a no-persistence message queue.
- no1youknowz 9y ago
  Aren't they fixing that with NATS Streaming Server?
  - vpol 9y ago
    They did. Partially. I use nats-streaming in Production. It still does not have real “clustering” solution, but for my small scale I’m able to manage ~7-10k messages per second. Which is fine for many companies.
  - manigandham 9y ago
    There's nothing to "fix" - NATS is a distributed and fast pub/sub system. NATS Streaming is actually a client, it uses NATS to communicate and basically just subscribes to some topics, saves messages it receives, and will then replay them back to you if asked. Imagine how you would build a persistent logging app on top of NATS, and that's what NATS Streaming is.
    - SEJeff 9y ago
      We built a metrics relay using it (nats) and are pushing > 500k metrics per second in bursts just fine. I've been pretty blown away with the perf of it actually. That said, it isn't durable, so normal here be dragons warning apply.
- manigandham 9y ago

di4na 9y ago

I really hope they would stop with that exactly once thing. It does not mean what they use it for, and it does generate a lot of confusion.

chairmanwow 9y ago
How so?
- danielblazevski 9y ago
  If you have a consumer on some other node that does nothing more that a println statement, you will see that there will be more than once delivery. Exactly once seems to refer to exactly once write to kafka and processing in kafka streams that write back to kafka and don't do stuff like make external http requests, in which case you'll need your logic of at most or at least once to delivery results to wherever you're sending the http requests. Confluence blogs do seem pretty misleading in terms of hiding these issues.
  - thanatos_dem 9y ago
    I think the term they meant to use was “idempotent”, but that doesn’t get attention the same way that claiming to violate the FLP result does.
    - danielblazevski 9y ago
      Not sure that I agree with that. I just think that they are opaque to what they really mean by exactly-once and definitely hide the fact that they do not offer exactly-once delivery.
      - deleted 9y ago
        [deleted]
      - thanatos_dem 9y ago
        We can look at the ProduceRequest logic in their design doc - (https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#bookmark=kix.x4k78mufyunt https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7C...) Check the sequence number map to determine if the PID is already present. If it is not, check sequence number. If it is 0, insert the PID, epoch, and sequence number into the PID mapping table and proceed to append the entries. If the sequence is not zero, return the InvalidSequenceNumber error code. If the PID is present in the mapping, check the epoch. If the epoch is older than the current one, return InvalidProducerEpoch error code. If the epoch matches the current epoch, check the sequence number: If it matches the current sequence number, allow the append. If the sequence number is older than the current sequence number, return DuplicateSequenceNumber. If it is newer, return InvalidSequenceNumber. If the epoch is newer, check the sequence number. If it is 0, update the mapping and allow the append. If it is not 0, return InvalidSequenceNumber. That's a pretty straightforward approach of enforcing idempotency; checking producer and sequence IDs and throwing them out if they aren't correctly ordered. The other big addition to the code base was cross-partition transactions, which, while pretty cool, is a far cry from "Exactly once delivery".

manigandham 9y ago

Congrats to the kafka/confluent team. Side note: https://pulsar.apache.org https://pulsar.apache.org also seems to be gaining traction and has a much better story around performance, pub/sub, multi-tenancy, and cross-dc replication. Will be interesting to see the evolution of both going forward.

delaSoul 9y ago
Has somebody compared pulsar with pravega(http://pravega.io http://pravega.io)?
slap_shot 9y ago
Interesting. Can someone give a quick/high level list what makes it better than Kafka and in what use cases?
- manigandham 9y ago
  I recommend reading the Pulsar architecture page: https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/ https://pulsar.apache.org/docs/latest/getting-started/Concep... Biggest difference is that Pulsar splits compute from storage. Pulsar brokers are stateless services that manage topics and handle clients (publishers and consumers) while reading/writing data from a lower tier of bookies, which is the Apache BookKeeper project for low-latency real-time storage workloads. Better performance and easier scaling compared to Kafka's partition rebalancing. It also natively supports data center replication and multiple clusters working together globally. Pulsar addressing is property/cluster/namespace/topic so multi-tenant isolation is built-in. Anywhere you use Kafka can use Pulsar and also consolidate pub/sub only systems that need lower latency than Kafka.
  - lmsp 9y ago
    I think it is beyond partition rebalancing. There is a fact that people didn't realize of making message broker `stateless`. It is actually much better on reacting to failures or shifting traffic, which is critical when running a messaging bus for online services. because it doesn't have to wait for copying the data of a whole partition when error occurs.

truth_seeker 9y ago

Kafka has come a long way. Kudos to confluent team. I came across some projects which claim to be full or partial alternative to Kafka with better scalability and less operational overhead. Jocko - https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko SMF - https://senior7515.github.io/smf/ https://senior7515.github.io/smf/ MapR-ES - https://mapr.com/blog/kafka-vs-mapr-streams-why-mapr/ https://mapr.com/blog/kafka-vs-mapr-streams-why-mapr/ Has anyone tried it in their project ?

manigandham 9y ago
None of those are alternatives. Jocko's creator now works for Confluent and it's not production ready or supported, more of a hobby project to recreate kafka in go without zookeeper. SMF is a prototype RPC system and libraries based on the C++ Seastar framework, not a messaging system in itself. MapR-ES is part of the MapR hadoop/data platform and also not standalone. If you want alternatives then look at NATS + NATS Streaming [1] or Apache Pulsar [2]. 1. http://nats.io http://nats.io 2. https://pulsar.apache.org/ https://pulsar.apache.org/
- truth_seeker 9y ago
  Thanks for sharing the Jocko's creators story. Yes. MapR-ES comes with its own File system and tool chain but its more closer alternative to Kafka than NATS as it follows Kafka 0.9 API and can act as drop in replacement. Although it does not support exactly once semantics yet.
  - manigandham 9y ago
    Usually alternatives are about providing similar functionality, not the exact interface. If you just want a Kafka API then there are multiple adapters available for most other messaging systems, including pulsar. MapR is an entire data platform and suite of services, one of which is the ES (event streams) offering. It's not drop-in for Kafka unless you're already running the MapR platform which is much more involved than just running Kafka.
  - 9y ago

pwdisswordfish 9y ago

So can it be used comfortably now with anything else than Java?

frankmcsherry 9y ago
I've been using it in Rust, but librdkafka had (has?) a bug that's been fixed in tree that cut the read throughput to about 10MB/s using loopback (edit: cuts throughput to 10MB/s in many settings, which hurts especially in loopback). With that fixed, it seemed pretty pleasant for my use cases (framed bytestreams at line rates, nothing fancy or web-scale). Edit: crate I was using, and enjoy (which brings in librdkafka): https://github.com/fede1024/rust-rdkafka https://github.com/fede1024/rust-rdkafka

qntmfred 9y ago

For the folks using kafka or kinesis or other products with streaming event architectures - do you have a replay capability? For example, I am using kinesis and I have a lambda that processes events and writes to a postgres database. so say 3 months from now I want to also create a lambda that firehoses to redshift. How do I get the first 3 months of my data into redshift? Right now I have a lambda that writes all events to S3, so I can replay them for other stream consumers (or even for debugging existing consumers). This seems like a reasonable (if naive) solution, but I don't see a lot of talk out there in the first place about ensuring this capability exists, and definitely haven't seen any more robust or thoughtful options discussed. Are people just not doing this generally?

breischl 9y ago
IDK about Kinesis, but for Kafka there are two major cases here: - If your Kafka retention period is long enough, then the data will still be sitting in Kafka. You don't need to replay, just have the new consumer start from the beginning. - If the retention period is not long enough, then you'll have to replay data. There's not any provision for that in Kafka itself - it's really just a somewhat-smart logfile. But whatever you're using to write into Kafka may have some way to do it. This is similar to most message queuing or eventing systems. eg, RabbitMQ.
manigandham 9y ago
There's nothing special about replay when it comes to a log. The only issue is retention and many managed services have limited retention based on data size or time frame. S3 archiving is common to keep the operational dataset small so reading it back and refilling Kafka for another consumer is actually pretty normal. If you run Kafka yourself, you can set an infinite retention and keep all the messages, and also use the compaction feature to keep the latest value for every key to reduce some of the bloat if that applies to your data, but if you don't need it instantly accessible then it's probably worth archiving to cheaper storage.
wenc 9y ago
This replayability is discussed in Designing Data Intensive Applications (DDIA), a book by Martin Kleppmann. Essentially you can use the Change Data Capture (CDC) information in your primary Postgres database, and pipe it through Kafka and replay it on any other data store.[0] This is also the basis of traditional database replication technology, where the change logs are replayed on other databases. Is this architecture common? Well, I suspect it is overkill for most smaller organizations due to increased infrastructure complexity. I wouldn't do this just for the sake of doing it -- you may find yourself saddled with an increased maintenance workload just keeping the infrastructure running. But if you truly have this use case, this is a well-known method for syncing data across various types of datastores (ie. so-called polyglot persistence). [0] https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://www.confluent.io/blog/bottled-water-real-time-integr...

zekrioca 9y ago

Is Kafka good for processing hundreads of GB size messages per second? Can also a Kafka Stream (output) be input to another Kafka stream dynamically?

spiffytech 9y ago
Kafka expects messages to be under 1MiB. If you need larger messages, they recommend you store the payload somewhere else and use Kafka to pass around pointers to the payload. So e.g., pass around an S3 URL.

banq 9y ago

kafka transactional producer not working on windows 7, this bug not yet fixed.https://issues.apache.org/jira/browse/KAFKA-6052 https://issues.apache.org/jira/browse/KAFKA-6052

brootstrap 9y ago

ugh... i dispise this crap.. " the “it just works” experience of using Kafka. " Blog post says how it just works, then shows a crazy diagram stringing together 8 different pieces of technology (which themselves are insanely deep). Is there any easy way to understand WTF this crap does without knowing about RDS, hadoop, monitoring, real time BS, streaming etc? Does it "just work" once you've spent weeks and months understanding wtf it is?

manigandham 9y ago
Nothing in that diagram has anything to do with Kafka, it's just showing that you can connect different systems together using a single message bus. If you are new to that concept then I recommend reading the seminal blog post about using logs as the data backbone: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... After that, you can choose whichever implementation you want, of which Kafka is just one option. And yes it does require some work to setup and run, as most software does, but it's no more or less than any other distributed system at this point.