7 ms·
Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe
by haolez 6y ago
Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D
- cmason 6y agoWhat do you use instead?
- NicoJuicy 6y agoNats
- haolez 6y agoCheap managed cloud services, like AWS SQS and Azure Storage Queue (I usually want some kind of persistence for my queues).
- gwenshap 6y agoConfluent Cloud Basic/Standard is a cheap managed Kafka. If the objection is to the deployment and not Kafka clients.
- KptMarchewa 6y agoWe might have different definitions of cheap.
- jganetsk 6y agoPub/Sub Lite is a cheap managed cloud service on GCP: https://cloud.google.com/pubsub/lite/docs https://cloud.google.com/pubsub/lite/docs With a Kafka compatibility shim: https://github.com/googleapis/java-pubsublite-kafka https://github.com/googleapis/java-pubsublite-kafka Disclaimer: I work for GCP.
- menzella-g 6y agoGCP offers Pub/Sub Lite, an inexpensive messaging product with Kafka-compatible client libraries. https://cloud.google.com/pubsub/lite/docs https://cloud.google.com/pubsub/lite/docs https://github.com/googleapis/java-pubsublite-kafka https://github.com/googleapis/java-pubsublite-kafka Disclaimer: I work on this product.
- dmlittle 6y agoHaven't used it personally myself but I've heard it enough to remember it. Redpanda[1] aims to be a Kafka replacement without having to worry about Zookeeper or the JVM [1] https://vectorized.io/ https://vectorized.io/
- math 6y agoYou can tune Kafka down fairly well if you know what you're doing, but it's not optimised for that OOTB. Or just use Confluent Cloud, which is fully managed and scales down as low as you want (costs cents per Gb). Disclosure: work for Confluent.
- unixhero 6y agoThank you for disclosing and not disclaiming.
- alex_anglin 6y agoWhy would someone choose Confluent Cloud over the Kafka offerings of Azure/AWS/GCP?
- d_t_w 6y agoConfluent Cloud has some nice point-and-click UI for creating associated Kafka resources like Schema Registries and Connect Clusters. My preference is MSK but I'm very comfortable with vanilla Kafka in AWS at a good price with auto-updates.
- tamale 6y agoOne nice thing about confluent cloud vs MSK is the minimum cost of a confluent cloud cluster is far, far cheaper than the minimal cost of an MSK cluster
- dividedbyzero 6y agoIs there a GCP offering that isn't just Confluent Cloud billed via Google?
- lornajane 6y agoYou can get managed Kafka on Aiven (disclaimer: I work there) on GCP, either through the marketplace or directly through Aiven.
- 6y ago
- toomanybits 6y agoI think that's one of the main points. Now you can run it as a single process more like a traditional broker (although it's obviously still a log).
- keithnz 6y agoyeah, really needs a use case that justifies it, I have a particular IoT backend where I made it pluggable between kafka and rabbitmq, ended up just using rabbitmq as it is simpler to work with / manage, and still not really pushing it in terms of performance with thousands of devices.
- colin_mccabe 6y agoPart of the reason we are removing Kafka's ZooKeeper dependency is to get rid of that "heaviness." Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode). For bigger projects, you may want to use a managed cloud service. Or if you do choose to manage it yourself, it will be easier running one service than two. Disclosure: I work for Confluent.
- pdimitar 6y agoYour clarification made me wonder: Is the single process deployment only doable via a container? Or will we actually have OS native process as well?
- colin_mccabe 6y agoYes, you can run a single OS native process in KRaft mode, without using Docker. Docker just avoids the need to install a JVM, but it is not required.
- jwandborg 6y agoA nit regarding the disclosure: I prefer it at the top of the message, and I think that's "best practice", but I don't know for sure.
- sumtechguy 6y agoOh it most certainly simplifies things. I am looking at half the number of boxes needed to run. Which is not insignificant in my cost structure. What is the migration strategy here? Is it doc'd up yet? I am having flashbacks to migration for follower partitions recently which required a decent amount of pre planning of partition layout. Also as it is pulling in the duties of ZK into kafka what sort of CPU/memory changes are you seeing? Is it 'meh' or all the way to 'you may want to add a couple of CPUs and a few more GB'? Also is it working ok with the stretched cluster? Also if you want to hit an interesting market you may want to look at 'does it run OK on a raspberry PI'.
- mrweasel 6y agoFor people who just need a queue, Kafka is a bit like using Kubernetes to run a single Docker container. We run a number of Kafka clusters, most are relatively low trafic, and the management overhead is pretty. Earlier version did require a bit more attention, but mostly it’s pretty simple to deal with.
- cfontes 6y agoThis is huge news. Kafka is awesome, but using it in local envs is a pain in the ass, if this is never becomes PROD ready it is already an immense achievement to be able to run Kafka locally with less complexity and overhead.