12 ms·

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usag

by adev_ 4y ago

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost. Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions). If all you need is a simple stupid publish/subscribe broker with topics/auth management, do a favour to yourself, stay away from it.

NovemberWhiskey 4y ago
I see Kafka deployed for things which have perhaps a few thousand messages per day. It's like "did you accidentally mis-specify by six orders of magnitude here?"
MuffinFlavored 4y ago
> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received. How many of us get paid to work jobs where we're basically told "shut up, this is what we're doing/using, go with it"?
- rad_gruchalski 4y ago
  And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
  - justinclift 4y ago
    Maybe something AMQP related instead?
    - rad_gruchalski 4y ago
      Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium number of large volume topics. Topics can be larger the a machine due to partitioning. Strict ordering per partition based on message arrival time. Queues (RabbitMQ or anything amqp) are relatively large number or small to medium volume topics. Ordering is an option, a topic must fit within a machine. Thos are orthogonal concepts. They can live next to each other. My first choice is always Kafka because: persistence, replication, scalability. Works fine as a single broker, can always scale horizontally, zookeeper is not that scary, especially with a good operator. If you think that Kafka is difficult to run, wait until you need replication in RabbitMQ. Good luck with leader election in your application layer. No fun.
      - MuffinFlavored 4y ago
        > Kafka is a distributed log When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Why do you need a sharded Kafka cluster? Most businesses are going to have Redis, SQL, and probably RabbitMQ. Where/why add Kafka to that stack?
        manv1 4y ago
        Kafka was designed for places with a huge data/log volume (GB/sec or more). From what I understand it's generally The Thing To Use when you have not very many (ie: thousands) high-volume sources that have good links. You push stuff into the logs and you can take your time chewing through the log. MQTT was designed for shitloads (hundreds of thousands+) of small devices on bad links connecting to the mothership. Neither of them really are a message queue. I mean, they queue messages, but they have different goals than, say, ActiveMQ or RabbitMQ (which both can be used as a backend for MQTT). Using Kafka for a message queue is overkill. It's more likely that MQTT fits your bill.
victor106 4y ago
>Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions). 100% this. Even using managed Kafka is a pain for most use cases. We replaced managed Kafka with a simple postgresql db using skip locked as a queue mechanism and the dev teams productivity tripled and our total cost of ownership decreased dramatically. Don’t think twice, think 10 times if you really need Kafka
- EdwardDiego 4y ago
  Yeah, if you're able to replace Kafka with PG then your stack didn't need Kafka, so that's good call.
foolfoolz 4y ago
> implying the operational costs of a server are captured in its per hour sticker price managed kafka has been around a while
septune 4y ago
Forget MQTT, Redis as a PUB/SUB will do 99% of the job most of the time.
- jck 4y ago
  Redis 7 supports sharded pubsub. Also, redis streams are excellent and perform really well for Kafka lite type use cases.
  - Gwypaas 4y ago
    Until you need backpressure, then you are left with awful out-of-band hacks and hope for the best.
lmm 4y ago
Kafka no longer requires zookeeper. If you need true master-master high availability from a datastore - which anyone who bothers with a load balancer for their application should demand, what's the point in running your application in a HA configuration if your datastore is a single point of failure - then to the best of my knowledge Kafka is still the least bad option available. It's not the easiest thing to operate, but I'll take it over Galera or Greenplum any day.
itpragmatik 4y ago
AWS SNS + SQS Fanout works pretty well and not too complicated.
- querulous 4y ago
  orders of magnitude more expensive than kafka tho. not feasible at scale
gvtek0 4y ago
>However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train Don't look now but this is how people end up with k8s as well. "We need Kubernetes because we need containers." Google et al convinced people it's the only way to run containers in prod.
EdwardDiego 4y ago
> notoriously hard to operate (Hello ZooKeeper) Eh? ZooKeeper is rock solid.
cduzz 4y ago
I really have to disagree -- for what I've used it for kafka's basically been reliable and amazingly simple to manage. The use case is, I think in the grand scheme of things pretty simple -- it's an "on-prem" infrastructure with some mixture of old and new servers and mixture of SSD and terrible old 3.5in rotational media. None of it is "cloud" -- just kafka clusters deployed with puppet with fluent and kafka rest proxy feeding the kafka and logstash or vector reading from it, but ... it just works. We've had one incident in the past 4 years, and that was because the network decided to go super asymmetric. Anyhow, I've got lots of problems, but "running kafka" isn't one of them.
rockwotj 4y ago
Check out http://redpanda.com http://redpanda.com for a simpler to manage, and 10x faster Kafka
- Proven 4y ago
  [dead]
- lytedev 4y ago
  Have you any experience with it in production?
  - rockwotj 4y ago
    Not yet! It's on my list. Have heard lots of good things from folks who are tho.
petre 4y ago
> And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost. What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? We use MQTT now, but with EMQX as a broker instead of Mosquitto. It has a HTTP API for managing users and ACLs which was easier to integrate than the equivalent Mosquitto MQTT API.
- adev_ 4y ago
  > What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? ZeroMQ is blazing fast, damn simple and battlefield proven. However it is also pretty low level. Things like ACL, topic filtering or user management often need to be reimplemented on top of it. That is not the case for most MQTT blockers. Also if security is a a big concern for you, I would pick up MQTT+HAProxy+TLS anytime over an ZMQ with CurveMQ and its home-made security.
  - deleted 4y ago
    [deleted]

by adev_ 4y ago

NovemberWhiskey 4y ago

I see Kafka deployed for things which have perhaps a few thousand messages per day. It's like "did you accidentally mis-specify by six orders of magnitude here?"

MuffinFlavored 4y ago

> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received. How many of us get paid to work jobs where we're basically told "shut up, this is what we're doing/using, go with it"?

rad_gruchalski 4y ago
And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
- justinclift 4y ago
  Maybe something AMQP related instead?
  - rad_gruchalski 4y ago
    Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium number of large volume topics. Topics can be larger the a machine due to partitioning. Strict ordering per partition based on message arrival time. Queues (RabbitMQ or anything amqp) are relatively large number or small to medium volume topics. Ordering is an option, a topic must fit within a machine. Thos are orthogonal concepts. They can live next to each other. My first choice is always Kafka because: persistence, replication, scalability. Works fine as a single broker, can always scale horizontally, zookeeper is not that scary, especially with a good operator. If you think that Kafka is difficult to run, wait until you need replication in RabbitMQ. Good luck with leader election in your application layer. No fun.
    - MuffinFlavored 4y ago
      > Kafka is a distributed log When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Why do you need a sharded Kafka cluster? Most businesses are going to have Redis, SQL, and probably RabbitMQ. Where/why add Kafka to that stack?
      - manv1 4y ago
        Kafka was designed for places with a huge data/log volume (GB/sec or more). From what I understand it's generally The Thing To Use when you have not very many (ie: thousands) high-volume sources that have good links. You push stuff into the logs and you can take your time chewing through the log. MQTT was designed for shitloads (hundreds of thousands+) of small devices on bad links connecting to the mothership. Neither of them really are a message queue. I mean, they queue messages, but they have different goals than, say, ActiveMQ or RabbitMQ (which both can be used as a backend for MQTT). Using Kafka for a message queue is overkill. It's more likely that MQTT fits your bill.

victor106 4y ago

>Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions). 100% this. Even using managed Kafka is a pain for most use cases. We replaced managed Kafka with a simple postgresql db using skip locked as a queue mechanism and the dev teams productivity tripled and our total cost of ownership decreased dramatically. Don’t think twice, think 10 times if you really need Kafka

EdwardDiego 4y ago
Yeah, if you're able to replace Kafka with PG then your stack didn't need Kafka, so that's good call.

foolfoolz 4y ago

> implying the operational costs of a server are captured in its per hour sticker price managed kafka has been around a while

septune 4y ago

Forget MQTT, Redis as a PUB/SUB will do 99% of the job most of the time.

jck 4y ago
Redis 7 supports sharded pubsub. Also, redis streams are excellent and perform really well for Kafka lite type use cases.
- Gwypaas 4y ago
  Until you need backpressure, then you are left with awful out-of-band hacks and hope for the best.

lmm 4y ago

Kafka no longer requires zookeeper. If you need true master-master high availability from a datastore - which anyone who bothers with a load balancer for their application should demand, what's the point in running your application in a HA configuration if your datastore is a single point of failure - then to the best of my knowledge Kafka is still the least bad option available. It's not the easiest thing to operate, but I'll take it over Galera or Greenplum any day.

itpragmatik 4y ago

AWS SNS + SQS Fanout works pretty well and not too complicated.

querulous 4y ago
orders of magnitude more expensive than kafka tho. not feasible at scale

gvtek0 4y ago

>However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train Don't look now but this is how people end up with k8s as well. "We need Kubernetes because we need containers." Google et al convinced people it's the only way to run containers in prod.

EdwardDiego 4y ago

> notoriously hard to operate (Hello ZooKeeper) Eh? ZooKeeper is rock solid.

cduzz 4y ago

I really have to disagree -- for what I've used it for kafka's basically been reliable and amazingly simple to manage. The use case is, I think in the grand scheme of things pretty simple -- it's an "on-prem" infrastructure with some mixture of old and new servers and mixture of SSD and terrible old 3.5in rotational media. None of it is "cloud" -- just kafka clusters deployed with puppet with fluent and kafka rest proxy feeding the kafka and logstash or vector reading from it, but ... it just works. We've had one incident in the past 4 years, and that was because the network decided to go super asymmetric. Anyhow, I've got lots of problems, but "running kafka" isn't one of them.

rockwotj 4y ago

Check out http://redpanda.com http://redpanda.com for a simpler to manage, and 10x faster Kafka

Proven 4y ago
[dead]
lytedev 4y ago
Have you any experience with it in production?
- rockwotj 4y ago
  Not yet! It's on my list. Have heard lots of good things from folks who are tho.

petre 4y ago

> And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost. What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? We use MQTT now, but with EMQX as a broker instead of Mosquitto. It has a HTTP API for managing users and ACLs which was easier to integrate than the equivalent Mosquitto MQTT API.

adev_ 4y ago
> What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? ZeroMQ is blazing fast, damn simple and battlefield proven. However it is also pretty low level. Things like ACL, topic filtering or user management often need to be reimplemented on top of it. That is not the case for most MQTT blockers. Also if security is a a big concern for you, I would pick up MQTT+HAProxy+TLS anytime over an ZMQ with CurveMQ and its home-made security.
- deleted 4y ago
  [deleted]