9 ms·

MQTT vs. Kafka: An IoT Advocate's Perspective

155 points by teleforce 4y ago

Jemaclus 4y ago
This article appears to be comparing MQTT and Kafka + Schema Registry. Using Schema Registry is not required to use Kafka, so OP overcomplicated their own set up for this comparison. There's no argument that Schema Registry is valuable, but it's not something that MQTT seems to provide out of the box, so the comparison seems flawed. I'd be interested in a comparison that is actually apples-to-apples instead of introducing complexity with Schema Registry.
- rad_gruchalski 4y ago
  Comparing mqtt to kafka is already apples vs oranges. Adding the schema registry to this is like throwing a pitaya into the mix. Edit: after reading the article a couple of times it’s clear this isn’t a comparison. “Vs” in the title is the problem. The first impression would have been better if the title was something like ”Kafka and MQTT“. To be honest, as kafka and mqtt often reside next to each other, they complement each other. A better use case would have been an app combining both: show when should mqtt hand over to kafka.
  - deleted 4y ago
    [deleted]
  - stingraycharles 4y ago
    Can confirm, I work with several workloads where MQTT is used to aggregate IoT data close to the source, which is then pushed into the cloud which uses Kafka.
- ryanjshaw 4y ago
  The author is also incorrect about message keys - these are optional and you would only use them for strict ordering when using > 1 partition, or log compaction where "latest is greatest" is good enough.
justinclift 4y ago
An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.
- sigwinch28 4y ago
  I wonder... how many issues was Kafka "soaking up" by dealing with concerns that applications and services didn't have to even consider? As in, I wonder how much application developer burden would be present if using MQTT instead.
  - justinclift 4y ago
    It's an interesting question. No idea how to go about quantifying it though.
    - sigwinch28 4y ago
      Fair.
    - JUNGLEISMASSIVE 4y ago
      [dead]
  - tsss 4y ago
    The vast majority of "Kafka problems" (or rather "Kafka Streams problems") we have with a managed solution at work are due to not fully understanding how it works and how to set it up. There's so much stuff you can configure and so much potential to misuse it. Typical problems are wrong configuration for acks, not understanding durability guarantees, not understanding exactly-once semantics, not naming hidden topics in Kafka Streams, not using schema registry serializers for hidden topics, choosing the wrong partitioning scheme (and wanting to change it later), using Kafka clients with different hash functions, using wrong schema compatibility mode, etc. etc.
mrkeen 4y ago
I jumped onto https://mqtt.org/ https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.
- deleted 4y ago
  [deleted]
- avereveard 4y ago
  > advertising the impossible eh if you read the finer print it's just a deduplication id appended to every message. blog doesn't go into detail on what happen when two client pust a message with the same it, or what happens if there is more than one failure (i.e. client fails to detect a service outage and during the service outage the message is consumed by the broker but persisting fails) but in general the usage of a at least once + a deduplication id is not something revolutionary.
- alecthomas 4y ago
  > I'm a big fan of advertising the impossible on the front page. Do you mean like Confluent do? https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ https://www.confluent.io/blog/exactly-once-semantics-are-pos...
  - codeflo 4y ago
    The impossibility of "exactly once" is a theorem, not an opinion. Knowing that, the article you linked is funny. It begins with maybe a thousand filler words complaining that all of this is "poorly understood", and it's not "impossible", just "very hard". Then it gets to the meat: Yeah well, you know, it's not quite "exactly once delivery", just "exactly once semantics", and to achieve that, messages need to be idempotent so that duplicates don't matter. We all know that. It's called "at least once".
hkt 4y ago
A better comparison with Kafka is redis streams. Similar semantics, a fraction of the operational overhead.
- FridgeSeal 4y ago
  Provided your surrounding tools plug into Redis streams. Oh and provided you don’t need the ordering and parallelism guarantees of Kafka’s partitions. Oh and provided you don’t need the same level of durability and fault tolerance, so yeah, exactly the same.
  - outworlder 4y ago
    Exactly. Tradeoffs exist everywhere, in both directions.
deleted 4y ago
[deleted]
gz5 4y ago
Good article (along with parts 2 and 3). Are there key differences in secure networking constructs (TLS, mTLS, VPN, whitelisted IPs, open ports, etc.) in the options described: + inbound to Kafka clusters and Kafka Connect? + inbound to Mosquitto MQTT broker? + inbound to Telegraf? + inbound to influxDB?
- physicles 4y ago
  Of those, MQTT should be the only thing you expose to the outside world. Secure it with regular TLS, possibly with mutual auth for your clients if you care about that. For the others, secure them using the same tech you use to secure your database. Ideally they wouldn't be accessible from the internet at all.
adev_ 4y ago
The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost. Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions). If all you need is a simple stupid publish/subscribe broker with topics/auth management, do a favour to yourself, stay away from it.
- NovemberWhiskey 4y ago
  I see Kafka deployed for things which have perhaps a few thousand messages per day. It's like "did you accidentally mis-specify by six orders of magnitude here?"
- MuffinFlavored 4y ago
  > However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received. How many of us get paid to work jobs where we're basically told "shut up, this is what we're doing/using, go with it"?
  - rad_gruchalski 4y ago
    And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
    - justinclift 4y ago
      Maybe something AMQP related instead?
yawniek 4y ago
many people seem to not have clarity on what a distributed log is and in which architecture its useful and in which not. if you are abusing a distributed log as a message queue, you are most of the time creating a mess.
- ed25519FUUU 4y ago
  The "abuse" you describe is pretty much how Kafka is used everywhere I've worked.
twawaaay 4y ago
One missing criterion is client complexity. MQTT is built to work well with very little resources on the client. Kafka, on the other hand, requires you to do things you just don't want on a small embedded device -- like opening multiple connections to multiple hosts. Kafka is also just a transport for messages while MQTT is much larger part of the stack and takes care of transporting individual values. Which means you need less other code on your super restricted device. That said, I don't understand all the complaining directed at Kafka in this thread. Kafka is a fantastic tool that provides unique properties and guarantees. As a tech lead/architect I love to have a good selection of tools for different situations. Kafka is very reliable tool that fils an important role of when creating distributed systems and is particularly nice because it is easy to reason about. The negative opinions I heard in the past are typically from people who try to use it for something that it is not well suited for (like efficient transfer of large volumes of data) or because they misunderstood how to use its guarantees to construct larger systems. At one place I met a team who was completely lost with their overloaded Kafka instance and requested to get external help to "further scale and tune" it. I just touched the piece of code on producer and on consumer to publish data in large files to S3 rather than push it all through Kafka. Instead, send a simple message to Kafka with the metadata and location of the payload in S3. And then the client to download it from the bucket. They were happy puppies in no time.
- anonymousDan 4y ago
  Regarding your last point, how did you handle deletion from S3? Did you not need to worry about atomic consumption of the metadata and data? I suppose you could have some kind of background gc task..
  - twawaaay 4y ago
    I think you are overcomplicating the problem for no reason. You build your system from simple guarantees: * message to Kafka is sent after the payload has been published to S3. This means if you have received message on Kafka, the payload is there, no need to worry about it -- you guarantee it because of order of publishing operations. * the object on S3 is immutable. This means it does not matter when you consume it, it stays the same. * the message on Kafka is immutable. This means it does not matter when you consume it, it stays the same. When the client reads the message off of Kafka topic, it just downloads the additional payload from S3. The payload is guaranteed to be there and exactly the same content as published. Once the message is fully processed, it commits this to Kafka topic and that's done. If the processing fails, the processing will be retried later by this or another node. The payload is still there until somebody decides to delete it. Deletion can be done in many different ways. You could have metadata for all those objects (Kafka topic is your metadata database!) and see what is the oldest timestamp on the offsets on all partitions still not committed. Then you delete from S3 all objects that are older that that. This requires that you publish to Kafka in the same order as you publish to S3 (within each partition).
skrtskrt 4y ago
there’s a lot of “Kafka causes so many issues!” comments here. I think it gets a bad rap because it gets introduced to orgs without the org having the requisite level of understanding. If your whole org is just on like a standard OLTP/OLAP setup, then suddenly there’s a Kafka queue, there’s going to he a serious learning curve and bumps along the way. If you’re incorrectly putting async event brokers as the datastore where you should be putting a synchronous DB and then streaming from the DB to kafka with an outbox pattern, you’re going to have a bad time. If you’re not modeling your queue depth and throughput you’re going to have a bad time. If you’re not modeling your concurrency scenarios and synchronization, you’re going to have a bad time.
speedgoose 4y ago
I hear from everyone using Kafka in production that it is hell unless you use Confluent. I gave a try to NATS JetStreams but I havn't been convinced by the performances of the Python client, nor the JavaScript one. I don't have extreme data, I just need descent performances. I'm thinking about giving a try to RabbitMQ streams. I have been very happy with RabbitMQ, the MQTT plugin isn't fully working (the big one is that retained messages are not sent to wildcard subscribers), but it should work with AMQP.
- EdwardDiego 4y ago
  Welp, here's a dissenting opinion - it's not. I've run self-managed, sorta managed (MSK), fully managed (Confluent Cloud), and somewhat managed (Strimzi). It is complex, yes, but it solves a very complicated problem. The issue tends to arise when people use it when simpler alternatives exist for their problem.
- amath 4y ago
  Have you tried Pulsar or Redpanda? Both seem mature enough and provide decent performance to probably meet your needs. What I hear is that Redpanda is a lot easier to manage than Kafka.
  - speedgoose 4y ago
    I haven’t took the time but I should. Redpanda is written in C++ and I tend to prefer safer programming languages, redis being the exception.
  - serial_dev 4y ago
    What's the license for Redpanda? I couldn't find anything but maybe it's because I'm on mobile.
    - deniscoady 4y ago
      Disclaimer, I work at Redpanda. Redpanda Community Edition is licensed with the Redpanda Business Source License (BSL). The core features are free and source-available. This license was inspired by MariaDB and CockroachDB and "for 99.999% of users, restrictions will not apply". The big restriction is that users "cannot provide Redpanda as a commercial streaming or queuing service to others" which is primarily to deter large cloud vendors from taking our work and impacting our ability to operate as a business. Here are some links for more details on how we came to this decision and how we license all our products and features. https://redpanda.com/blog/bsl-source-available-license https://redpanda.com/blog/bsl-source-available-license https://docs.redpanda.com/docs/get-started/licenses/ https://docs.redpanda.com/docs/get-started/licenses/
Kinrany 4y ago
Is there a ZeroMQ-style protocol for a Kafka-like event log? Messaging protocols are many, but can't provide the semantics necessary for async communication.
JaggerFoo 4y ago
I designed a facility IoT system with AWS products: IoT Core (MQTT broker), SiteWise (analytic dashboard), S3 and Lambda. I found the AWS offerings to have everything I needed in one place with a low cost. Added benefits were being able support NIST cybersecurity requirements (CMMC v2, Level 2) in addition to the IoT system. I was a fan of NATS and Kafka in the past, but AWS tooling makes IoT relatively easy. Cheers
physicles 4y ago
The blog post series seems to bury the lede -- it isn't until part 3 [1] that we get to the insight that MQTT and Kafka solve different problems and therefore can have complementary roles in the same system. We use this architecture for IoT: MQTT for edge because it's standard and super good enough, and Kafka because it turns out we want to do more than one thing with the data as it streams in so not using Kafka would end up being more complicated. Here's a key insight though: for IoT you don't want to use an actual MQTT broker, like Mosquitto or HiveMQ. If you do, it's hard to avoid data loss. Server side you have something that subscribes to those MQTT topics and pushes the data into Kafka. What do you do when that thing needs to be restarted? Ok, you can use persistent sessions in your MQTT broker. But how much memory does your MQTT broker need? What if your MQTT broker crashes? Oops, now your MQTT broker needs its own persistent database to keep track of all those messages in limbo. What you want is an MQTT gateway -- something that looks like an MQTT broker to the devices, but the server side does something different with received messages. When it gets the MQTT PUBLISH command, it sends the message to Kafka, waits for the ack from Kafka, and only then sends PUBACK back. Presto, the MQTT thing is now stateless and horizontally scalable. Your clients just need some retry logic. Maybe Kafka's mqtt-proxy does this, I don't know. I don't think it's mentioned in part 3. But it's a key property of such a system. I'm guessing Amazon's IoT gateway does this, because once you've thought about it hard enough it becomes obvious this is how it needs to work. 1: https://www.influxdata.com/blog/mqtt-vs-kafka-iot-advocates-perspective-part-3/ https://www.influxdata.com/blog/mqtt-vs-kafka-iot-advocates-...
- sally_glance 4y ago
  Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?
  - FridgeSeal 4y ago
    If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, they’re just buffered (with all the extra overhead that entails). The stateful approach also hides system back pressure from clients, so in the event of network/storage/service degradation/failure, rather than backing off producing messages, or applying some other kind of application specific decision; clients keep producing messages as if nothing is wrong, because the stateful broker is buffering them locally. Storage and resource reqs for the broker continue to grow, until they either suddenly die, or stop accepting messages. Even worse, when downstream comes back up, the brokers are they going to dump all their messages and possibly re-overload recovering systems?
mstaoru 4y ago
Most MQTT brokers are not great at storing data. The pattern that works the best for me in solving the "edge delivery" use case is a lightweight clustered MQTT broker (e.g. VerneMQ) with a little Lua script inside to push everything into Redis Streams (of course, clustered as well) immediately. Kafka is another alternative, but it's not that lightweight, operationally, and often Redis Streams are "good enough". With an LB in front of both Verne and Redis, this setup is pretty decent for IoT data ingress.
jake_morrison 4y ago
I have done a lot of work with IoT-ish data, e.g. sending location and telemetry data from vehicles and remote sensors over unreliable cellular networks. I need the ability to queue data on the device to deal with patchy connectivity and some policy for deleting messages in the queue. For example, I might throw away unsent location updates that are "stale" to save space. I might need to prioritize some messages, e.g. "lithium-ion battery pack overheating". I may be running on embedded hardware that is too small to run Linux. The connection to the modem might be serial. Data size and bandwidth usage can make a difference. I might get 2MB/month for $3. Bytes count if I want to send frequent updates to get more precision on the location. I generally know exactly what messages I am sending, so using a compiled format like gRPC or COAP (https://en.wikipedia.org/wiki/Constrained_Application_Protocol https://en.wikipedia.org/wiki/Constrained_Application_Protoc...) can be better than JSON. I may need to get through multiple layers of network address translation, making it different to send messages back. So keeping a persistent TCP connection can help. But using TCP for one-off connections wastes bytes, so UDP can be useful if I don't care about losing packets. I may want to encrypt or digitally sign your messages. So, I end up doing a lot of work to handle queueing locally, but using a pretty simple message-oriented binary protocol to send to the server. The server can do whatever it wants, e.g. write it to a Kafka queue. Amazon's IoT framework nails a lot of these points. https://aws.amazon.com/blogs/compute/building-an-aws-iot-core-device-using-aws-serverless-and-an-esp32/ https://aws.amazon.com/blogs/compute/building-an-aws-iot-cor...
ar9av 4y ago
I've been combing the interwebs and following countless tutorials for different types of IoT data gather solutions using various messaging services and brokers for an scale weighment system that I've been developing using VueJS (eventually looking migrate to Nuxt). So far the concept is simple, a weighment scale has an RS232 (COM) port that streams data out for the tare, net, and gross weights using some kind of micro-controller architecture, Raspberry Pi's, Arduino, PyBoard etc. via some sort of messaging service. Clients connect to a page where they can "pub/sub" to scales to see the current weight of specific scales and using a remote device. The page acts as a UI to either manage inventory, register consumed ingredients, or register completed tasks. So far most of the development scripts and test environments I've built all seem to do basically the same thing and will gather the data that's needed but I'm curios if anyone has dealt with anything like this in the wild and if there are any caveats that I'm missing or other technologies that would be better suited for what I'm trying to achieve? Right now Kafka or RabbitMQ seem to be my main choices for message brokers mostly because they are fairly easy to setup via Docker. If anyone has any recommendation on libraries I should look into that would be awesome! The UI is coming together nicely, I started it in React but switched to Vue3 after I fell in love with the component architecture and composition API.
- mighty_plant 4y ago
  I once had to develop something similar. The solution my company had chosen was to transmit the weights and possible errors via a Websocket connection between the UI and the scale adapter - a small Java service doing the raw socket communication. Weirdly the commands to stop weighing and set tare, get tare, etc. would be send via RabbitMQ from the UI server to the adapter instead of using the WS connection.
- wmeitzler 4y ago
  Having been down this road, I strongly recommend you lock down your requirements before drowning in the sea of transport options. There are a lot of offerings out there that, from the device perspective, basically amount to a reinvention of HTTP(S). Many of these systems exist with the intent to retain the underlying TCP (really, the expensive-to-establish TLS) stream, which is not as much of a concern after HTTP 1.2 retained existing connections. What are your latency requirements for RS232 to DOM paint? Are you storing time-series data, or just the last value? How fast does the data change, or, when do you want to be notified? Do you have circumstances that periodically disconnect your device from the Internet? It doesn't sound like you have a constrained environment i.e. a cellular connection or battery budget. It also sounds like this is a small-volume project, that you are not building the next Twitter. If you are interested in historical data, do not overlook the value of generating a larger file on-edge, compressing it, and sending it with a boring-ass HTTP POST. If you want "fast", just set a trigger (i.e. the scale changed by more than 3 ounces in 1 second) to send a small update POST. Periodically perform a GET for config updates, and you're off to the races. Keeping your messaging layer stone-cold boring enables you to use whatever whiz-bang backend you want, and leverage existing knowledge and tools of the HTTP world. Or just slap it in a database. If you're interested in time series, check out Timescale for Postgres.
Animats 4y ago
Controlling emergency generators via AWS is scary.
jinmingjian 4y ago
MQTT and Kafka are different things. MQTT is not necessarily better or worse than Kafka, vice versa. But, sadly, people are often caught in their own loops, when there are or can be created better options. I have created a new, free, single binary data-service platform for IoT, JoinBase: https://joinbase.io/ https://joinbase.io/ This single binary data-service platform has proved: 1. Kafka is not more suitable for industry or higher performance than MQTT, even from the protocol level With carefully crafting, JoinBase has saturated one PCIE 3.0 NVME sustained write bandwidth (25 million msg/s) in single modern node. This is, in fact, can not been done by the Java-based Kafka. We have provided FREE full functionality community for testing: https://joinbase.io/products/ https://joinbase.io/products/ 2. Use MQTT and Kafka together is unnecessary. This only makes your pipeline more complex, expensive, but much unstable and slower. JoinBase can do arbitrary message preprocessing and auto-view(WIP). Streaming does not have to be owned through a separate monster. 3. High performance or ease of use, has nothing to do with the size of the software if the product can be properly engineered. 5MB Single binary JoinBase is enough to beat many monsters in the IoT/AIoT data pipleine: + sustained batch MQTT message write throughput: ~10x faster than Kafka and ~5x faster than that of one popular broker + basic SQL analytics: 3-4x faster than ClickHouse + HTTP interface concurrent queries: ~100x higher than ClickHouse + ... More could be seen in our 2022 summary blog: https://joinbase.io/blog/joinbase-2023/ https://joinbase.io/blog/joinbase-2023/ There are historical reasons for all of these, of course. But it could be great that we break out of own mindset loops.
amrx101 4y ago
Another MQTT vs Kafka. It's been re-iterated many times that MQTT and Kafka solve different problems.
kerblang 4y ago
Since nobody else mentioned it: Often you'll want to have multiple consumers of a given topic for load-balancing/failover - that's on the consumer side. The support for this in the MQTT standard is poorly defined, and popular libraries handle it poorly to the point that I can't recommend trying it. Message queues right-and-proper load-balance consumers by locking the entire topic once per delivery; Kafka does it by assigning "partitions" of a topic to different consumers, and thus you have faster, lock-free delivery (Kafka is essentially a "hot-rodded" message queue, just like hot-rodding a car by removing certain "proper" guard rails).

155 points by teleforce 4y ago

This article appears to be comparing MQTT and Kafka + Schema Registry. Using Schema Registry is not required to use Kafka, so OP overcomplicated their own set up for this comparison. There's no argument that Schema Registry is valuable, but it's not something that MQTT seems to provide out of the box, so the comparison seems flawed. I'd be interested in a comparison that is actually apples-to-apples instead of introducing complexity with Schema Registry.

rad_gruchalski 4y ago
Comparing mqtt to kafka is already apples vs oranges. Adding the schema registry to this is like throwing a pitaya into the mix. Edit: after reading the article a couple of times it’s clear this isn’t a comparison. “Vs” in the title is the problem. The first impression would have been better if the title was something like ”Kafka and MQTT“. To be honest, as kafka and mqtt often reside next to each other, they complement each other. A better use case would have been an app combining both: show when should mqtt hand over to kafka.
- deleted 4y ago
  [deleted]
- stingraycharles 4y ago
  Can confirm, I work with several workloads where MQTT is used to aggregate IoT data close to the source, which is then pushed into the cloud which uses Kafka.
ryanjshaw 4y ago
The author is also incorrect about message keys - these are optional and you would only use them for strict ordering when using > 1 partition, or log compaction where "latest is greatest" is good enough.

justinclift 4y ago

An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.

sigwinch28 4y ago
I wonder... how many issues was Kafka "soaking up" by dealing with concerns that applications and services didn't have to even consider? As in, I wonder how much application developer burden would be present if using MQTT instead.
- justinclift 4y ago
  It's an interesting question. No idea how to go about quantifying it though.
  - sigwinch28 4y ago
    Fair.
  - JUNGLEISMASSIVE 4y ago
    [dead]
- tsss 4y ago
  The vast majority of "Kafka problems" (or rather "Kafka Streams problems") we have with a managed solution at work are due to not fully understanding how it works and how to set it up. There's so much stuff you can configure and so much potential to misuse it. Typical problems are wrong configuration for acks, not understanding durability guarantees, not understanding exactly-once semantics, not naming hidden topics in Kafka Streams, not using schema registry serializers for hidden topics, choosing the wrong partitioning scheme (and wanting to change it later), using Kafka clients with different hash functions, using wrong schema compatibility mode, etc. etc.

mrkeen 4y ago

I jumped onto https://mqtt.org/ https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.

deleted 4y ago
[deleted]
avereveard 4y ago
> advertising the impossible eh if you read the finer print it's just a deduplication id appended to every message. blog doesn't go into detail on what happen when two client pust a message with the same it, or what happens if there is more than one failure (i.e. client fails to detect a service outage and during the service outage the message is consumed by the broker but persisting fails) but in general the usage of a at least once + a deduplication id is not something revolutionary.
alecthomas 4y ago
> I'm a big fan of advertising the impossible on the front page. Do you mean like Confluent do? https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ https://www.confluent.io/blog/exactly-once-semantics-are-pos...
- codeflo 4y ago
  The impossibility of "exactly once" is a theorem, not an opinion. Knowing that, the article you linked is funny. It begins with maybe a thousand filler words complaining that all of this is "poorly understood", and it's not "impossible", just "very hard". Then it gets to the meat: Yeah well, you know, it's not quite "exactly once delivery", just "exactly once semantics", and to achieve that, messages need to be idempotent so that duplicates don't matter. We all know that. It's called "at least once".

hkt 4y ago

A better comparison with Kafka is redis streams. Similar semantics, a fraction of the operational overhead.

FridgeSeal 4y ago
Provided your surrounding tools plug into Redis streams. Oh and provided you don’t need the ordering and parallelism guarantees of Kafka’s partitions. Oh and provided you don’t need the same level of durability and fault tolerance, so yeah, exactly the same.
- outworlder 4y ago
  Exactly. Tradeoffs exist everywhere, in both directions.

deleted 4y ago

[deleted]

gz5 4y ago

Good article (along with parts 2 and 3). Are there key differences in secure networking constructs (TLS, mTLS, VPN, whitelisted IPs, open ports, etc.) in the options described: + inbound to Kafka clusters and Kafka Connect? + inbound to Mosquitto MQTT broker? + inbound to Telegraf? + inbound to influxDB?

physicles 4y ago
Of those, MQTT should be the only thing you expose to the outside world. Secure it with regular TLS, possibly with mutual auth for your clients if you care about that. For the others, secure them using the same tech you use to secure your database. Ideally they wouldn't be accessible from the internet at all.

adev_ 4y ago

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost. Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions). If all you need is a simple stupid publish/subscribe broker with topics/auth management, do a favour to yourself, stay away from it.

NovemberWhiskey 4y ago
I see Kafka deployed for things which have perhaps a few thousand messages per day. It's like "did you accidentally mis-specify by six orders of magnitude here?"
MuffinFlavored 4y ago
> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received. How many of us get paid to work jobs where we're basically told "shut up, this is what we're doing/using, go with it"?
- rad_gruchalski 4y ago
  And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
  - justinclift 4y ago
    Maybe something AMQP related instead?

yawniek 4y ago

many people seem to not have clarity on what a distributed log is and in which architecture its useful and in which not. if you are abusing a distributed log as a message queue, you are most of the time creating a mess.

ed25519FUUU 4y ago
The "abuse" you describe is pretty much how Kafka is used everywhere I've worked.

twawaaay 4y ago

One missing criterion is client complexity. MQTT is built to work well with very little resources on the client. Kafka, on the other hand, requires you to do things you just don't want on a small embedded device -- like opening multiple connections to multiple hosts. Kafka is also just a transport for messages while MQTT is much larger part of the stack and takes care of transporting individual values. Which means you need less other code on your super restricted device. That said, I don't understand all the complaining directed at Kafka in this thread. Kafka is a fantastic tool that provides unique properties and guarantees. As a tech lead/architect I love to have a good selection of tools for different situations. Kafka is very reliable tool that fils an important role of when creating distributed systems and is particularly nice because it is easy to reason about. The negative opinions I heard in the past are typically from people who try to use it for something that it is not well suited for (like efficient transfer of large volumes of data) or because they misunderstood how to use its guarantees to construct larger systems. At one place I met a team who was completely lost with their overloaded Kafka instance and requested to get external help to "further scale and tune" it. I just touched the piece of code on producer and on consumer to publish data in large files to S3 rather than push it all through Kafka. Instead, send a simple message to Kafka with the metadata and location of the payload in S3. And then the client to download it from the bucket. They were happy puppies in no time.

anonymousDan 4y ago
Regarding your last point, how did you handle deletion from S3? Did you not need to worry about atomic consumption of the metadata and data? I suppose you could have some kind of background gc task..
- twawaaay 4y ago
  I think you are overcomplicating the problem for no reason. You build your system from simple guarantees: * message to Kafka is sent after the payload has been published to S3. This means if you have received message on Kafka, the payload is there, no need to worry about it -- you guarantee it because of order of publishing operations. * the object on S3 is immutable. This means it does not matter when you consume it, it stays the same. * the message on Kafka is immutable. This means it does not matter when you consume it, it stays the same. When the client reads the message off of Kafka topic, it just downloads the additional payload from S3. The payload is guaranteed to be there and exactly the same content as published. Once the message is fully processed, it commits this to Kafka topic and that's done. If the processing fails, the processing will be retried later by this or another node. The payload is still there until somebody decides to delete it. Deletion can be done in many different ways. You could have metadata for all those objects (Kafka topic is your metadata database!) and see what is the oldest timestamp on the offsets on all partitions still not committed. Then you delete from S3 all objects that are older that that. This requires that you publish to Kafka in the same order as you publish to S3 (within each partition).

skrtskrt 4y ago

there’s a lot of “Kafka causes so many issues!” comments here. I think it gets a bad rap because it gets introduced to orgs without the org having the requisite level of understanding. If your whole org is just on like a standard OLTP/OLAP setup, then suddenly there’s a Kafka queue, there’s going to he a serious learning curve and bumps along the way. If you’re incorrectly putting async event brokers as the datastore where you should be putting a synchronous DB and then streaming from the DB to kafka with an outbox pattern, you’re going to have a bad time. If you’re not modeling your queue depth and throughput you’re going to have a bad time. If you’re not modeling your concurrency scenarios and synchronization, you’re going to have a bad time.

speedgoose 4y ago

I hear from everyone using Kafka in production that it is hell unless you use Confluent. I gave a try to NATS JetStreams but I havn't been convinced by the performances of the Python client, nor the JavaScript one. I don't have extreme data, I just need descent performances. I'm thinking about giving a try to RabbitMQ streams. I have been very happy with RabbitMQ, the MQTT plugin isn't fully working (the big one is that retained messages are not sent to wildcard subscribers), but it should work with AMQP.

EdwardDiego 4y ago
Welp, here's a dissenting opinion - it's not. I've run self-managed, sorta managed (MSK), fully managed (Confluent Cloud), and somewhat managed (Strimzi). It is complex, yes, but it solves a very complicated problem. The issue tends to arise when people use it when simpler alternatives exist for their problem.
amath 4y ago
Have you tried Pulsar or Redpanda? Both seem mature enough and provide decent performance to probably meet your needs. What I hear is that Redpanda is a lot easier to manage than Kafka.
- speedgoose 4y ago
  I haven’t took the time but I should. Redpanda is written in C++ and I tend to prefer safer programming languages, redis being the exception.
- serial_dev 4y ago
  What's the license for Redpanda? I couldn't find anything but maybe it's because I'm on mobile.
  - deniscoady 4y ago
    Disclaimer, I work at Redpanda. Redpanda Community Edition is licensed with the Redpanda Business Source License (BSL). The core features are free and source-available. This license was inspired by MariaDB and CockroachDB and "for 99.999% of users, restrictions will not apply". The big restriction is that users "cannot provide Redpanda as a commercial streaming or queuing service to others" which is primarily to deter large cloud vendors from taking our work and impacting our ability to operate as a business. Here are some links for more details on how we came to this decision and how we license all our products and features. https://redpanda.com/blog/bsl-source-available-license https://redpanda.com/blog/bsl-source-available-license https://docs.redpanda.com/docs/get-started/licenses/ https://docs.redpanda.com/docs/get-started/licenses/

Kinrany 4y ago

Is there a ZeroMQ-style protocol for a Kafka-like event log? Messaging protocols are many, but can't provide the semantics necessary for async communication.

JaggerFoo 4y ago

I designed a facility IoT system with AWS products: IoT Core (MQTT broker), SiteWise (analytic dashboard), S3 and Lambda. I found the AWS offerings to have everything I needed in one place with a low cost. Added benefits were being able support NIST cybersecurity requirements (CMMC v2, Level 2) in addition to the IoT system. I was a fan of NATS and Kafka in the past, but AWS tooling makes IoT relatively easy. Cheers

physicles 4y ago

The blog post series seems to bury the lede -- it isn't until part 3 [1] that we get to the insight that MQTT and Kafka solve different problems and therefore can have complementary roles in the same system. We use this architecture for IoT: MQTT for edge because it's standard and super good enough, and Kafka because it turns out we want to do more than one thing with the data as it streams in so not using Kafka would end up being more complicated. Here's a key insight though: for IoT you don't want to use an actual MQTT broker, like Mosquitto or HiveMQ. If you do, it's hard to avoid data loss. Server side you have something that subscribes to those MQTT topics and pushes the data into Kafka. What do you do when that thing needs to be restarted? Ok, you can use persistent sessions in your MQTT broker. But how much memory does your MQTT broker need? What if your MQTT broker crashes? Oops, now your MQTT broker needs its own persistent database to keep track of all those messages in limbo. What you want is an MQTT gateway -- something that looks like an MQTT broker to the devices, but the server side does something different with received messages. When it gets the MQTT PUBLISH command, it sends the message to Kafka, waits for the ack from Kafka, and only then sends PUBACK back. Presto, the MQTT thing is now stateless and horizontally scalable. Your clients just need some retry logic. Maybe Kafka's mqtt-proxy does this, I don't know. I don't think it's mentioned in part 3. But it's a key property of such a system. I'm guessing Amazon's IoT gateway does this, because once you've thought about it hard enough it becomes obvious this is how it needs to work. 1: https://www.influxdata.com/blog/mqtt-vs-kafka-iot-advocates-perspective-part-3/ https://www.influxdata.com/blog/mqtt-vs-kafka-iot-advocates-...

sally_glance 4y ago
Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?
- FridgeSeal 4y ago
  If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, they’re just buffered (with all the extra overhead that entails). The stateful approach also hides system back pressure from clients, so in the event of network/storage/service degradation/failure, rather than backing off producing messages, or applying some other kind of application specific decision; clients keep producing messages as if nothing is wrong, because the stateful broker is buffering them locally. Storage and resource reqs for the broker continue to grow, until they either suddenly die, or stop accepting messages. Even worse, when downstream comes back up, the brokers are they going to dump all their messages and possibly re-overload recovering systems?

mstaoru 4y ago

Most MQTT brokers are not great at storing data. The pattern that works the best for me in solving the "edge delivery" use case is a lightweight clustered MQTT broker (e.g. VerneMQ) with a little Lua script inside to push everything into Redis Streams (of course, clustered as well) immediately. Kafka is another alternative, but it's not that lightweight, operationally, and often Redis Streams are "good enough". With an LB in front of both Verne and Redis, this setup is pretty decent for IoT data ingress.

jake_morrison 4y ago

I have done a lot of work with IoT-ish data, e.g. sending location and telemetry data from vehicles and remote sensors over unreliable cellular networks. I need the ability to queue data on the device to deal with patchy connectivity and some policy for deleting messages in the queue. For example, I might throw away unsent location updates that are "stale" to save space. I might need to prioritize some messages, e.g. "lithium-ion battery pack overheating". I may be running on embedded hardware that is too small to run Linux. The connection to the modem might be serial. Data size and bandwidth usage can make a difference. I might get 2MB/month for $3. Bytes count if I want to send frequent updates to get more precision on the location. I generally know exactly what messages I am sending, so using a compiled format like gRPC or COAP (https://en.wikipedia.org/wiki/Constrained_Application_Protocol https://en.wikipedia.org/wiki/Constrained_Application_Protoc...) can be better than JSON. I may need to get through multiple layers of network address translation, making it different to send messages back. So keeping a persistent TCP connection can help. But using TCP for one-off connections wastes bytes, so UDP can be useful if I don't care about losing packets. I may want to encrypt or digitally sign your messages. So, I end up doing a lot of work to handle queueing locally, but using a pretty simple message-oriented binary protocol to send to the server. The server can do whatever it wants, e.g. write it to a Kafka queue. Amazon's IoT framework nails a lot of these points. https://aws.amazon.com/blogs/compute/building-an-aws-iot-core-device-using-aws-serverless-and-an-esp32/ https://aws.amazon.com/blogs/compute/building-an-aws-iot-cor...

ar9av 4y ago

I've been combing the interwebs and following countless tutorials for different types of IoT data gather solutions using various messaging services and brokers for an scale weighment system that I've been developing using VueJS (eventually looking migrate to Nuxt). So far the concept is simple, a weighment scale has an RS232 (COM) port that streams data out for the tare, net, and gross weights using some kind of micro-controller architecture, Raspberry Pi's, Arduino, PyBoard etc. via some sort of messaging service. Clients connect to a page where they can "pub/sub" to scales to see the current weight of specific scales and using a remote device. The page acts as a UI to either manage inventory, register consumed ingredients, or register completed tasks. So far most of the development scripts and test environments I've built all seem to do basically the same thing and will gather the data that's needed but I'm curios if anyone has dealt with anything like this in the wild and if there are any caveats that I'm missing or other technologies that would be better suited for what I'm trying to achieve? Right now Kafka or RabbitMQ seem to be my main choices for message brokers mostly because they are fairly easy to setup via Docker. If anyone has any recommendation on libraries I should look into that would be awesome! The UI is coming together nicely, I started it in React but switched to Vue3 after I fell in love with the component architecture and composition API.

mighty_plant 4y ago
I once had to develop something similar. The solution my company had chosen was to transmit the weights and possible errors via a Websocket connection between the UI and the scale adapter - a small Java service doing the raw socket communication. Weirdly the commands to stop weighing and set tare, get tare, etc. would be send via RabbitMQ from the UI server to the adapter instead of using the WS connection.
wmeitzler 4y ago
Having been down this road, I strongly recommend you lock down your requirements before drowning in the sea of transport options. There are a lot of offerings out there that, from the device perspective, basically amount to a reinvention of HTTP(S). Many of these systems exist with the intent to retain the underlying TCP (really, the expensive-to-establish TLS) stream, which is not as much of a concern after HTTP 1.2 retained existing connections. What are your latency requirements for RS232 to DOM paint? Are you storing time-series data, or just the last value? How fast does the data change, or, when do you want to be notified? Do you have circumstances that periodically disconnect your device from the Internet? It doesn't sound like you have a constrained environment i.e. a cellular connection or battery budget. It also sounds like this is a small-volume project, that you are not building the next Twitter. If you are interested in historical data, do not overlook the value of generating a larger file on-edge, compressing it, and sending it with a boring-ass HTTP POST. If you want "fast", just set a trigger (i.e. the scale changed by more than 3 ounces in 1 second) to send a small update POST. Periodically perform a GET for config updates, and you're off to the races. Keeping your messaging layer stone-cold boring enables you to use whatever whiz-bang backend you want, and leverage existing knowledge and tools of the HTTP world. Or just slap it in a database. If you're interested in time series, check out Timescale for Postgres.

Animats 4y ago

Controlling emergency generators via AWS is scary.

jinmingjian 4y ago

MQTT and Kafka are different things. MQTT is not necessarily better or worse than Kafka, vice versa. But, sadly, people are often caught in their own loops, when there are or can be created better options. I have created a new, free, single binary data-service platform for IoT, JoinBase: https://joinbase.io/ https://joinbase.io/ This single binary data-service platform has proved: 1. Kafka is not more suitable for industry or higher performance than MQTT, even from the protocol level With carefully crafting, JoinBase has saturated one PCIE 3.0 NVME sustained write bandwidth (25 million msg/s) in single modern node. This is, in fact, can not been done by the Java-based Kafka. We have provided FREE full functionality community for testing: https://joinbase.io/products/ https://joinbase.io/products/ 2. Use MQTT and Kafka together is unnecessary. This only makes your pipeline more complex, expensive, but much unstable and slower. JoinBase can do arbitrary message preprocessing and auto-view(WIP). Streaming does not have to be owned through a separate monster. 3. High performance or ease of use, has nothing to do with the size of the software if the product can be properly engineered. 5MB Single binary JoinBase is enough to beat many monsters in the IoT/AIoT data pipleine: + sustained batch MQTT message write throughput: ~10x faster than Kafka and ~5x faster than that of one popular broker + basic SQL analytics: 3-4x faster than ClickHouse + HTTP interface concurrent queries: ~100x higher than ClickHouse + ... More could be seen in our 2022 summary blog: https://joinbase.io/blog/joinbase-2023/ https://joinbase.io/blog/joinbase-2023/ There are historical reasons for all of these, of course. But it could be great that we break out of own mindset loops.

amrx101 4y ago

Another MQTT vs Kafka. It's been re-iterated many times that MQTT and Kafka solve different problems.

kerblang 4y ago

Since nobody else mentioned it: Often you'll want to have multiple consumers of a given topic for load-balancing/failover - that's on the consumer side. The support for this in the MQTT standard is poorly defined, and popular libraries handle it poorly to the point that I can't recommend trying it. Message queues right-and-proper load-balance consumers by locking the entire topic once per delivery; Kafka does it by assigning "partitions" of a topic to different consumers, and thus you have faster, lock-free delivery (Kafka is essentially a "hot-rodded" message queue, just like hot-rodding a car by removing certain "proper" guard rails).