5 ms·

The blog post series seems to bury the lede -- it isn't until part 3 [1] that we get to the insight that MQTT and Kafka solve different problems and therefore c

The blog post series seems to bury the lede -- it isn't until part 3 [1] that we get to the insight that MQTT and Kafka solve different problems and therefore can have complementary roles in the same system. We use this architecture for IoT: MQTT for edge because it's standard and super good enough, and Kafka because it turns out we want to do more than one thing with the data as it streams in so not using Kafka would end up being more complicated. Here's a key insight though: for IoT you don't want to use an actual MQTT broker, like Mosquitto or HiveMQ. If you do, it's hard to avoid data loss. Server side you have something that subscribes to those MQTT topics and pushes the data into Kafka. What do you do when that thing needs to be restarted? Ok, you can use persistent sessions in your MQTT broker. But how much memory does your MQTT broker need? What if your MQTT broker crashes? Oops, now your MQTT broker needs its own persistent database to keep track of all those messages in limbo. What you want is an MQTT gateway -- something that looks like an MQTT broker to the devices, but the server side does something different with received messages. When it gets the MQTT PUBLISH command, it sends the message to Kafka, waits for the ack from Kafka, and only then sends PUBACK back. Presto, the MQTT thing is now stateless and horizontally scalable. Your clients just need some retry logic. Maybe Kafka's mqtt-proxy does this, I don't know. I don't think it's mentioned in part 3. But it's a key property of such a system. I'm guessing Amazon's IoT gateway does this, because once you've thought about it hard enough it becomes obvious this is how it needs to work. 1: https://www.influxdata.com/blog/mqtt-vs-kafka-iot-advocates-perspective-part-3/ https://www.influxdata.com/blog/mqtt-vs-kafka-iot-advocates-...

sally_glance 4y ago
Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?
- FridgeSeal 4y ago
  If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, they’re just buffered (with all the extra overhead that entails). The stateful approach also hides system back pressure from clients, so in the event of network/storage/service degradation/failure, rather than backing off producing messages, or applying some other kind of application specific decision; clients keep producing messages as if nothing is wrong, because the stateful broker is buffering them locally. Storage and resource reqs for the broker continue to grow, until they either suddenly die, or stop accepting messages. Even worse, when downstream comes back up, the brokers are they going to dump all their messages and possibly re-overload recovering systems?
  - sally_glance 4y ago
    Thx, now I get it - so the difference between the something and the gateway is that something would be an actual MQTT subscriber, while gateway is actually just a facade implementing the MQTT protocol backed by Kafka. Sounds neat, but I suspect conforming to the MQTT specs but not actually being a 'full' broker might be easier said than done... Basically it will have to simulate MQTT behaviour while actually living by the Kafka rules.

sally_glance 4y ago

Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?

FridgeSeal 4y ago
If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, they’re just buffered (with all the extra overhead that entails). The stateful approach also hides system back pressure from clients, so in the event of network/storage/service degradation/failure, rather than backing off producing messages, or applying some other kind of application specific decision; clients keep producing messages as if nothing is wrong, because the stateful broker is buffering them locally. Storage and resource reqs for the broker continue to grow, until they either suddenly die, or stop accepting messages. Even worse, when downstream comes back up, the brokers are they going to dump all their messages and possibly re-overload recovering systems?
- sally_glance 4y ago
  Thx, now I get it - so the difference between the something and the gateway is that something would be an actual MQTT subscriber, while gateway is actually just a facade implementing the MQTT protocol backed by Kafka. Sounds neat, but I suspect conforming to the MQTT specs but not actually being a 'full' broker might be easier said than done... Basically it will have to simulate MQTT behaviour while actually living by the Kafka rules.