4 ms·

Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of

Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?

FridgeSeal 4y ago
If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, they’re just buffered (with all the extra overhead that entails). The stateful approach also hides system back pressure from clients, so in the event of network/storage/service degradation/failure, rather than backing off producing messages, or applying some other kind of application specific decision; clients keep producing messages as if nothing is wrong, because the stateful broker is buffering them locally. Storage and resource reqs for the broker continue to grow, until they either suddenly die, or stop accepting messages. Even worse, when downstream comes back up, the brokers are they going to dump all their messages and possibly re-overload recovering systems?
- sally_glance 4y ago
  Thx, now I get it - so the difference between the something and the gateway is that something would be an actual MQTT subscriber, while gateway is actually just a facade implementing the MQTT protocol backed by Kafka. Sounds neat, but I suspect conforming to the MQTT specs but not actually being a 'full' broker might be easier said than done... Basically it will have to simulate MQTT behaviour while actually living by the Kafka rules.

FridgeSeal 4y ago

If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, they’re just buffered (with all the extra overhead that entails). The stateful approach also hides system back pressure from clients, so in the event of network/storage/service degradation/failure, rather than backing off producing messages, or applying some other kind of application specific decision; clients keep producing messages as if nothing is wrong, because the stateful broker is buffering them locally. Storage and resource reqs for the broker continue to grow, until they either suddenly die, or stop accepting messages. Even worse, when downstream comes back up, the brokers are they going to dump all their messages and possibly re-overload recovering systems?

sally_glance 4y ago
Thx, now I get it - so the difference between the something and the gateway is that something would be an actual MQTT subscriber, while gateway is actually just a facade implementing the MQTT protocol backed by Kafka. Sounds neat, but I suspect conforming to the MQTT specs but not actually being a 'full' broker might be easier said than done... Basically it will have to simulate MQTT behaviour while actually living by the Kafka rules.