5 ms·

Regarding MQTT in general - is my reading of the spec correct in that there are actually no message ordering guarantees across different topics ? This would imp

Regarding MQTT in general - is my reading of the spec correct in that there are actually no message ordering guarantees across different topics ? This would imply that the common HA pattern of a single device publishing/receiving commands/status on multiple semantically-named topics (eg /dev001/onoff + /dev001/color) is actually a subtly incorrect use of the protocol? And the way to maintain ordering with the protocol is to have only a single topic for commands and a single topic for status updates? Because this is what I've concluded, but I'd love to be wrong!

Arnavion 3y ago
Correct. There is no ordering guarantee across topics, and no ordering guarantee across different QoS levels on the same topic. It would not be possible to shard topics across multiple nodes if there was an ordering requirement per client.
- mindslight 3y ago
  That is unfortunate. The spec is pretty clear, but I keep hoping someone will point out that some brokers make that guarantee regardless of it being absent in the spec. Instead it just feels like everyone just ignores that much usage is subtly incorrect, and it's fine in practice. I do understand the ability implied by the lack of guarantee, it just seems entirely unnecessary, especially for home automation. Sharding in general only seems useful for surveillance companies in the business of massively centralizing computing infrastructure to begin with. And furthermore with MQTT it seems similar scalability could be achieved by scaling out across clients rather than across topics. There is little need to disaggregate if you don't aggregate in the first place!
tveita 3y ago
I've been surprised that a lot of developers have no idea how to use a queue effectively. I think you are right to be suspicious of any "established patterns" you come across, and even more of fancily named queue-based paradigms. I haven't looked at the HA space, but I guess that explains this absolutely atrocious example they give of a Kafka schema in AsyncAPI. https://github.com/asyncapi/spec/blob/master/examples/streetlights-kafka-asyncapi.yml https://github.com/asyncapi/spec/blob/master/examples/street...
NortySpock 3y ago
Speaking only as a Home Assistant novice, not as someone who read the mqtt spec cover-to-cover... I assume the reason was to keep topics to a single responsibility; if a trigger wants to subscribe to light.state= [on,off] notifications, it only wakes up if that single topic gets notified. It would not have to process light.color notifications or light.brightness notifications. Sending the smallest payload to the "correct" topic was probably the goal, especially meaning that battery-operated devices only had to monitor the one topic. (Sending gets more complicated, but maybe it balances out in terms of reducing un-needed wakes due to events you do not care about.) But I agree, that means you lose atomic state change events, unless you also had a topic that had the entire state.
- mindslight 3y ago
  The MQTT spec is actually quite straightforward and not at all tiring, so if you do end up having a reason to read it, don't hesitate to dig in! (I'm not saying you need to though) I understand the rationale for multiple topics. I went down this road when I was trying to design the layout for my own MQTT-controlled device. I would have really liked taking advantage of topics as something closer to an in-system semantic model, rather than only using MQTT for a bare minimum stream and then stuffing more layers on top. FWIW It's not merely losing atomic state changes that's the problem, but rather that updates can end up out of order. For example sending the commands [/light/brightness = 30, /light/state = on] can end up turning the light on with the previous (perhaps full) brightness before dimming it. And mitigating this by creating an additional topic with the entire state undoes most of the advantages of splitting it out in the first place, leaving more complexity and resource usage for little gain.

Arnavion 3y ago

Correct. There is no ordering guarantee across topics, and no ordering guarantee across different QoS levels on the same topic. It would not be possible to shard topics across multiple nodes if there was an ordering requirement per client.

mindslight 3y ago
That is unfortunate. The spec is pretty clear, but I keep hoping someone will point out that some brokers make that guarantee regardless of it being absent in the spec. Instead it just feels like everyone just ignores that much usage is subtly incorrect, and it's fine in practice. I do understand the ability implied by the lack of guarantee, it just seems entirely unnecessary, especially for home automation. Sharding in general only seems useful for surveillance companies in the business of massively centralizing computing infrastructure to begin with. And furthermore with MQTT it seems similar scalability could be achieved by scaling out across clients rather than across topics. There is little need to disaggregate if you don't aggregate in the first place!

tveita 3y ago

I've been surprised that a lot of developers have no idea how to use a queue effectively. I think you are right to be suspicious of any "established patterns" you come across, and even more of fancily named queue-based paradigms. I haven't looked at the HA space, but I guess that explains this absolutely atrocious example they give of a Kafka schema in AsyncAPI. https://github.com/asyncapi/spec/blob/master/examples/streetlights-kafka-asyncapi.yml https://github.com/asyncapi/spec/blob/master/examples/street...

NortySpock 3y ago

Speaking only as a Home Assistant novice, not as someone who read the mqtt spec cover-to-cover... I assume the reason was to keep topics to a single responsibility; if a trigger wants to subscribe to light.state= [on,off] notifications, it only wakes up if that single topic gets notified. It would not have to process light.color notifications or light.brightness notifications. Sending the smallest payload to the "correct" topic was probably the goal, especially meaning that battery-operated devices only had to monitor the one topic. (Sending gets more complicated, but maybe it balances out in terms of reducing un-needed wakes due to events you do not care about.) But I agree, that means you lose atomic state change events, unless you also had a topic that had the entire state.

mindslight 3y ago
The MQTT spec is actually quite straightforward and not at all tiring, so if you do end up having a reason to read it, don't hesitate to dig in! (I'm not saying you need to though) I understand the rationale for multiple topics. I went down this road when I was trying to design the layout for my own MQTT-controlled device. I would have really liked taking advantage of topics as something closer to an in-system semantic model, rather than only using MQTT for a bare minimum stream and then stuffing more layers on top. FWIW It's not merely losing atomic state changes that's the problem, but rather that updates can end up out of order. For example sending the commands [/light/brightness = 30, /light/state = on] can end up turning the light on with the previous (perhaps full) brightness before dimming it. And mitigating this by creating an additional topic with the entire state undoes most of the advantages of splitting it out in the first place, leaving more complexity and resource usage for little gain.