4 ms·

"Designed for at most once, at least once and exactly once message delivery" Exactly once message delivery cannot be guaranteed from a theoretical point of vie

"Designed for at most once, at least once and exactly once message delivery" Exactly once message delivery cannot be guaranteed from a theoretical point of view (if it does exist then the Two Generals Problem can be solved, which has been proven to not have a solution). I don't know how MQTT can get around that, especially in an environment where transient network issues are expected.

bdamm 7y ago
Well, there's a confirmation sequence that's basically a three-phase commit.
- vector3-dot-no 7y ago
  But that would mean they solved a theoretical and/or practical problem in a practical way and who does that? /s
- jonquark 7y ago
  A QoS 2 flow is 2-phase (Publish and PubRec being the first phase and PubRel and PubComp being the second).
nurettin 7y ago
Clients can choose to ignore multiple deliveries of the same message. They keep the message id around until they get an acknowledgement from the server that it won't be sending any more of that message. This is part of the protocol. What it doesn't really guarantee is the order, which requires even more clever programming and more network exchanges.
jhallenworld 7y ago
MQTT may take infinite time to deliver a message on an unreliable network, plus you assume error detection is perfect. I think this is why it doesn't violate two generals.
swiftcoder 7y ago
Even with the 3-phase commit, almost every production MQTT broker sacrifices the "exactly once" requirement to achieve reasonable performance.
- jonquark 7y ago
  I'm not sure this statement is accurate. Certainly the MQTT broker I work on (IBM WIoTP Message Gateway) and a number of our competitors support QoS 2 in production (though I am aware a number of brokers don't support it).
  - Arnavion 7y ago
    I imagine there are servers that drop QoS 1 or 2 messages that never complete the protocol after X amount of time, so that they don't keep consuming resources indefinitely. MQTT 5 standardizes this by adding a message expiry interval field, though even there it's opt-in by the publisher and defaults to forever.
ustolemyname 7y ago
It's worth noting that this guarantee also only applies to messages between a client and broker. The protocol does not guarantee messages are consumed, only that they reached the broker. So if guaranteed delivery of a specific message to consumers is something you need MQTT tends not to be a good fit as you have to build application layer acknowledgements on everything.
firebacon 7y ago
[Edited] Two Generals does not apply since it deals with the problem of guaranteeing that two parties agree on the exact same consistent state after a finite amount of time/messages. MQTT's "exactly once" delivery is basically an eventually consistent mechanism -- it just guarantees that the message is eventually delivered to the broker which then does some simple deduplication. Eventually may be infinite. So, no violation of the space time continuum here... In other words, MQTT's exactly once delivery conceptually works as if the server would store a big hashmap of all message IDs it has seen once and the rejecting any duplicates using that hashmap. The client the simply keeps retrying with the same message ID forever. This guarantees that the message is received by the server "exactly once". The actual protocol is more involved, but the details are basically just an optimization to prevent the server from having to keep around the "received messages" hashmap forever. The problem that you're thinking of that MQTT of course doesn't solve is that it is impossible to perform any externally visible action (such as moving an actuator, storing a record into another database system or detonating a bomb) "exactly once" in the general case when you take into account the possibility of failures in between the two steps of performing the action and storing the information that you have performed it. You can either first commit and then perform the action, which means you can not safely retry after having failed between the commit and performing the action, because you might have already performed it ("at most once"). Or you can first perform the action and then commit, potentially redoing the action after a failed commit ("at least once"). There is, conceptually, no way around this unless we get help from the environment, such as an atomic perform+commit primitive. To see how this relates to two generals, think about the exchange of the "performed" bit between the external system and the storage as an exchange of messages that can be lossy. We can not come up with any algorithm that guarantees that the external system and the storage always agree on the state of the performed bit after any finite number of steps.

bdamm 7y ago

Well, there's a confirmation sequence that's basically a three-phase commit.

vector3-dot-no 7y ago
But that would mean they solved a theoretical and/or practical problem in a practical way and who does that? /s
jonquark 7y ago
A QoS 2 flow is 2-phase (Publish and PubRec being the first phase and PubRel and PubComp being the second).

nurettin 7y ago

Clients can choose to ignore multiple deliveries of the same message. They keep the message id around until they get an acknowledgement from the server that it won't be sending any more of that message. This is part of the protocol. What it doesn't really guarantee is the order, which requires even more clever programming and more network exchanges.

jhallenworld 7y ago

MQTT may take infinite time to deliver a message on an unreliable network, plus you assume error detection is perfect. I think this is why it doesn't violate two generals.

swiftcoder 7y ago

Even with the 3-phase commit, almost every production MQTT broker sacrifices the "exactly once" requirement to achieve reasonable performance.

jonquark 7y ago
I'm not sure this statement is accurate. Certainly the MQTT broker I work on (IBM WIoTP Message Gateway) and a number of our competitors support QoS 2 in production (though I am aware a number of brokers don't support it).
- Arnavion 7y ago
  I imagine there are servers that drop QoS 1 or 2 messages that never complete the protocol after X amount of time, so that they don't keep consuming resources indefinitely. MQTT 5 standardizes this by adding a message expiry interval field, though even there it's opt-in by the publisher and defaults to forever.

ustolemyname 7y ago

It's worth noting that this guarantee also only applies to messages between a client and broker. The protocol does not guarantee messages are consumed, only that they reached the broker. So if guaranteed delivery of a specific message to consumers is something you need MQTT tends not to be a good fit as you have to build application layer acknowledgements on everything.

firebacon 7y ago

[Edited] Two Generals does not apply since it deals with the problem of guaranteeing that two parties agree on the exact same consistent state after a finite amount of time/messages. MQTT's "exactly once" delivery is basically an eventually consistent mechanism -- it just guarantees that the message is eventually delivered to the broker which then does some simple deduplication. Eventually may be infinite. So, no violation of the space time continuum here... In other words, MQTT's exactly once delivery conceptually works as if the server would store a big hashmap of all message IDs it has seen once and the rejecting any duplicates using that hashmap. The client the simply keeps retrying with the same message ID forever. This guarantees that the message is received by the server "exactly once". The actual protocol is more involved, but the details are basically just an optimization to prevent the server from having to keep around the "received messages" hashmap forever. The problem that you're thinking of that MQTT of course doesn't solve is that it is impossible to perform any externally visible action (such as moving an actuator, storing a record into another database system or detonating a bomb) "exactly once" in the general case when you take into account the possibility of failures in between the two steps of performing the action and storing the information that you have performed it. You can either first commit and then perform the action, which means you can not safely retry after having failed between the commit and performing the action, because you might have already performed it ("at most once"). Or you can first perform the action and then commit, potentially redoing the action after a failed commit ("at least once"). There is, conceptually, no way around this unless we get help from the environment, such as an atomic perform+commit primitive. To see how this relates to two generals, think about the exchange of the "performed" bit between the external system and the storage as an exchange of messages that can be lossy. We can not come up with any algorithm that guarantees that the external system and the storage always agree on the state of the performed bit after any finite number of steps.