4 ms·

Won't this lead to inconsistent states if you don't do monotonically increasing tokens? I.e. your storage system has two nodes and there are two read-modify-wr

Won't this lead to inconsistent states if you don't do monotonically increasing tokens? I.e. your storage system has two nodes and there are two read-modify-write processes running. Process 1 acquires the first token "abc" and process two also acquires the token "abc". Now process 1 commits, the token is changed to "cde" and the change streamed to node 2. Due to network delay, the change to node 2 is delayed. Meanwhile process 2 commits to node 2 with token "abc". Node 2 accepts the change because it has not received the message from node 1 and your system is now in an inconsistent state. Note that this cannot happen in a scenario where we have monotonically increasing fencing tokens because that requirement forces the nodes to agree on a total order of operations before they can supply the fencing token.

computerfan494 2y ago
In the above description of optimistic locking, it is assumed that it is impossible to issue the same token to multiple clients. Nodes can agree that a given token has also never been issued before just like a monotonically increasing value. The nice property about non-monitonically-increasing tokens is that nodes may generate them without coordinating if you can make other assumptions about that system. A good example is when nodes use an ID they were assigned beforehand as part of the token generation, guaranteeing that the leasing tokens they mint will not conflict with other nodes' as long as node IDs are not reused.
- bootsmann 2y ago
  I have a hard time wrapping my head around what you are proposing here. Say client A requests data, they get the token a-abc. Then client B requests data, they get the token b-cde. Client A commits their write, does the storage reject it because they already issued another token (the one from client B) or does it accept it?
  - computerfan494 2y ago
    My understanding of what the OP was discussing is an optimistic locking system where the nodes only accept commits if the last issued token matches the token included in the commit. While agreeing on the last token requested requires coordination, unlike monotonically increasing tokens you could have well-behaved clients generate token content themselves without coordination. That may or may not be useful as a property.
    - bootsmann 2y ago
      Got it, thank you for clarifying this.
jmull 2y ago
"node1", "node2", and "storage" are three separate things in the distributed environment. Only storage accepts changes, and it's what verifies the incoming token matches the current token. So node2 doesn't get to accept changes. It can only send changes to storage, which may or may not be accepted by it.
- bootsmann 2y ago
  If the storage is a singular entity then this is not a distributed systems problem at all, no?

computerfan494 2y ago

In the above description of optimistic locking, it is assumed that it is impossible to issue the same token to multiple clients. Nodes can agree that a given token has also never been issued before just like a monotonically increasing value. The nice property about non-monitonically-increasing tokens is that nodes may generate them without coordinating if you can make other assumptions about that system. A good example is when nodes use an ID they were assigned beforehand as part of the token generation, guaranteeing that the leasing tokens they mint will not conflict with other nodes' as long as node IDs are not reused.

bootsmann 2y ago
I have a hard time wrapping my head around what you are proposing here. Say client A requests data, they get the token a-abc. Then client B requests data, they get the token b-cde. Client A commits their write, does the storage reject it because they already issued another token (the one from client B) or does it accept it?
- computerfan494 2y ago
  My understanding of what the OP was discussing is an optimistic locking system where the nodes only accept commits if the last issued token matches the token included in the commit. While agreeing on the last token requested requires coordination, unlike monotonically increasing tokens you could have well-behaved clients generate token content themselves without coordination. That may or may not be useful as a property.
  - bootsmann 2y ago
    Got it, thank you for clarifying this.

jmull 2y ago

"node1", "node2", and "storage" are three separate things in the distributed environment. Only storage accepts changes, and it's what verifies the incoming token matches the current token. So node2 doesn't get to accept changes. It can only send changes to storage, which may or may not be accepted by it.

bootsmann 2y ago
If the storage is a singular entity then this is not a distributed systems problem at all, no?