6 ms·

Those two concepts are not mutually exclusive. Each change can be idempotent by marking some identifying information about the state it was trying to update.

Those two concepts are not mutually exclusive. Each change can be idempotent by marking some identifying information about the state it was trying to update. increase balance by 10 (it is currently 100) balance is now 110 increase balance by 10 (it is currently 100) <- duplicate event! event has already been applied balance is now 110 It can definitely break down at some point (maybe the user really did want to increase by 10 two times, they just clicked twice really fast?). But idempotency and change-based flow can live together.

Salgat 5y ago
This is where CQRS comes in. Every unique action has its own command id, so if a user clicks submit twice, the same command id is being sent twice and after the first success all future ones don't affect anymore change, they just return automatic success since that command id succeeded in the past. Combine this with versioning, where you assert an event stream version on write, and now you're able to affect change on a known state that cannot have changed (and if it did change, your command would fail and cause no persisted changes server-side).
- valenterry 5y ago
  That means commands have to be stored as well (at least for some time) in a non-eventual-consistent way?
  - dharmaturtle 5y ago
    If a command is successfully converted/persisted into an event, that event should contain the command's ID. When the command is replayed, the domain aggregate/event stream will contain that command ID, and will therefore reject/ignore the replayed command.
    - Salgat 5y ago
      Exactly. When you are affecting a change on a stream, it has to read the events anyways, so it will be able to tell if the command id is already associated with the stream since each event has the command id in its metadata.
    - valenterry 5y ago
      So just to get it right: 1. it means that there is only one instance that processes commands and 2. this instance checks all command ids if they appeared before (so keeps track of all ids in an efficient way)?
      - dharmaturtle 5y ago
        > it means that there is only one instance that processes commands Not necessarily. Depends on how you shard the aggregate/streams. Since aggregates _by definition_ maintain their invariants, you can have literally a shard for every aggregate. Overkill, but it illustrates the point. Since there's no state shared between aggregates, you can process each aggregate/stream on its own instance. > this instance checks all command ids if they appeared before (so keeps track of all ids in an efficient way) Yep. If you trust the client and don't mind losing old commands, this isn't too hard - use a monotonically increasing number (e.g. ms since the epoch) as part of the commandId. Any command with an id less than the commandId in the aggregate will be rejected as "out of date". Note that this serves as an optimistic concurrency check. If you don't trust the client (or you don't want to lose old commands issued on a possibly out of date view) and just use GUIDs as commandIds, then yes you'll need to keep track of some ids - but not necessarily all. Do you really need commandIds past 1000? (Depends on your domain.) You'll need to "roll up" the events into some summary view anyway to check/maintain business invariants, so keeping, say, the last 100 commandIds as part of that summary view is simple.
        valenterry 5y ago
        Thank you, that makes sense and a lot more clear to me. Now I only wish there would be a good alternatives for when you want to have multiple instances per aggregate (even if you may have to give up certain guarantees).
        dharmaturtle 5y ago
        Interesting... just to be clear, an _individual stream of user events_ is an aggregate. E.g. a user Bob Smith's events are independent of another user like Jane Doe. It isn't the case that the entire "User Table" is a single aggregate - each row in it is an aggregate. If you're trying to do multimaster db storage, like a phone app syncing with some cloud server, where an aggregate resides in two separate locations, this is where event sourcing shines. It's just like git, right, and so you can do all the stuff that git can do - merge event streams, rebase events... it's actually the reason why my pet project switched from an RDBMS to event sourcing. Related video: https://skillsmatter.com/skillscasts/1980-cqrs-not-just-for-server-systems https://skillsmatter.com/skillscasts/1980-cqrs-not-just-for-...
- codebeaker 5y ago
  I'm well aware that there's no absolute standards for this, but I've built a number of CQRS systems, and I didn't ever have idempotency IDs as a core tenant of the system. What literature, or framework or ecosystem holds this to be a hard-and-fast requirement of CQRS? Personally I've solved this problem similar to Git, modelling CQRS on top of Merkle trees so your system at least knows if your incoming edit (command|commit) is being applied on HEAD (no changes to the model since the command started) or to HEAD+n (needing to re-run some command validations, and "rebase" the incoming command).
  - regularfry 5y ago
    That's generating an idempotency ID on the fly, isn't it?
  - lolski 5y ago
    I don't think there is any "standard" that strictly governs the implementation of a CQRS system. We just need to be aware of the nuances of the fault-tolerance aspect in distributed systems: - idempotency is ONLY a hard-requirement if the events are transmitted via lossy channels such as network. ID is one out of several ways to ensure idempotency. If you're implementing CQRS for your distributed system then it's needed - otherwise, if the events aren't lossy and can be processed atomically, you don't need to ensure idempotency. think implementing CQRS for your user interface where events are just transmitted between different objects in memory rather than via network
valenterry 5y ago
> It can definitely break down at some point (maybe the user really did want to increase by 10 two times, they just clicked twice really fast?) Well, that is exactly what OP meant - in the context of a distributed system it is not "at some point" it is just a given.
thewakalix 5y ago
What if this is concurrent with "decrease balance by 10"? If the decrease occurs between the two increases, then the increases won't be detected as duplicates.
- FridgeSeal 5y ago
  It’s my understanding that these systems don’t use the value, they’ll instead reference some kind of “state id” that refers to the state of the system as the result is some previous, accepted, change. This way, even if a decrease races the duplicate increase, the duplicate is still correctly detected.

Salgat 5y ago

This is where CQRS comes in. Every unique action has its own command id, so if a user clicks submit twice, the same command id is being sent twice and after the first success all future ones don't affect anymore change, they just return automatic success since that command id succeeded in the past. Combine this with versioning, where you assert an event stream version on write, and now you're able to affect change on a known state that cannot have changed (and if it did change, your command would fail and cause no persisted changes server-side).

valenterry 5y ago
That means commands have to be stored as well (at least for some time) in a non-eventual-consistent way?
- dharmaturtle 5y ago
  If a command is successfully converted/persisted into an event, that event should contain the command's ID. When the command is replayed, the domain aggregate/event stream will contain that command ID, and will therefore reject/ignore the replayed command.
  - Salgat 5y ago
    Exactly. When you are affecting a change on a stream, it has to read the events anyways, so it will be able to tell if the command id is already associated with the stream since each event has the command id in its metadata.
  - valenterry 5y ago
    So just to get it right: 1. it means that there is only one instance that processes commands and 2. this instance checks all command ids if they appeared before (so keeps track of all ids in an efficient way)?
    - dharmaturtle 5y ago
      > it means that there is only one instance that processes commands Not necessarily. Depends on how you shard the aggregate/streams. Since aggregates _by definition_ maintain their invariants, you can have literally a shard for every aggregate. Overkill, but it illustrates the point. Since there's no state shared between aggregates, you can process each aggregate/stream on its own instance. > this instance checks all command ids if they appeared before (so keeps track of all ids in an efficient way) Yep. If you trust the client and don't mind losing old commands, this isn't too hard - use a monotonically increasing number (e.g. ms since the epoch) as part of the commandId. Any command with an id less than the commandId in the aggregate will be rejected as "out of date". Note that this serves as an optimistic concurrency check. If you don't trust the client (or you don't want to lose old commands issued on a possibly out of date view) and just use GUIDs as commandIds, then yes you'll need to keep track of some ids - but not necessarily all. Do you really need commandIds past 1000? (Depends on your domain.) You'll need to "roll up" the events into some summary view anyway to check/maintain business invariants, so keeping, say, the last 100 commandIds as part of that summary view is simple.
      - valenterry 5y ago
        Thank you, that makes sense and a lot more clear to me. Now I only wish there would be a good alternatives for when you want to have multiple instances per aggregate (even if you may have to give up certain guarantees).
        dharmaturtle 5y ago
        Interesting... just to be clear, an _individual stream of user events_ is an aggregate. E.g. a user Bob Smith's events are independent of another user like Jane Doe. It isn't the case that the entire "User Table" is a single aggregate - each row in it is an aggregate. If you're trying to do multimaster db storage, like a phone app syncing with some cloud server, where an aggregate resides in two separate locations, this is where event sourcing shines. It's just like git, right, and so you can do all the stuff that git can do - merge event streams, rebase events... it's actually the reason why my pet project switched from an RDBMS to event sourcing. Related video: https://skillsmatter.com/skillscasts/1980-cqrs-not-just-for-server-systems https://skillsmatter.com/skillscasts/1980-cqrs-not-just-for-...
codebeaker 5y ago
I'm well aware that there's no absolute standards for this, but I've built a number of CQRS systems, and I didn't ever have idempotency IDs as a core tenant of the system. What literature, or framework or ecosystem holds this to be a hard-and-fast requirement of CQRS? Personally I've solved this problem similar to Git, modelling CQRS on top of Merkle trees so your system at least knows if your incoming edit (command|commit) is being applied on HEAD (no changes to the model since the command started) or to HEAD+n (needing to re-run some command validations, and "rebase" the incoming command).
- regularfry 5y ago
  That's generating an idempotency ID on the fly, isn't it?
- lolski 5y ago
  I don't think there is any "standard" that strictly governs the implementation of a CQRS system. We just need to be aware of the nuances of the fault-tolerance aspect in distributed systems: - idempotency is ONLY a hard-requirement if the events are transmitted via lossy channels such as network. ID is one out of several ways to ensure idempotency. If you're implementing CQRS for your distributed system then it's needed - otherwise, if the events aren't lossy and can be processed atomically, you don't need to ensure idempotency. think implementing CQRS for your user interface where events are just transmitted between different objects in memory rather than via network

valenterry 5y ago

> It can definitely break down at some point (maybe the user really did want to increase by 10 two times, they just clicked twice really fast?) Well, that is exactly what OP meant - in the context of a distributed system it is not "at some point" it is just a given.

thewakalix 5y ago

What if this is concurrent with "decrease balance by 10"? If the decrease occurs between the two increases, then the increases won't be detected as duplicates.

FridgeSeal 5y ago
It’s my understanding that these systems don’t use the value, they’ll instead reference some kind of “state id” that refers to the state of the system as the result is some previous, accepted, change. This way, even if a decrease races the duplicate increase, the duplicate is still correctly detected.