3 ms·

b) You're asking for occasional "eventual consistency" trouble when A's state lags or has moved on ahead of the event If you allow A's state to lag behind it's

b) You're asking for occasional "eventual consistency" trouble when A's state lags or has moved on ahead of the event If you allow A's state to lag behind it's own events, then how are you ever going to create a sane system? Surely A either has to be ahead or at the state that caused the event to emit, or events are pointless.

SideburnsOfDoom 4y ago
> A's state to lag behind it's own events, Real systems don't have just 2 services. There can be 100s and the "own events" assumption may not hold.
- delusional 4y ago
  Sure, but in a thin events model someone would "own" the events since otherwise the subscriber wouldn't know where to query the actual data. What would you even do with an event saying a customer changes address if querying that address then produces the old one. I'm genuinely curious how such an architecture would work. You don't have to respond directly here, but if you have any reference to further reading, I'd appreciate it.
  - SideburnsOfDoom 4y ago
    > I'm genuinely curious how such an architecture would work. Complex systems are the way that they are because they got that way over time. It is not my goal to defend or even characterise a system that I did not create. I am here telling you the issue that I saw: one event consumer, at an edge case, ran substantially behind another, and when they attempted to co-ordinate over http, this failed. And how it was successfully resolved: fatter events removed the need for co-ordination between these two altogether. This was IMHO a more elegant design - it avoided he issues of the the thin events.
    - delusional 4y ago
      Ah, so A and C where both subscribed to B, but during A's processing of the event it assumed C had already processed it and tried to look up some state. Is that correctly understood? This sounds more like an architectural deficiency (as you say probably from architectural decay) than a systematic design edge case. I can't quite understand what information A would need to get from C that could be included in the fat event but not simply queried from B.
      - SideburnsOfDoom 4y ago
        > Ah, so A and C where both subscribed to B, but during A's processing of the event it assumed C had already processed it and tried to look up some state. Is that correctly understood? yes, though you're down the rabbit-hole on this one issue. My point (aside from the fact we actually saw this specific issue and it took a long time to correctly diagnose) is that with thin events followed by a http query call-back, You're asking for occasional "eventual consistency" trouble. Data races will happen occasionally - this is inevitable in the design. At the tail end of the latency distributions, too fast or too slow, or service A is now having a blip, or you now hit the new version just deployed or whatever, things will go wrong by mis-sequencing in surprising and hard to follow ways (example given that you're fixating on) in complex real systems, and it's a win to avoid that chaos entirely, with fat events.
tremon 4y ago
If you allow A's state to lag behind it's own events That's a mischaracterization. A's state is not lagging its emitted events; instead, A's state may have been changed at the time A's event is processed.
- delusional 4y ago
  The comment I quoted says: > when A's state lags or has moved on ahead of the event That sounds like it can EITHER be ahead or behind. Specifically, I do not understand it as A's state can either lag OR be ahead, not that "lags" is a synonym for "moved ahead"
  - deleted 4y ago
    [deleted]
- SideburnsOfDoom 4y ago
  The "own events" was the faulty assumption. it's not always the same service that both emits the events, and is the place to go to over http for data. It "seems logical" to also build that store from listening to events, but it can cause issues as mentioned.

SideburnsOfDoom 4y ago

> A's state to lag behind it's own events, Real systems don't have just 2 services. There can be 100s and the "own events" assumption may not hold.

delusional 4y ago
Sure, but in a thin events model someone would "own" the events since otherwise the subscriber wouldn't know where to query the actual data. What would you even do with an event saying a customer changes address if querying that address then produces the old one. I'm genuinely curious how such an architecture would work. You don't have to respond directly here, but if you have any reference to further reading, I'd appreciate it.
- SideburnsOfDoom 4y ago
  > I'm genuinely curious how such an architecture would work. Complex systems are the way that they are because they got that way over time. It is not my goal to defend or even characterise a system that I did not create. I am here telling you the issue that I saw: one event consumer, at an edge case, ran substantially behind another, and when they attempted to co-ordinate over http, this failed. And how it was successfully resolved: fatter events removed the need for co-ordination between these two altogether. This was IMHO a more elegant design - it avoided he issues of the the thin events.
  - delusional 4y ago
    Ah, so A and C where both subscribed to B, but during A's processing of the event it assumed C had already processed it and tried to look up some state. Is that correctly understood? This sounds more like an architectural deficiency (as you say probably from architectural decay) than a systematic design edge case. I can't quite understand what information A would need to get from C that could be included in the fat event but not simply queried from B.
    - SideburnsOfDoom 4y ago
      > Ah, so A and C where both subscribed to B, but during A's processing of the event it assumed C had already processed it and tried to look up some state. Is that correctly understood? yes, though you're down the rabbit-hole on this one issue. My point (aside from the fact we actually saw this specific issue and it took a long time to correctly diagnose) is that with thin events followed by a http query call-back, You're asking for occasional "eventual consistency" trouble. Data races will happen occasionally - this is inevitable in the design. At the tail end of the latency distributions, too fast or too slow, or service A is now having a blip, or you now hit the new version just deployed or whatever, things will go wrong by mis-sequencing in surprising and hard to follow ways (example given that you're fixating on) in complex real systems, and it's a win to avoid that chaos entirely, with fat events.

tremon 4y ago

If you allow A's state to lag behind it's own events That's a mischaracterization. A's state is not lagging its emitted events; instead, A's state may have been changed at the time A's event is processed.

delusional 4y ago
The comment I quoted says: > when A's state lags or has moved on ahead of the event That sounds like it can EITHER be ahead or behind. Specifically, I do not understand it as A's state can either lag OR be ahead, not that "lags" is a synonym for "moved ahead"
- deleted 4y ago
  [deleted]
SideburnsOfDoom 4y ago
The "own events" was the faulty assumption. it's not always the same service that both emits the events, and is the place to go to over http for data. It "seems logical" to also build that store from listening to events, but it can cause issues as mentioned.