3 ms·

I was very bullish on event sourcing, based on my experiences with databases that struggled with historical queries. Since then I've recognised that they come w

I was very bullish on event sourcing, based on my experiences with databases that struggled with historical queries. Since then I've recognised that they come with a lot of effort and don't play nicely with existing systems. I've also learned that, as usual, I didn't know enough and that others had gotten to this problem and thought about it more thoroughly than I ever could. > These questions could be answered in seconds if we had a full history. I agree, but that does not necessarily require an event sourcing design. I think the better fit here is bitemporal databases. A bitemporal database can answer pretty much any query about what is currently true, what was true and what will be true in future; about the story of a sequence or the existence of a sequence or of a point in time; about not only what "is" or "was" true but when you believed it to be true. So I can not only ask "what was the total value of the cart last thursday", I can ask "what did we believe the total value was, before we applied a correction?" I've recently given this general area of systems a lot of thought, trying to square away several different schools of thought around data. Principally stream processing in the style of Akidau/Chernyak/Lax, bitemporal databases as described by Snodgrass and dimensional modelling as described by Kimball. Plus some ideas pinched from Concourse and a hasty skimming of Enterprise Integration Patterns by Hohpe, Woolf et al. As it happens I will be trying to pitch some of this lunatic handwaving to colleagues this coming week. My basic goal is that you should have pluggability without needing to rewrite the universe. Streaming without giving up tables. Tables without having to convert them always to streams. Different views of data treated on their own terms, in their own form, without elevating one or the other to being The One True Way Of Managing Data. Alternatively, I'm wrong.

ryanmarsh 8y ago
Since then I've recognised that they come with a lot of effort and don't play nicely with existing systems. I'm curious if you think this is because the primitives don't exist (seems everyone doing CQRS/ES roll their own) or if it isn an "essential complexity" of doing CQRS/ES? I'm leaning more towards the former and less of the latter. Although I'm only a month down the road of implementing an CQRS/ES system.
- jacques_chester 8y ago
  A mix. Accidental complexity plays a role, it's just hard to boil an ancient ocean of data. But I have also come to think that streaming or eventing systems have come to overstate the argument that the stream is the "true" system. It is and it isn't. The analogy a lot of folk hit on before I did is to calculus, including streaming folks. But I think it actually proves that in "streams and tables", neither is the truest of them all. I can take the instantaneous velocity of an object, or I can take it over a span of time, or I can calculate its acceleration, or perhaps the distance that has been traveled. These are all functions that can be reached by deriving or integration from each other. But none of them is the high lord master formula. They are just different representations that make sense in different cases.
  - deleted 8y ago
    [deleted]

ryanmarsh 8y ago

Since then I've recognised that they come with a lot of effort and don't play nicely with existing systems. I'm curious if you think this is because the primitives don't exist (seems everyone doing CQRS/ES roll their own) or if it isn an "essential complexity" of doing CQRS/ES? I'm leaning more towards the former and less of the latter. Although I'm only a month down the road of implementing an CQRS/ES system.

jacques_chester 8y ago
A mix. Accidental complexity plays a role, it's just hard to boil an ancient ocean of data. But I have also come to think that streaming or eventing systems have come to overstate the argument that the stream is the "true" system. It is and it isn't. The analogy a lot of folk hit on before I did is to calculus, including streaming folks. But I think it actually proves that in "streams and tables", neither is the truest of them all. I can take the instantaneous velocity of an object, or I can take it over a span of time, or I can calculate its acceleration, or perhaps the distance that has been traveled. These are all functions that can be reached by deriving or integration from each other. But none of them is the high lord master formula. They are just different representations that make sense in different cases.
- deleted 8y ago
  [deleted]