5 ms·

Eventual Business Consistency

134 points by cratermoon 3y ago

Terr_ 3y ago
Another approach to this kind of thing would be storing most of your data as a log of events ("event sourcing") and having "Retroactive Address Change" as its own business-process, one which can be linked to the other steps like sending a refund check or crediting an account. In that architecture, the database-row will often remain plain and boring with just one "current address." That "projection" (a result of events-so-far) would simply switch whenever the "Retroactive Address Change" event comes down the pipeline, which would be good enough for, say, the system that is sending out monthly statements. Anybody needing more, like the exact history of changes and realizations, would be running their own processor and projection, scanning the event-stream in a way suitable to their specific use-case.
- gtramont 3y ago
  Came here looking for this comment. It seems that Kent's re-discovering (re-inventing?) event sourcing…?
  - svilen_dobrev 3y ago
    it's event-sourcing re-discovering accounting, back to ~1500 Luca Pacioly maybe..
    - iamwil 3y ago
      He readily admits it as a rebranding of an old concept. Sometimes, we need things reframed so we can tell the story to people that aren't engineers.
- skrebbel 3y ago
  I have never seen a productive application of event sourcing at scale. The idea is super elegant but there’s so much extra overhead. Your data size is easily 10x bigger, unless things nearly never change anyway (and then why do it, right?). Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups by manual database edits / script runs gets harder because it means you’ll create a mismatch between the events and the main db, so you ought to inject events instead, but you’re under time pressure so what to do, and the list goes on. I wish these things weren’t issues because in a very real way event sourcing feels like the “obviously best way” to deal with pretty much any kind of structured data storage. I want it to succeed. Does anyone know of a system that successfully did/does event sourcing at scale? How did you deal with these downsides?
  - bob1029 3y ago
    > Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups ... Event sourcing is a completely different way of thinking about business information. Of course you are going to run into things that look like "downsides". Consider the desired semantics of event sourcing and how your concerns run directly in the face of these semantics. If you aren't comfortable with what qualitatively seem like "trivial" updates, then I don't think a large enough leap of faith has been made. We like to empathize with the machine but sometimes we take it a bit too far. Event sourcing absolutely works and it works at the most massive scales the best. If you can dedicate an entire wing of your 15th floor office to "enterprise messaging and events", this stuff 5000% works. It seems to be at the small scale that it fails most frequently. At small scale, it's usually just one developer who has enough motivation to see it through, and the moment they shift their focus it falls apart. I saw it work in manufacturing in ways I cannot do justice in HN comment size limits.
    - turnsout 3y ago
      Yes, the event model is the only thing that makes sense. Just wanted to say, I don't think Kent's idea is in conflict with the Event Sourcing pattern, as long as these "Effective/Posting" events are stored append-only. Once you start updating the "Effective" date, you're losing data (a customer might call back and ask what the old Effective date was, to understand a paper bill, and you've lost it).
    - bbkane 3y ago
      I would *love* to read a blog post/Google Doc/GitHub Gist/??? about this if you get the time to write one
  - cratermoon 3y ago
    > Does anyone know of a system that successfully did/does event sourcing at scale? The git version control system. > Even simple things such as fixing a typo require a Command to be stored as an Event etc etc Yup. It's a commit in git, but yes, there's a command and it's store in the git history as an event.
n0w 3y ago
The more I learn about the relational model from a first order logic perspective, the more I wonder what we would have ended up with if more people thought of data/databases in this way (as opposed to the more record/object school of thought). Event sourcing and CRDT like distributed data using something like the Event Calculus?
ckastner 3y ago
See also: https://en.wikipedia.org/wiki/Temporal_database https://en.wikipedia.org/wiki/Temporal_database
Galanwe 3y ago
This kind data modelling happens all the time in quantitative finance, where you need to run historical scenarios (backtests). We just call that "point in time" (PiT) data. There can be multiple dimensions of timestamping, but you often end up with: - published date: when was this information publicly available - received date: when was this information actually received in our systems - effective date: when does this information starts applying
- LAC-Tech 3y ago
  so bi-temporal extends to n-temporal. interesting...
  - cratermoon 3y ago
    There's an old saying "two is an impossible number". In programming, we either deal with one of a thing, or a collection of things. The cases where there are exactly two things are uncommon (cue HN coming up with a bunch of exceptions). In the case of points in time, even bi-temporal leaves out a third, implicit point: now.
    - pphysch 3y ago
      Fascinating. While 2 is indeed a rare quantity/cardinality, "2-ness" is a ubiquitous quality of digital systems.
      - Terr_ 3y ago
        By human choice, though. We could be using N different thresholds of electrical charge or current, but instead we define ranges as "0-enough" and "1-enough".
    - lordwarnut 3y ago
      The irony being of course that there are two scenarios.
      - 3y ago
sails 3y ago
In case anyone is wondering, if this isn't dealt with by the system designer, then what ends up happening far too often is the data team is tasked with inferring the sequence of events based on a series of clues across this and other business systems to come up with some kind of pre-eventual-consistency guess.
LAC-Tech 3y ago
Very cool article. It's a problem I knew kind of existed but never thought about formulating. I'm glad it's been at least called something since the 90s - this means maybe there's good stuff to read! Bit annoying as I'm writing an event store in rust and now I want to make it bitemporal. But annoying in a good way.
- cratermoon 3y ago
  > now I want to make it bitemporal Hint: make it n-temporal. Bi-temporality is then a special case easily handled.
  - refset 3y ago
    Except making n-dimensional indexes both fast and scalable is hard. Often its safest to build your way up, one index at a time, until you fully understand the workload requirements.
    - Groxx 3y ago
      N-dimensional event correction logic is also likely to be a lot more complicated than two, which you might be able to handle exhaustively with just a bit of care (and without breaking out the logical proof systems)
      - cratermoon 3y ago
        Probably not as hard and complex as changing the bi-temporal implementation to n-temporal later.
svilen_dobrev 3y ago
heh. when i read the title it suggested completely different tangent.. - that pure technical stuff is always eventualy consistent with business needs/goals. Sometimes very close (success). Sometimes very perpendicular (failure). Nevermind, bi(tri)-temporality is amazing concept (together with "relations Are Objects too" someone mentioned long time ago... ; Back in the days ~2006-7 i wrote a library [1] to allow for it over plain SQLalchemy.. in a bigger project that did not see light. Heh, recently i made a wrapper for xtdb for python [2].. for another project which may or may not happen too). But it is very difficult to grasp for common programmers, let alone common people. (Actualy, similar to time jitter, and other dimensional inconsistencies. wow-and-flutter). People like to think proceduraly i.e. in sequence.. Now, 20y later, temporality of things should be default way of handling data - not a quirky write-your-own-wrongly or use whatever weird extension.. have fun [1] https://github.com/svilendobrev/timed https://github.com/svilendobrev/timed + https://github.com/svilendobrev/dbcook https://github.com/svilendobrev/dbcook [2] https://github.com/svilendobrev/py-xtdb-datomic https://github.com/svilendobrev/py-xtdb-datomic
refset 3y ago
[Re-posting my comment from yesterday https://news.ycombinator.com/item?id=37000572 https://news.ycombinator.com/item?id=37000572] > Part of the reason it hasn’t taken off is because of the additional complexity it imposes on programmers. My take is that it's specifically the additional complexity it imposes on _database_ programmers - the impacts across storage design, foreign keys and schema evolution when all data is bitemporal are profound. SQL:2011 introduced "temporal tables" but the design was arguably incomplete and implementing it ubiquitously requires a real re-think of database internals, so we're now over a decade on and these capabilities are still far from commonplace. > However, I think part of the reason it hasn’t become more popular, given the benefits it brings, is just the name. No disagreement from me there! Disclosure: I work on xtdb.com, building a new database engine where bitemporality is the default. I also spoke with Kent about this last week and am running a webinar about bitemporality more generally on Thursday: https://attendee.gotowebinar.com/register/2960607012900067930?source=hacker-news https://attendee.gotowebinar.com/register/296060701290006793...
- refset 3y ago
  People here may also be interested to see this analysis of the state of SQL:2011 "temporal table" feature adoption: https://illuminatedcomputing.com/posts/2019/08/sql2011-survey/ https://illuminatedcomputing.com/posts/2019/08/sql2011-surve... I don't think much has really changed since, and I'm not sure Postgres is any closer to addressing this natively (although there have been extensions, e.g. https://github.com/scalegenius/pg_bitemporal https://github.com/scalegenius/pg_bitemporal).
piyh 3y ago
This is second nature to me with payroll processing.
- refset 3y ago
  Makes sense, payroll are usually the last people to find out when somebody's salary has changed :) Which databases have you used for it? Any tips?
pphysch 3y ago
It's tempting to go "wow bitemporality is a jolly great idea! Let's make a bitemporal database where every field is bitemporal" but this elides the significant jump in complexity/overhead of such systems. Bitemporality should be explicitly enabled on data where required, part of the database schema rather than default behavior. This could be a big chunk of your data, but unlikely to be 100%.
- refset 3y ago
  I mostly agree, but it's the same as the argument for immutability in systems more generally: if you can afford the RAM / GC / storage costs then great. Eventually though costs will decrease and new systems will routinely capitalize on the productivity benefits in spite of reduced peak efficiency. HTAP database systems illustrate this trend. But it's not clear to me that existing, popular database engines will ever be able to evolve to make bitemporality an efficient or easy default choice. And in the meantime countless person-years of effort are wasted rediscovering and solving the same time-related problems over and over. A world where bitemporality was (somehow) a realistic default choice would be a simpler world, I reckon.
  - pphysch 3y ago
    > A world where bitemporality was (somehow) a realistic default choice would be a simpler world, I reckon. I think MongoDB sorta proves this theory false: "let's avoid RDBMS schema/migration hell by making everything a flexible JSON document". As it turns out, JSON columns are a jolly good idea, but making that the dominant default i.e. MongoDB led to a bunch of other (perhaps unanticipated) issues. I believe it would be a similar situation with bitemporality. So yeah, it would be cool if e.g. Postgres had fast native support for bitemporal fields in a way that makes them accessible without changing the whole paradigm.
    - refset 3y ago
      I'm not convinced about that comparison. MongoDB entirely disregarded the relational model whereas bitemporality—as envisioned by Snodgrass et al. and standardised in SQL:2011—is merely an extension/evolution of it. > it would be cool if e.g. Postgres had fast native support for bitemporal fields in a way that makes them accessible without changing the whole paradigm 100% agreed with you there. And there has been various work on patches & extensions in this area previously. Unfortunately I'm not sure Postgres' internals will ever be able to flex hard enough to deliver a decent "default choice" experience. At least not until some other system has proven the UX is really worth the effort (which I guess MongoDB did similarly achieve with native JSON handling).
awinter-py 3y ago
have always wanted to see time / versioning / history exposed as a first class citizen in sql schemas and this invokes my eternal pining for datomic's approach to temporal history
proteal 3y ago
I built a similar system for one of my old insurance clients. Every 3 months we would send an updated list of what we wanted the insurance company to cover. This batching worked great because changes came in daily and without it we would have to amend the policy for each individual change. We tracked the date the client requested a change (posting date in the article) alongside an effective date (the date we would base cost calculations off of). Often the client would forget to report changes to us, so the reporting delay worked in our favor. For example, we would find out out on Friday that a client needed coverage starting the previous Monday. We would make sure there were no known losses and backdate coverage in our database. The insurance carriers would receive the report in a few months and be none the wiser. Carriers will backdate coverage like this, but it can be messy with the paperwork. Tracking posting dates, effective dates, and reporting dates really cut down on the complexity of managing the day-to-day insurance needs of my client.

134 points by cratermoon 3y ago

Another approach to this kind of thing would be storing most of your data as a log of events ("event sourcing") and having "Retroactive Address Change" as its own business-process, one which can be linked to the other steps like sending a refund check or crediting an account. In that architecture, the database-row will often remain plain and boring with just one "current address." That "projection" (a result of events-so-far) would simply switch whenever the "Retroactive Address Change" event comes down the pipeline, which would be good enough for, say, the system that is sending out monthly statements. Anybody needing more, like the exact history of changes and realizations, would be running their own processor and projection, scanning the event-stream in a way suitable to their specific use-case.

gtramont 3y ago
Came here looking for this comment. It seems that Kent's re-discovering (re-inventing?) event sourcing…?
- svilen_dobrev 3y ago
  it's event-sourcing re-discovering accounting, back to ~1500 Luca Pacioly maybe..
  - iamwil 3y ago
    He readily admits it as a rebranding of an old concept. Sometimes, we need things reframed so we can tell the story to people that aren't engineers.
skrebbel 3y ago
I have never seen a productive application of event sourcing at scale. The idea is super elegant but there’s so much extra overhead. Your data size is easily 10x bigger, unless things nearly never change anyway (and then why do it, right?). Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups by manual database edits / script runs gets harder because it means you’ll create a mismatch between the events and the main db, so you ought to inject events instead, but you’re under time pressure so what to do, and the list goes on. I wish these things weren’t issues because in a very real way event sourcing feels like the “obviously best way” to deal with pretty much any kind of structured data storage. I want it to succeed. Does anyone know of a system that successfully did/does event sourcing at scale? How did you deal with these downsides?
- bob1029 3y ago
  > Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups ... Event sourcing is a completely different way of thinking about business information. Of course you are going to run into things that look like "downsides". Consider the desired semantics of event sourcing and how your concerns run directly in the face of these semantics. If you aren't comfortable with what qualitatively seem like "trivial" updates, then I don't think a large enough leap of faith has been made. We like to empathize with the machine but sometimes we take it a bit too far. Event sourcing absolutely works and it works at the most massive scales the best. If you can dedicate an entire wing of your 15th floor office to "enterprise messaging and events", this stuff 5000% works. It seems to be at the small scale that it fails most frequently. At small scale, it's usually just one developer who has enough motivation to see it through, and the moment they shift their focus it falls apart. I saw it work in manufacturing in ways I cannot do justice in HN comment size limits.
  - turnsout 3y ago
    Yes, the event model is the only thing that makes sense. Just wanted to say, I don't think Kent's idea is in conflict with the Event Sourcing pattern, as long as these "Effective/Posting" events are stored append-only. Once you start updating the "Effective" date, you're losing data (a customer might call back and ask what the old Effective date was, to understand a paper bill, and you've lost it).
  - bbkane 3y ago
    I would *love* to read a blog post/Google Doc/GitHub Gist/??? about this if you get the time to write one
- cratermoon 3y ago
  > Does anyone know of a system that successfully did/does event sourcing at scale? The git version control system. > Even simple things such as fixing a typo require a Command to be stored as an Event etc etc Yup. It's a commit in git, but yes, there's a command and it's store in the git history as an event.

n0w 3y ago

The more I learn about the relational model from a first order logic perspective, the more I wonder what we would have ended up with if more people thought of data/databases in this way (as opposed to the more record/object school of thought). Event sourcing and CRDT like distributed data using something like the Event Calculus?

ckastner 3y ago

See also: https://en.wikipedia.org/wiki/Temporal_database https://en.wikipedia.org/wiki/Temporal_database

Galanwe 3y ago

This kind data modelling happens all the time in quantitative finance, where you need to run historical scenarios (backtests). We just call that "point in time" (PiT) data. There can be multiple dimensions of timestamping, but you often end up with: - published date: when was this information publicly available - received date: when was this information actually received in our systems - effective date: when does this information starts applying

LAC-Tech 3y ago
so bi-temporal extends to n-temporal. interesting...
- cratermoon 3y ago
  There's an old saying "two is an impossible number". In programming, we either deal with one of a thing, or a collection of things. The cases where there are exactly two things are uncommon (cue HN coming up with a bunch of exceptions). In the case of points in time, even bi-temporal leaves out a third, implicit point: now.
  - pphysch 3y ago
    Fascinating. While 2 is indeed a rare quantity/cardinality, "2-ness" is a ubiquitous quality of digital systems.
    - Terr_ 3y ago
      By human choice, though. We could be using N different thresholds of electrical charge or current, but instead we define ranges as "0-enough" and "1-enough".
  - lordwarnut 3y ago
    The irony being of course that there are two scenarios.
    - 3y ago

sails 3y ago

In case anyone is wondering, if this isn't dealt with by the system designer, then what ends up happening far too often is the data team is tasked with inferring the sequence of events based on a series of clues across this and other business systems to come up with some kind of pre-eventual-consistency guess.

LAC-Tech 3y ago

Very cool article. It's a problem I knew kind of existed but never thought about formulating. I'm glad it's been at least called something since the 90s - this means maybe there's good stuff to read! Bit annoying as I'm writing an event store in rust and now I want to make it bitemporal. But annoying in a good way.

cratermoon 3y ago
> now I want to make it bitemporal Hint: make it n-temporal. Bi-temporality is then a special case easily handled.
- refset 3y ago
  Except making n-dimensional indexes both fast and scalable is hard. Often its safest to build your way up, one index at a time, until you fully understand the workload requirements.
  - Groxx 3y ago
    N-dimensional event correction logic is also likely to be a lot more complicated than two, which you might be able to handle exhaustively with just a bit of care (and without breaking out the logical proof systems)
    - cratermoon 3y ago
      Probably not as hard and complex as changing the bi-temporal implementation to n-temporal later.

svilen_dobrev 3y ago

heh. when i read the title it suggested completely different tangent.. - that pure technical stuff is always eventualy consistent with business needs/goals. Sometimes very close (success). Sometimes very perpendicular (failure). Nevermind, bi(tri)-temporality is amazing concept (together with "relations Are Objects too" someone mentioned long time ago... ; Back in the days ~2006-7 i wrote a library [1] to allow for it over plain SQLalchemy.. in a bigger project that did not see light. Heh, recently i made a wrapper for xtdb for python [2].. for another project which may or may not happen too). But it is very difficult to grasp for common programmers, let alone common people. (Actualy, similar to time jitter, and other dimensional inconsistencies. wow-and-flutter). People like to think proceduraly i.e. in sequence.. Now, 20y later, temporality of things should be default way of handling data - not a quirky write-your-own-wrongly or use whatever weird extension.. have fun [1] https://github.com/svilendobrev/timed https://github.com/svilendobrev/timed + https://github.com/svilendobrev/dbcook https://github.com/svilendobrev/dbcook [2] https://github.com/svilendobrev/py-xtdb-datomic https://github.com/svilendobrev/py-xtdb-datomic

refset 3y ago

[Re-posting my comment from yesterday https://news.ycombinator.com/item?id=37000572 https://news.ycombinator.com/item?id=37000572] > Part of the reason it hasn’t taken off is because of the additional complexity it imposes on programmers. My take is that it's specifically the additional complexity it imposes on _database_ programmers - the impacts across storage design, foreign keys and schema evolution when all data is bitemporal are profound. SQL:2011 introduced "temporal tables" but the design was arguably incomplete and implementing it ubiquitously requires a real re-think of database internals, so we're now over a decade on and these capabilities are still far from commonplace. > However, I think part of the reason it hasn’t become more popular, given the benefits it brings, is just the name. No disagreement from me there! Disclosure: I work on xtdb.com, building a new database engine where bitemporality is the default. I also spoke with Kent about this last week and am running a webinar about bitemporality more generally on Thursday: https://attendee.gotowebinar.com/register/2960607012900067930?source=hacker-news https://attendee.gotowebinar.com/register/296060701290006793...

refset 3y ago
People here may also be interested to see this analysis of the state of SQL:2011 "temporal table" feature adoption: https://illuminatedcomputing.com/posts/2019/08/sql2011-survey/ https://illuminatedcomputing.com/posts/2019/08/sql2011-surve... I don't think much has really changed since, and I'm not sure Postgres is any closer to addressing this natively (although there have been extensions, e.g. https://github.com/scalegenius/pg_bitemporal https://github.com/scalegenius/pg_bitemporal).

piyh 3y ago

This is second nature to me with payroll processing.

refset 3y ago
Makes sense, payroll are usually the last people to find out when somebody's salary has changed :) Which databases have you used for it? Any tips?

pphysch 3y ago

It's tempting to go "wow bitemporality is a jolly great idea! Let's make a bitemporal database where every field is bitemporal" but this elides the significant jump in complexity/overhead of such systems. Bitemporality should be explicitly enabled on data where required, part of the database schema rather than default behavior. This could be a big chunk of your data, but unlikely to be 100%.

refset 3y ago
I mostly agree, but it's the same as the argument for immutability in systems more generally: if you can afford the RAM / GC / storage costs then great. Eventually though costs will decrease and new systems will routinely capitalize on the productivity benefits in spite of reduced peak efficiency. HTAP database systems illustrate this trend. But it's not clear to me that existing, popular database engines will ever be able to evolve to make bitemporality an efficient or easy default choice. And in the meantime countless person-years of effort are wasted rediscovering and solving the same time-related problems over and over. A world where bitemporality was (somehow) a realistic default choice would be a simpler world, I reckon.
- pphysch 3y ago
  > A world where bitemporality was (somehow) a realistic default choice would be a simpler world, I reckon. I think MongoDB sorta proves this theory false: "let's avoid RDBMS schema/migration hell by making everything a flexible JSON document". As it turns out, JSON columns are a jolly good idea, but making that the dominant default i.e. MongoDB led to a bunch of other (perhaps unanticipated) issues. I believe it would be a similar situation with bitemporality. So yeah, it would be cool if e.g. Postgres had fast native support for bitemporal fields in a way that makes them accessible without changing the whole paradigm.
  - refset 3y ago
    I'm not convinced about that comparison. MongoDB entirely disregarded the relational model whereas bitemporality—as envisioned by Snodgrass et al. and standardised in SQL:2011—is merely an extension/evolution of it. > it would be cool if e.g. Postgres had fast native support for bitemporal fields in a way that makes them accessible without changing the whole paradigm 100% agreed with you there. And there has been various work on patches & extensions in this area previously. Unfortunately I'm not sure Postgres' internals will ever be able to flex hard enough to deliver a decent "default choice" experience. At least not until some other system has proven the UX is really worth the effort (which I guess MongoDB did similarly achieve with native JSON handling).

awinter-py 3y ago

have always wanted to see time / versioning / history exposed as a first class citizen in sql schemas and this invokes my eternal pining for datomic's approach to temporal history

proteal 3y ago

I built a similar system for one of my old insurance clients. Every 3 months we would send an updated list of what we wanted the insurance company to cover. This batching worked great because changes came in daily and without it we would have to amend the policy for each individual change. We tracked the date the client requested a change (posting date in the article) alongside an effective date (the date we would base cost calculations off of). Often the client would forget to report changes to us, so the reporting delay worked in our favor. For example, we would find out out on Friday that a client needed coverage starting the previous Monday. We would make sure there were no known losses and backdate coverage in our database. The insurance carriers would receive the report in a few months and be none the wiser. Carriers will backdate coverage like this, but it can be messy with the paperwork. Tracking posting dates, effective dates, and reporting dates really cut down on the complexity of managing the day-to-day insurance needs of my client.