3 ms·

Don't store the state like this. Just append an update to an immutable data store, and program the logic to unfold from the latest records. See Diatomic: http

by aeonik 3y ago

Don't store the state like this. Just append an update to an immutable data store, and program the logic to unfold from the latest records. See Diatomic: https://docs.datomic.com/pro/getting-started/brief-overview.html https://docs.datomic.com/pro/getting-started/brief-overview....

convolvatron 3y ago
i like this model, but i think the goal was to enforce a constraint (non-negative balance) and fail the operation, reporting that result to the requester. how do you do that in the monotonic style?
- pico303 3y ago
  You don’t. The original suggestion isn’t a good one for a constrained system like bank balances.
- btown 3y ago
  In theory, in event sourcing, a reader would enumerate every proposed transaction, in order, and skip any invalid ones (or ones invalidated by a separate invalidation stream). This is expensive, so a materialized Balances table might be created and maintained - not by the database per se, but by a dedicated stream ingesting process. The user might also asynchronously see a set of failed operations - synchronous is harder, but doable if you block on polling for the stream processing result, or have an “await key” primitive. Or just wait a second or two to refresh the page. There’s a lot of machinery involved, and a subtle logic change or error could invalidate balances for your entire client base. The transactional approach, with event sourcing used as an audit utility used by humans to verify correctness, is often a better path.
  - maxbond 3y ago
    This episode of Signals & Threads discusses a system (an exchange for OTC securities) which is both event sourced and which has constraints. It works in the same way you've described modulo some implementation details. https://signalsandthreads.com/state-machine-replication-and-why-you-should-care/ https://signalsandthreads.com/state-machine-replication-and-... Most of us are never going to work with a system with the scale and SLA to necessitate this approach, but it's an enjoyable interview nonetheless.
  - Salgat 3y ago
    A separate invalidation stream sounds like a really bad idea and is asking for race conditions. Just put the invalidation events on the same stream with the information necessary to revert the original transaction (such as the balance to restore in addition to stating the original transaction is invalidated); it's okay to denormalize on the same stream in these situations.
    - convolvatron 3y ago
      having a separate 'acceptor' log onto the same history actually works. thanks.
jimbob45 3y ago
That sounds like you’re suggesting Event Sourcing?
Spiwux 3y ago
Are you suggesting everybody should use event sourcing instead of using transactions? Because the general consensus on event sourcing is "don't use it unless you absolutely have to."
- strken 3y ago
  Unfortunately financial systems are the one example everyone uses for transactions, and also the most obvious place where event sourcing is close to mandatory.
- kmeisthax 3y ago
  If you maintained financial records the way that database example code says you should, you'd be a terrible accountant. Standard practice for accountancy is to record each transaction twice: once for the debited account and again for the credited account. You even create accounts for expenses and income; which are explicitly allowed to go negative (otherwise you couldn't record income). This is known as double-entry accounting, and it's equivalent to what you deride as "event sourcing". To put it back into the language of databases, double-entry accounting is the third normal form of accounting. You wouldn't store a "sum total of blog posts on a website" row in your database schema now would you? Why would you have a "sum total of transactions" or single entries for your transactions that only show that money came in but not where it went?
  - crdrost 3y ago
    I think you are objecting to a poorly chosen example and that is okay. It _is_ a poorly chosen example. The exact pedantry that you are making is kind of poorly chosen, in double-entry bookkeeping you would say that you record the transaction proper either once or n + 1 times, depending on how pedantic you are being. (Double-entry bookkeeping discovered the feature that in filesystems is called “journaling,” you always write first to the transaction log, which is the “1” if you answer 1, then you update the n account ledger pages involved in the transaction with their individual deltas plus the transaction log ID.) A real bank also allows accounts to go negative all the time, which is another criticism. The “event sourcing” is actually just the transaction log, which is not double entry, which is why I say that the pedantry is poorly chosen. The ledger, in combination with “closing the books for the year,” actually instantiates a generic construction for persisting data structures, see lectures at [1], where you take periodic snapshots of a value to limit your worst-case reconstruction time while storing a tractable buffer of deltas. The point of the ledger thus is to denormalize the data to make it efficient to answer the question, “how did this client's balance change over time?” without seeking over the whole transaction log. [1] https://courses.csail.mit.edu/6.851/spring21/lectures/ https://courses.csail.mit.edu/6.851/spring21/lectures/
yowlingcat 3y ago
This is not just an ignorant idea but a stupid idea. Just because you can't figure out the right way to do it in vanilla SQL doesn't mean it can't be done [1]. This strawman is just a different form of the NoSQL strawman [2]. Edit: Ah, looks like I've struck a nerve with the event sourcing Datomic fanboys. Enjoy using your golden hammer and turning every problem into a nail! By the way, if you ever make my way into my organization and waste my time with this nonsense, I WILL fire you. Nobody should tolerate your desire to solve already-solved problems with poorer quality solutions you have selected for their novelty. At that point, I'll kindly ask you to please explain to me what a write ahead log is and what it does, because if you don't, I'll happily explain it for you. [1] https://brandur.org/idempotency-keys https://brandur.org/idempotency-keys [2] https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
- paranoidrobot 3y ago
  Re your edit: I suspect that down voter's issue with your post is the aggressive nature of it.
- deleted 3y ago
  [deleted]

by aeonik 3y ago

convolvatron 3y ago

i like this model, but i think the goal was to enforce a constraint (non-negative balance) and fail the operation, reporting that result to the requester. how do you do that in the monotonic style?

pico303 3y ago
You don’t. The original suggestion isn’t a good one for a constrained system like bank balances.
btown 3y ago
In theory, in event sourcing, a reader would enumerate every proposed transaction, in order, and skip any invalid ones (or ones invalidated by a separate invalidation stream). This is expensive, so a materialized Balances table might be created and maintained - not by the database per se, but by a dedicated stream ingesting process. The user might also asynchronously see a set of failed operations - synchronous is harder, but doable if you block on polling for the stream processing result, or have an “await key” primitive. Or just wait a second or two to refresh the page. There’s a lot of machinery involved, and a subtle logic change or error could invalidate balances for your entire client base. The transactional approach, with event sourcing used as an audit utility used by humans to verify correctness, is often a better path.
- maxbond 3y ago
  This episode of Signals & Threads discusses a system (an exchange for OTC securities) which is both event sourced and which has constraints. It works in the same way you've described modulo some implementation details. https://signalsandthreads.com/state-machine-replication-and-why-you-should-care/ https://signalsandthreads.com/state-machine-replication-and-... Most of us are never going to work with a system with the scale and SLA to necessitate this approach, but it's an enjoyable interview nonetheless.
- Salgat 3y ago
  A separate invalidation stream sounds like a really bad idea and is asking for race conditions. Just put the invalidation events on the same stream with the information necessary to revert the original transaction (such as the balance to restore in addition to stating the original transaction is invalidated); it's okay to denormalize on the same stream in these situations.
  - convolvatron 3y ago
    having a separate 'acceptor' log onto the same history actually works. thanks.

jimbob45 3y ago

That sounds like you’re suggesting Event Sourcing?

Spiwux 3y ago

Are you suggesting everybody should use event sourcing instead of using transactions? Because the general consensus on event sourcing is "don't use it unless you absolutely have to."

strken 3y ago
Unfortunately financial systems are the one example everyone uses for transactions, and also the most obvious place where event sourcing is close to mandatory.
kmeisthax 3y ago
If you maintained financial records the way that database example code says you should, you'd be a terrible accountant. Standard practice for accountancy is to record each transaction twice: once for the debited account and again for the credited account. You even create accounts for expenses and income; which are explicitly allowed to go negative (otherwise you couldn't record income). This is known as double-entry accounting, and it's equivalent to what you deride as "event sourcing". To put it back into the language of databases, double-entry accounting is the third normal form of accounting. You wouldn't store a "sum total of blog posts on a website" row in your database schema now would you? Why would you have a "sum total of transactions" or single entries for your transactions that only show that money came in but not where it went?
- crdrost 3y ago
  I think you are objecting to a poorly chosen example and that is okay. It _is_ a poorly chosen example. The exact pedantry that you are making is kind of poorly chosen, in double-entry bookkeeping you would say that you record the transaction proper either once or n + 1 times, depending on how pedantic you are being. (Double-entry bookkeeping discovered the feature that in filesystems is called “journaling,” you always write first to the transaction log, which is the “1” if you answer 1, then you update the n account ledger pages involved in the transaction with their individual deltas plus the transaction log ID.) A real bank also allows accounts to go negative all the time, which is another criticism. The “event sourcing” is actually just the transaction log, which is not double entry, which is why I say that the pedantry is poorly chosen. The ledger, in combination with “closing the books for the year,” actually instantiates a generic construction for persisting data structures, see lectures at [1], where you take periodic snapshots of a value to limit your worst-case reconstruction time while storing a tractable buffer of deltas. The point of the ledger thus is to denormalize the data to make it efficient to answer the question, “how did this client's balance change over time?” without seeking over the whole transaction log. [1] https://courses.csail.mit.edu/6.851/spring21/lectures/ https://courses.csail.mit.edu/6.851/spring21/lectures/

yowlingcat 3y ago

This is not just an ignorant idea but a stupid idea. Just because you can't figure out the right way to do it in vanilla SQL doesn't mean it can't be done [1]. This strawman is just a different form of the NoSQL strawman [2]. Edit: Ah, looks like I've struck a nerve with the event sourcing Datomic fanboys. Enjoy using your golden hammer and turning every problem into a nail! By the way, if you ever make my way into my organization and waste my time with this nonsense, I WILL fire you. Nobody should tolerate your desire to solve already-solved problems with poorer quality solutions you have selected for their novelty. At that point, I'll kindly ask you to please explain to me what a write ahead log is and what it does, because if you don't, I'll happily explain it for you. [1] https://brandur.org/idempotency-keys https://brandur.org/idempotency-keys [2] https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs

paranoidrobot 3y ago
Re your edit: I suspect that down voter's issue with your post is the aggressive nature of it.
deleted 3y ago
[deleted]