3 ms·

Isn't having to replay every transaction in history to query current balance rather inefficient?

globular-toast 3y ago
It's a trade off. Immutability is a big enough win to pay the price of a few extra CPU cycles. In practice it's very quick anyway. Computers are very good at adding numbers together (it's almost like it's what they were made for!). I keep all my accounts for over 12 years in one file and I still don't notice any delay when calculating balances etc. If it does become a problem then one solution is to take sums at some point in time and start the ledger again paying each account from a contra account containing the previous balance.
- BillyTheKing 3y ago
  I mean, yes for personal accounts with a couple of thousand transactions - sure. But esp. in payments or in exchanges there can be accounts with millions of transactions a day. So you get a double-whammy - those are hot-accounts which also slow down the system, recalculating the balance for those accounts each time a transaction happens isn't an option
  - globular-toast 3y ago
    Another way to do it is to keep the balances online. This is how something like Bitcoin works which is essentially one giant public ledger. It's a classic time/memory trade off.
qznc 3y ago
You can cache it. ;)
- Solvency 3y ago
  technical speaking what does 10 years of transactions look like when it's "cached" with one current year still being raw text lines? how are the two reconciled?
  - airstrike 3y ago
    basically 9 years are cached and the last year is recalculated as needed. you have a source of truth for what the data looked like at moment T+9 years, you can just take as granted. and then process the transactions following that date "live" as needed
  - inopinatus 3y ago
    Table rotation on a periodic (e.g. annual or financial year) basis is a common strategy, with carried-over opening balance records at the top coded to a dummy account. So the journal for the last ten years may actually be in tables journal_2013 through journal_2023. You can also then move these quiescent partitions to cheaper storage. After a few years, even to read-only archival medium. It’s not completely unknown for financial institutions to have to refer back to a paper form to audit or restate older accounts from inception.
rockostrich 3y ago
Once a period of time is reconciled then you can pre-compute whatever information you need and then you're only computing the current balance for the current period of time that hasn't been reconciled.
Mister_Snuggles 3y ago
The act of "posting" a transaction updates the ledger to reflect the current balance. Think of the ledger as "Current state" and the transactions as "how we got there".
inopinatus 3y ago
Computers are astonishingly fast at summing integers.
- epcoa 3y ago
  They are not comparably astonishingly fast at retrieving that data.
  - inopinatus 3y ago
    Accounting data is so compact that the word “comparably” is doing all the heavy lifting in that statement. A large accounting system certainly prefers to have large main memory, but even in the worst case scenario, we read from persistent storage at GB/sec and the processor has no trouble keeping up. You don’t have to get particularly smart to operate a billion-row ledger; only beyond that does it start to be interesting.
- thfuran 3y ago
  They're pretty fast, but if everyone writes software that does 10,000x more computation than necessary on that basis, suddenly they're slow.
  - inopinatus 3y ago
    “If everyone is incompetent, all software will be bad”. Got it, thanks.
    - thfuran 3y ago
      You were the one advocating for incompetence.
      - inopinatus 3y ago
        That was not the incredible burn you hoped for, because it kinda suggests a lack of consideration for the practical design of a modern computer, in which memory is tiered in orders of magnitude of time latency from CPU registers on down through on-die caches and so forth, each tier effectively acting as a local cache for the next level. Writes are super expensive and bust cache lines and coherency all the way down to main memory and in many applications touch persistent storage and even introduce network latencies in distributed systems. By contrast, ALUs are stupendously well optimized for the baseline computational task of adding up integers. The difference between nanoseconds and milliseconds is vast. It is consequently very often cheaper to recalculate a sum using data already in cache memory than to save it and later fetch it, or (even worse) to read a value precalculated by a neighbouring computer. Furthermore, the distinction only matters for accounting systems at high scale. For the average small business accounting, it is de minimis: your computer performs more computation when resizing a browser window than it does in recalculating an account balance from scratch.

It's a trade off. Immutability is a big enough win to pay the price of a few extra CPU cycles. In practice it's very quick anyway. Computers are very good at adding numbers together (it's almost like it's what they were made for!). I keep all my accounts for over 12 years in one file and I still don't notice any delay when calculating balances etc. If it does become a problem then one solution is to take sums at some point in time and start the ledger again paying each account from a contra account containing the previous balance.

BillyTheKing 3y ago
I mean, yes for personal accounts with a couple of thousand transactions - sure. But esp. in payments or in exchanges there can be accounts with millions of transactions a day. So you get a double-whammy - those are hot-accounts which also slow down the system, recalculating the balance for those accounts each time a transaction happens isn't an option
- globular-toast 3y ago
  Another way to do it is to keep the balances online. This is how something like Bitcoin works which is essentially one giant public ledger. It's a classic time/memory trade off.

qznc 3y ago

You can cache it. ;)

Solvency 3y ago
technical speaking what does 10 years of transactions look like when it's "cached" with one current year still being raw text lines? how are the two reconciled?
- airstrike 3y ago
  basically 9 years are cached and the last year is recalculated as needed. you have a source of truth for what the data looked like at moment T+9 years, you can just take as granted. and then process the transactions following that date "live" as needed
- inopinatus 3y ago
  Table rotation on a periodic (e.g. annual or financial year) basis is a common strategy, with carried-over opening balance records at the top coded to a dummy account. So the journal for the last ten years may actually be in tables journal_2013 through journal_2023. You can also then move these quiescent partitions to cheaper storage. After a few years, even to read-only archival medium. It’s not completely unknown for financial institutions to have to refer back to a paper form to audit or restate older accounts from inception.

rockostrich 3y ago

Once a period of time is reconciled then you can pre-compute whatever information you need and then you're only computing the current balance for the current period of time that hasn't been reconciled.

Mister_Snuggles 3y ago

The act of "posting" a transaction updates the ledger to reflect the current balance. Think of the ledger as "Current state" and the transactions as "how we got there".

inopinatus 3y ago

Computers are astonishingly fast at summing integers.

epcoa 3y ago
They are not comparably astonishingly fast at retrieving that data.
- inopinatus 3y ago
  Accounting data is so compact that the word “comparably” is doing all the heavy lifting in that statement. A large accounting system certainly prefers to have large main memory, but even in the worst case scenario, we read from persistent storage at GB/sec and the processor has no trouble keeping up. You don’t have to get particularly smart to operate a billion-row ledger; only beyond that does it start to be interesting.
thfuran 3y ago
They're pretty fast, but if everyone writes software that does 10,000x more computation than necessary on that basis, suddenly they're slow.
- inopinatus 3y ago
  “If everyone is incompetent, all software will be bad”. Got it, thanks.
  - thfuran 3y ago
    You were the one advocating for incompetence.
    - inopinatus 3y ago
      That was not the incredible burn you hoped for, because it kinda suggests a lack of consideration for the practical design of a modern computer, in which memory is tiered in orders of magnitude of time latency from CPU registers on down through on-die caches and so forth, each tier effectively acting as a local cache for the next level. Writes are super expensive and bust cache lines and coherency all the way down to main memory and in many applications touch persistent storage and even introduce network latencies in distributed systems. By contrast, ALUs are stupendously well optimized for the baseline computational task of adding up integers. The difference between nanoseconds and milliseconds is vast. It is consequently very often cheaper to recalculate a sum using data already in cache memory than to save it and later fetch it, or (even worse) to read a value precalculated by a neighbouring computer. Furthermore, the distinction only matters for accounting systems at high scale. For the average small business accounting, it is de minimis: your computer performs more computation when resizing a browser window than it does in recalculating an account balance from scratch.