4 ms·
I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does
by trollbridge 2y ago
I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles:
- The order of log entries does not matter.
- Users of the log are peers. No client / server distinction.
- When appending a log entry, you can send a copy of the append to all your peers.
- You can ask your peers to refresh the latest log entries.
- When creating a new entry, it is a very good idea to have a nonce field. (I use nano IDs for this purpose along with a timestamp, which is probabilistically unique.)
- If you want to do database style queries of the data, load all the log entries into an in memory database and query away.
- You can append a log entry containing a summary of all log entries you have so far. For example: you’ve been given 10 new customer entries. You can create a log entry of “We have 10 customers as of this date.”
- When creating new entries, prepare the entry or list of entries in memory, allow the user to edit/revise them as a draft, then when they click “Save”, they are in the permanent record.
- To fix a mistake in an entry, create a new entry that “negates” that entry.
A lot of parallelism / concurrency problems just go away with this design.
- XorNot 2y agoHow do you know summary entries are valid if order doesn't matter? I.e. "we have 10 customers as of this date" can become immediately invalid if a new entry is appended afterwards with a date before that summary entry (i.e. because it was on a peer which hadn't yet sent it)
- clayg 2y agoIME you have to be willing to recalculate the summaries up to some kind of consistency window. Yes you may be changing history and you may have a business reason not to address that revision immediately (you've already billed them?) - but the system can still learn it made a mistake and fix it (add activity from Jan 30 evening that comes in late to the Feb bill?)
- withinboredom 2y agoRealistically, you never store summaries in the log. Instead, you store what it took to calculate them. So you won't store "we have 10 customers on this date with this range" but instead store "we found these 10 customers on this date with this range". This assumes you can store infinite sized lists in your log, but realistically, this is never a concern if you can keep your time windows small enough. Then, you periodically do a reconciliation and log corrections (look for entries not summarized -- easily done via a bloom filter which can tell you what entries are definitely NOT in your set) over a longer period. For example, we had a 28-day reconciliation period at one company I worked at (and handled over 120 million events per day). If you appended an event earlier than 28 days prior, it was simply ignored. This very rarely happened, but allowed us to fix bugs with events for up to 28 days.
- Kinrany 2y ago> The order of log entries does not matter. This is surprising, Kafka-like logs are all strictly ordered.
- cduzz 2y ago* within a partition
- trollbridge 2y agoThe reason I made ordering not matter is so that multiple peers don’t have to worry about keeping the exact same order when appending to each other’s logs. The log entries do have timestamps on them. You can sort by timestamp, but a peer has the right to append an new entry that’s older than the latest timestamp.
- grahamj 2y agoThe lack of ordering is surprising. Without that you can’t stream without a buffer.
- trollbridge 2y agoThe idea is that all peers eventually have the same collection of log entries, with an agreed-upon sort method. Currently that sort method is time stamps. When it’s time to have identical replicas, a log entry must be appended that says the log is closed. Once all peers have this, the log is now immutable and can be treated as such. An example of this (for a payroll system) is a collection of entries for time clock in and outs, payroll taxes withheld, direct deposits to employees, etc when it’s time to do a payroll. The log is closed and that data can never be altered, but now the data can be relied on by someone else. Conceptually this is surprisingly similar to the type of internal logs a typical database like PostgreSQL keeps.
- refset 2y ago> a log entry must be appended that says the log is closed Related concepts: 'epoch' (distributed consensus), 'watermark' (out-of-order stream processing)
- log4shell 2y agoCalling a WAL a ledger, why? Ledger sounds fancier but why would it be a ledger in this case?
- hcarvalhoalves 2y agoI believe "ledger" implies commutative property (order does not matter).
- log4shell 2y agoI am not aware of any such implicit connection of ledger and commutative property, also couldn't find anything as my google-fu is letting me down. Anything I can refer to? Generally curious to know use of term ledger outside of accounting and blockchains. I have seen it used to mean WAL before, so I am taking this with a dose of skepticism.
- hcarvalhoalves 2y agoIn a double-entry ledger the order of transactions doesn't matter, the balance is the sum of entries. Depending on the data model of your log, if calculating the current state is a commutative operation, I think it's fair to call it a "ledger".
- trollbridge 2y agoWe called it a ledger since we stored financial data and basically used the “ledger” format from plain text accounting, initially.
- log4shell 2y agoThat answers my question, thanks! Ordering does not matter part has me curious too, I will read other comment and come back to you l.
- glitchc 2y agoHow do you manage log size for high-transaction systems?
- trollbridge 2y agoIt’s not terribly high transaction, yet. If it becomes that way, I would partition the one by entry type so all the high transaction stuff gets stuffed in a particular log, and then ensure it can be summarised regularly.
- random3 2y agoif order does not matter how do you implement deletes?
- trollbridge 2y agoYou don’t do deletes. The log is append only. If you want to get rid of an entry, you create a new entry that negates the existing entry.