4 ms·
This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi
by georgewfraser 7y ago
This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf
The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. The problem of implementing a distributed database is reduced to implementing a distributed log.
However, there is one oddity of the Waltz design: the central locking mechanism. Other deterministic databases don't have anything like this, because it's not necessary. You can keep track of locks locally on each replica, and you can rely on every replica to reach the same conclusions about which transactions succeeded and failed, because they are deterministic.
Can anyone clarify why they are managing centralized locks?
- dmix 7y agoDaniel Abadi also has an excellent paper on column databases: http://db.csail.mit.edu/pubs/abadi-column-stores.pdf http://db.csail.mit.edu/pubs/abadi-column-stores.pdf He's a great person at giving high level overviews of database systems before digging into the implementations.
- hvidgaard 7y agoI have not gone into this at all, but my initial thought is that as long as every log entry also have a unique id of the last log entry at the time of it's writing, you can always piece together a deterministic log without any locking or voting. Just share everything you know.
- majidazimi 7y agoEssentially if each instance can keep track of validating transactions locally, the whole thing can be easily implemented with Bookkeeper and distributed log API [0]. Bookeeper guarantees single writer (the master node which is appending to log) for each ledger. All it needs to be done is to put concurrency control metadata (like version, timestamp, ...) inside the message so each node can validate stuff locally. [0] https://bookkeeper.apache.org/docs/4.9.2/api/distributedlog-api/ https://bookkeeper.apache.org/docs/4.9.2/api/distributedlog-...
- vadikkiev 7y agoI ask for your help! Hello everyone, I'd like to get 2 karma point so that I would have 2 karma points in total, so that I can prove my HackerNews account into my Keybase account. Unfortunately I have no idea how can I gain karma, since I haven't used HackerNews that much. Can someone help me. Thanks.
- matchagreentea 7y agoThe blog post didn't fully specify, but the source code suggests that the locks are tracked per-partition rather than managed centrally.