3 ms·
This may be hopelessly naive - but is the rebuilt sqlite db expected to be identical? Could a sha256 hash of the file be written into the raft log at intervals,
by e12e 6y ago
This may be hopelessly naive - but is the rebuilt sqlite db expected to be identical? Could a sha256 hash of the file be written into the raft log at intervals, and be used as a "valid" starting point?
Ie:
log1: empty db
log2..N: changes
log3: snapshot at N
log4..M: changes
You start with empty db, and replay up to N; or if you have a db that matches log3 - start with that and only replay log4+?
- otoolep 6y agoThat's not a bad idea, might have promise. Hashing large databases files after every few log entries might be a performance hit....not sure. Something to think about -- thanks.
- ignoramous 6y agoSnapshotting at intervals is how most point-in-time restores scale up. The db is snapshotted at a preset time (easy way: lock db for writes whilst snap is in progress, preferably on read replicas) and then restored again (preferably, elsewhere) to see if restore did indeed work as expected (and isn't corrupt). The latest log sequence number could also be stored in the same db (in a separate table) so that it is captured along with the db-snapshot, atomically.