3 ms·
> I'm adding S3 replication support directly into LiteFS. Nice! There's a lot of value one can get out of a blob store, despite it appearing seemingly at odds
by ignoramous 4y ago
> I'm adding S3 replication support directly into LiteFS.
Nice! There's a lot of value one can get out of a blob store, despite it appearing seemingly at odds with block-device dependent systems, like most sql dbms.
When a database at BigCloud layered replication (real-time backups) atop S3, they did so by shipping both the WAL and the on-disk files. For write heavy tables, WAL was streamed every second, and on-disk files (snapshots) every 30mins (or at some apt size-based threshold).
While WAL streaming also doubled-up as a key foundation for them to build materialized views, support real-time triggers, and act as an online data-verification layer; S3 itself served as an insurance against hardware errors (memory, cpu, network, disk) and data corruption.
https://web.archive.org/web/20220712155558/https://www.usenix.org/system/files/atc22-vig.pdf https://web.archive.org/web/20220712155558/https://www.useni... (keyword search S3)
Elasticsearch / OpenSearch does something similar but it only implements snapshot-based replication to S3 (periodic backups).
https://web.archive.org/web/20190722153122/https://www.microsoft.com/en-us/research/wp-content/uploads/2008/02/tr-2008-25.pdf https://web.archive.org/web/20190722153122/https://www.micro... / https://archive.is/Q5jUj https://archive.is/Q5jUj (docs)