3 ms·

> The caveat is that Litestream replication is asynchronous. A restore can miss the newest local writes if the SQLite volume disappears before they are copied.

by orf 4mo ago

> The caveat is that Litestream replication is asynchronous. A restore can miss the newest local writes if the SQLite volume disappears before they are copied. That is fine for many AI and experimentation workflows In short: SQLite is not all you need, unless you’re just experimenting don’t actually care about durability, in which case you also need litestream + object storage. Right.

0cf8612b2e1e 4mo ago
Postgres also does not synchronously replicate for free. You can setup both to get a confirmation write if you require that durability.
- orf 4mo ago
  > postgresql also does not synchronously replicate By default. Generally your primary database is in a completely different failure category than a kubernetes node running an ephemeral workflow pod.
  - 0cf8612b2e1e 4mo ago
    Either you have durable storage or you do not. SQLite and Postgres can both ensure local durability of commits. If you want distributed durability, you need to ship that data elsewhere. That is another Postgres node, object store, whatever that’s still an external dependency.
- paulddraper 4mo ago
  Not for free, but without the needing additional software. synchronous_commit = on
  - 0cf8612b2e1e 4mo ago
    That’s about the local transaction, not replication. SQLite WAL also gives you strict durability. PRAGMA synchronous = full
bootsmann 4mo ago
S3 is strongly consistent, if you need it anyways you can just use s3 keys to deconflict and store the workflow state.
- orf 4mo ago
  Yes, but directly using s3 as a key-value database is completely different from using SQLite + litestream.
gwking 4mo ago
The suitability of Litestream for production disaster recovery is also an open question in my mind. I used 0.3.x for several years and when I tried to upgrade to the 0.5.x series there were runaway disk usage problems that would have caused downtime had they made it to prod. As far as I can tell these have not been entirely addressed, although recent bug reports suggest that they might be getting closer. I want to love it, and I don't take open source projects like this for granted. But during my last production upgrade I chose to decommission Litestream in favor of a dumber, less granular solution using sqlite3_rsync and nightly backups because there is no point in using a backup system that is not rock solid.
paulddraper 4mo ago
"Durable workflows without the durability" That's distributed workflows :)
dilyevsky 4mo ago
i mean it's durable as long as nothing crashes or litestream has a data corruption bug which only happens every other release...
faangguyindia 4mo ago
try setting up replication/failover in postgres, it's much more work.

by orf 4mo ago

0cf8612b2e1e 4mo ago

Postgres also does not synchronously replicate for free. You can setup both to get a confirmation write if you require that durability.

orf 4mo ago
> postgresql also does not synchronously replicate By default. Generally your primary database is in a completely different failure category than a kubernetes node running an ephemeral workflow pod.
- 0cf8612b2e1e 4mo ago
  Either you have durable storage or you do not. SQLite and Postgres can both ensure local durability of commits. If you want distributed durability, you need to ship that data elsewhere. That is another Postgres node, object store, whatever that’s still an external dependency.
paulddraper 4mo ago
Not for free, but without the needing additional software. synchronous_commit = on
- 0cf8612b2e1e 4mo ago
  That’s about the local transaction, not replication. SQLite WAL also gives you strict durability. PRAGMA synchronous = full

bootsmann 4mo ago

S3 is strongly consistent, if you need it anyways you can just use s3 keys to deconflict and store the workflow state.

orf 4mo ago
Yes, but directly using s3 as a key-value database is completely different from using SQLite + litestream.

gwking 4mo ago

The suitability of Litestream for production disaster recovery is also an open question in my mind. I used 0.3.x for several years and when I tried to upgrade to the 0.5.x series there were runaway disk usage problems that would have caused downtime had they made it to prod. As far as I can tell these have not been entirely addressed, although recent bug reports suggest that they might be getting closer. I want to love it, and I don't take open source projects like this for granted. But during my last production upgrade I chose to decommission Litestream in favor of a dumber, less granular solution using sqlite3_rsync and nightly backups because there is no point in using a backup system that is not rock solid.

paulddraper 4mo ago

"Durable workflows without the durability" That's distributed workflows :)

dilyevsky 4mo ago

i mean it's durable as long as nothing crashes or litestream has a data corruption bug which only happens every other release...

faangguyindia 4mo ago

try setting up replication/failover in postgres, it's much more work.