4 ms·

litefs and litestream are interesting, but they all continue to not support confirmation that a transaction is durably replicated OR that failover won’t cause d

by dastbe 4y ago

litefs and litestream are interesting, but they all continue to not support confirmation that a transaction is durably replicated OR that failover won’t cause data loss. until that point it just seems like a sequence of experiments.

LunaSea 4y ago
I wonder if fly.io & co have built home grown solutions for this?
- plugin-baby 4y ago
  Fly.io bought Litestream
- hardwaresofton 4y ago
  well they're in the best/only(?) spot to do it -- owning the platform the end-programmer code is running on. SQLite is well built and extensible but hard to extend without the cooperation of the end-programmer, so to speak. SQLite is unfortunately (?) kind of hard to modify for external processes and while it's built very extensibly it's often not the thing you can kind of just... turn on, if that makes sense, and SQLite lives in the address space of the executing program. You end up with stuff like hacking LD_PRELOAD[0]. Note: Litestream (Ben) was acquihired essentially by Fly.io (so that should explain all their recent SQLite content!). [0]: https://github.com/cventers/sqlite3-preload https://github.com/cventers/sqlite3-preload
benbjohnson 4y ago
Litestream/LiteFS author here. I agree that synchronous replication is important and we have plans to implement it in LiteFS. Because LiteFS supports a loose membership model, quorum-based acknowledgement doesn't really work as well since the quorum can change. We have some other pieces to put into place before synchronous replication can work well. However, I disagree that it's just a "sequence of experiments". There are a lot of applications that can benefit from asynchronous replication. Synchronous acknowledgement of writes can impose a high cost on throughput so many folks use async replication, even in systems like Postgres.
- dastbe 4y ago
  synchronous replication to replicas is not necessary for data durability and doesn’t have to be a huge drag on throughput. for example, you can achieve high throughput by pipelining uncommitted transactions and then tracking when they are durably committed to backing store (like s3) for when to ack back to clients. and when dealing with failover, you can use the central store for determining place in ledger rather than whatever is on the replica that happens to get leadership.
- losfair 4y ago
  Maybe the inherent overhead of synchronous replication is more on latency rather than throughput.
- radicality 4y ago
  Perhaps semi-sync replication could be a good middle ground too. That way you can have 1 replica extremely close to the master (same datacenter / building), while other async replicas are potentially further away / have slower commit times.

by dastbe 4y ago

LunaSea 4y ago

I wonder if fly.io & co have built home grown solutions for this?

plugin-baby 4y ago
Fly.io bought Litestream
hardwaresofton 4y ago
well they're in the best/only(?) spot to do it -- owning the platform the end-programmer code is running on. SQLite is well built and extensible but hard to extend without the cooperation of the end-programmer, so to speak. SQLite is unfortunately (?) kind of hard to modify for external processes and while it's built very extensibly it's often not the thing you can kind of just... turn on, if that makes sense, and SQLite lives in the address space of the executing program. You end up with stuff like hacking LD_PRELOAD[0]. Note: Litestream (Ben) was acquihired essentially by Fly.io (so that should explain all their recent SQLite content!). [0]: https://github.com/cventers/sqlite3-preload https://github.com/cventers/sqlite3-preload

benbjohnson 4y ago

Litestream/LiteFS author here. I agree that synchronous replication is important and we have plans to implement it in LiteFS. Because LiteFS supports a loose membership model, quorum-based acknowledgement doesn't really work as well since the quorum can change. We have some other pieces to put into place before synchronous replication can work well. However, I disagree that it's just a "sequence of experiments". There are a lot of applications that can benefit from asynchronous replication. Synchronous acknowledgement of writes can impose a high cost on throughput so many folks use async replication, even in systems like Postgres.

dastbe 4y ago
synchronous replication to replicas is not necessary for data durability and doesn’t have to be a huge drag on throughput. for example, you can achieve high throughput by pipelining uncommitted transactions and then tracking when they are durably committed to backing store (like s3) for when to ack back to clients. and when dealing with failover, you can use the central store for determining place in ledger rather than whatever is on the replica that happens to get leadership.
losfair 4y ago
Maybe the inherent overhead of synchronous replication is more on latency rather than throughput.
radicality 4y ago
Perhaps semi-sync replication could be a good middle ground too. That way you can have 1 replica extremely close to the master (same datacenter / building), while other async replicas are potentially further away / have slower commit times.