6 ms·

I've been following this, and am anxious for the direct-to-sqlite replication. One of rqlite's big limitations is that it resyncs the entire DB at startup time

by mrkurt 6y ago

I've been following this, and am anxious for the direct-to-sqlite replication. One of rqlite's big limitations is that it resyncs the entire DB at startup time. Being able to start with a "snapshot" and then incrementally replicate changes would be a big help.

benbjohnson 6y ago
That can be painful for sure. Litestream will do a snapshot on startup if it detects that it can't pick up from where it left off in the WAL. That can happen if Litestream is shut down and another process perform a checkpoint. But generally a restart will just use the existing snapshot & continue with the WAL replication.
CuriouslyC 6y ago
This is tangential, but depending on your sql needs, CouchDB's replication story is amazing, and you can replicate to the browser using PouchDB. There is an optional SQL layer, but obviously the good replication story comes with some trade-offs.
jlongster 6y ago
I built an app (https://actualbudget.com/ https://actualbudget.com/) that uses a local sqlite db and syncs changes, and that's exactly how it works. It takes quite a different approach though, using CRDTs to represent changes and those are synced around. When a fresh client comes into play, it downloads the latest sqlite snapshot from a server and then syncs up.
- pdimitar 6y ago
  Consider adding family sharing or just general making / joining of a group with shared expenses and accounts. Me and my wife use such an app currently but it lacks features and ergonomy.
otoolep 6y ago
rqlite creator here. I'm not sure I follow why it's a "big limitation"? Is it causing you long start-up times? I'm definitely interested in improving this, if it's an issue. What are you actually seeing? Also, rqlite does do log truncation (as per Raft spec), so after a certain amount of log entries (8192 by default) node restarts work exactly like you suggested. The SQLite database is restored from a snapshot, and any remaining Raft Log entries are applied to the database.
- mrkurt 6y ago
  Ah, ok that's some nuance I didn't know about! We're storing a few GB of data in the sqlite DB. Rebuilding those when rqlite restarts is slow and intensive process compared to just using the file on disk over again. Our particular use case means we'll end up restarting 100+ replica nodes all at once, so the way we're doing things makes it more painful than necessary.
  - otoolep 6y ago
    But how do you know it's intensive? Are you watching disk IO? Is there a noticeable delay when the node starts before it's ready to receive requests? Try setting "-raft-snap" to a lower number, maybe 1024, and see if it helps. You'll have much fewer log entries to apply on startup. However the node will perform a snapshot more often, and writes are blocked during the snapshotting. It's a trade-off. It might be possible to always restart using some sort of snapshot, independent of Raft, but that would add significant complexity to rqlite. The fact the SQLite database is built from scratch on startup, from the data in Raft log, means rqlite is much more robust.
    - mrkurt 6y ago
      Oh, we're reading the sqlite files directly. rqlite is really just a mechanism for us to propagate read only data to a bunch of clients. We need that sqlite file to never go away. Even a few seconds is bad. And since our replicas are spread all over the world, it's not feasible to move 1GB+ data from the "servers" fast enough. Is there a way for us to use that sqlite file without it ever going away? We've thought about hardlinking it elsewhere and replacing the hardlink when rqlite is up, but haven't built any tooling to do that.

by mrkurt 6y ago

benbjohnson 6y ago

That can be painful for sure. Litestream will do a snapshot on startup if it detects that it can't pick up from where it left off in the WAL. That can happen if Litestream is shut down and another process perform a checkpoint. But generally a restart will just use the existing snapshot & continue with the WAL replication.

CuriouslyC 6y ago

This is tangential, but depending on your sql needs, CouchDB's replication story is amazing, and you can replicate to the browser using PouchDB. There is an optional SQL layer, but obviously the good replication story comes with some trade-offs.

jlongster 6y ago

I built an app (https://actualbudget.com/ https://actualbudget.com/) that uses a local sqlite db and syncs changes, and that's exactly how it works. It takes quite a different approach though, using CRDTs to represent changes and those are synced around. When a fresh client comes into play, it downloads the latest sqlite snapshot from a server and then syncs up.

pdimitar 6y ago
Consider adding family sharing or just general making / joining of a group with shared expenses and accounts. Me and my wife use such an app currently but it lacks features and ergonomy.

otoolep 6y ago

rqlite creator here. I'm not sure I follow why it's a "big limitation"? Is it causing you long start-up times? I'm definitely interested in improving this, if it's an issue. What are you actually seeing? Also, rqlite does do log truncation (as per Raft spec), so after a certain amount of log entries (8192 by default) node restarts work exactly like you suggested. The SQLite database is restored from a snapshot, and any remaining Raft Log entries are applied to the database.

mrkurt 6y ago
Ah, ok that's some nuance I didn't know about! We're storing a few GB of data in the sqlite DB. Rebuilding those when rqlite restarts is slow and intensive process compared to just using the file on disk over again. Our particular use case means we'll end up restarting 100+ replica nodes all at once, so the way we're doing things makes it more painful than necessary.
- otoolep 6y ago
  But how do you know it's intensive? Are you watching disk IO? Is there a noticeable delay when the node starts before it's ready to receive requests? Try setting "-raft-snap" to a lower number, maybe 1024, and see if it helps. You'll have much fewer log entries to apply on startup. However the node will perform a snapshot more often, and writes are blocked during the snapshotting. It's a trade-off. It might be possible to always restart using some sort of snapshot, independent of Raft, but that would add significant complexity to rqlite. The fact the SQLite database is built from scratch on startup, from the data in Raft log, means rqlite is much more robust.
  - mrkurt 6y ago
    Oh, we're reading the sqlite files directly. rqlite is really just a mechanism for us to propagate read only data to a bunch of clients. We need that sqlite file to never go away. Even a few seconds is bad. And since our replicas are spread all over the world, it's not feasible to move 1GB+ data from the "servers" fast enough. Is there a way for us to use that sqlite file without it ever going away? We've thought about hardlinking it elsewhere and replacing the hardlink when rqlite is up, but haven't built any tooling to do that.