6 ms·

I know it's only tangentially related, but is there any similar database that allows offline replicas or delayed synchronization? I.e. allow for copies to have

I know it's only tangentially related, but is there any similar database that allows offline replicas or delayed synchronization? I.e. allow for copies to have local offline writes and be brought up to synchronize? There is couchdb, but that seems to have fallen out of favor. I'm totally fine with having to explicitly handle collisions or prevent collisions client-side. Both dqlite and rqlite seem to use raft which requires an online quorum (I assume). [edit] I know CRDTs would be a potential solution, but I haven't seen any readymade, simple DB based on them.

samatman 5y ago
The SQLite built-in changeset and patchset extensions might be what you're looking for: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html
- uniqueuid 5y ago
  Thanks, that's extremely interesting. Sqlite continues to surprise me!
- zimpenfish 5y ago
  I wonder if this will make it to the CLI client once they've stabilised the API. (Most of the sqlite files I want to offline/delay sync are accessed 99% of the time by the CLI client, not via the library.)
ramses0 5y ago
https://pouchdb.com https://pouchdb.com
- uniqueuid 5y ago
  Right, but that only runs in browsers? Is there a standalone executable or can I embed it in a non-js-language?
rom99 5y ago
Care to elaborate on "couchdb has fallen out of favour"? Currently evaluating it for the very case you describe
- uniqueuid 5y ago
  Well, I've read about couchdb every few years, and people in the last ~5 years were complaining about an awkward query syntax, slow embedded JS engine (IIRC), client-side complexity of the document data model and effort to maintain. Similar things could probably be said about mongodb. If your data is small enough, it probably doesn't matter.
  - rom99 5y ago
    Thanks, we're not dealing with very large data sizes: maybe up to 100K documents, maybe using large binary attachments up to 100s MB per attachment. We use another DB product (ArangoDB) as our main application DB, so would be using something like CouchDB only for the offline-first and peer-to-peer replication capability - then referencing items in CouchDB from our main application DB, where we're more than happy about query language and other features. I had a somewhat wild idea that someone (maybe us...) could implement the couch DB replication protocol for ArangoDB using the ArangoDB FOXX framework. Not sure how feasible that might turn out to be though, wonder if anyone else thought of it...
- couchaway9000 5y ago
  I started using CouchDB around 2010. Overtime CouchDB has been removing more and more functionality and features that made CouchDB great IMHO (e.g couchapps), the 2.x (IIRC) made a lot of changes and totally broke a large application and made it too challenging to update so we stayed on a 1.x version (whatever version was last before the big couch merge.) we're moving off of Couch entirely in the future. I'd have a look at Couchbase if you're looking for replication features too.
- ies7 5y ago
  We have a pwa with pouchdb sync thousands handphones from couchdb every morning. Couchdb sync is great and pouchdb take care the db of whatever browser in your user phone. I could say we pick couchdb just because of their sync and pouchdb. For our main db we use postgres.
zrail 5y ago
Litestream might be what you're looking for? https://litestream.io/ https://litestream.io/
- zimpenfish 5y ago
  That's more of a backup/restore kind of thing - you can't restore to an existing file, for example, which means any local changes (even non-conflicting ones) would need to be blatted before getting any remote ones. Might suit some workflows, though?
joshbuddy 5y ago
I'm working on an offline/delayed syncing sqlite that deals with conflicts. It uses last-write-win for conflict resolution, the target audience is small sql databases you want replicated among a small group of people. If it sounds like something that might be useful to you, I'd love to talk about your needs.
- oldgregg 5y ago
  Looking for solution as well. I want a p2p solution where users can "subscribe" to a public SQLite database and received signed blocks/wal from the swarm. Edits would go into a distributed mempool to be signed by the key owner.
- uniqueuid 5y ago
  Yes that would pretty much fit my requirements. Feel free to email me at p <at> atrifle <dot> net.
XMPPwocky 5y ago
Depending on what you want to do with it, there's always https://github.com/cannadayr/git-sqlite https://github.com/cannadayr/git-sqlite
- kjeetgill 5y ago
  This is really cool, but I was a little disappointed. I clicked on it thinking it'd be the exact opposite: An implementation of git with all the version control stored in a sqlite DB. I've always wanted git-like branching, forking, commit, access to my datasets.
cryptonector 5y ago
Using CRDTs requires that you define all the relevant operations as monoids. A lot of work has to go into the choices of monoids to yield usable results. Monoids work fantastically well for things like upvotes on videos or whatever (so you'll notice that's always the first example given), but they don't really work for things like primary / unique keys, yet primary/unique keys are a critical requirement of most RDBMSes and schemas. Ergo, CRDTs don't work that well for relational databases. The situation is very different for collaborative text editing as there's no primary keys nor schema there, and humans are able to detect and correct nonsensical results. Mind you, I want CRDTs to work for RDBMSes -- the promise of CRDTs is fantastic. But do they? I suspect they do not.

samatman 5y ago

The SQLite built-in changeset and patchset extensions might be what you're looking for: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html

uniqueuid 5y ago
Thanks, that's extremely interesting. Sqlite continues to surprise me!
zimpenfish 5y ago
I wonder if this will make it to the CLI client once they've stabilised the API. (Most of the sqlite files I want to offline/delay sync are accessed 99% of the time by the CLI client, not via the library.)

ramses0 5y ago

https://pouchdb.com https://pouchdb.com

uniqueuid 5y ago
Right, but that only runs in browsers? Is there a standalone executable or can I embed it in a non-js-language?

rom99 5y ago

Care to elaborate on "couchdb has fallen out of favour"? Currently evaluating it for the very case you describe

uniqueuid 5y ago
Well, I've read about couchdb every few years, and people in the last ~5 years were complaining about an awkward query syntax, slow embedded JS engine (IIRC), client-side complexity of the document data model and effort to maintain. Similar things could probably be said about mongodb. If your data is small enough, it probably doesn't matter.
- rom99 5y ago
  Thanks, we're not dealing with very large data sizes: maybe up to 100K documents, maybe using large binary attachments up to 100s MB per attachment. We use another DB product (ArangoDB) as our main application DB, so would be using something like CouchDB only for the offline-first and peer-to-peer replication capability - then referencing items in CouchDB from our main application DB, where we're more than happy about query language and other features. I had a somewhat wild idea that someone (maybe us...) could implement the couch DB replication protocol for ArangoDB using the ArangoDB FOXX framework. Not sure how feasible that might turn out to be though, wonder if anyone else thought of it...
couchaway9000 5y ago
I started using CouchDB around 2010. Overtime CouchDB has been removing more and more functionality and features that made CouchDB great IMHO (e.g couchapps), the 2.x (IIRC) made a lot of changes and totally broke a large application and made it too challenging to update so we stayed on a 1.x version (whatever version was last before the big couch merge.) we're moving off of Couch entirely in the future. I'd have a look at Couchbase if you're looking for replication features too.
ies7 5y ago
We have a pwa with pouchdb sync thousands handphones from couchdb every morning. Couchdb sync is great and pouchdb take care the db of whatever browser in your user phone. I could say we pick couchdb just because of their sync and pouchdb. For our main db we use postgres.

zrail 5y ago

Litestream might be what you're looking for? https://litestream.io/ https://litestream.io/

zimpenfish 5y ago
That's more of a backup/restore kind of thing - you can't restore to an existing file, for example, which means any local changes (even non-conflicting ones) would need to be blatted before getting any remote ones. Might suit some workflows, though?

joshbuddy 5y ago

I'm working on an offline/delayed syncing sqlite that deals with conflicts. It uses last-write-win for conflict resolution, the target audience is small sql databases you want replicated among a small group of people. If it sounds like something that might be useful to you, I'd love to talk about your needs.

oldgregg 5y ago
Looking for solution as well. I want a p2p solution where users can "subscribe" to a public SQLite database and received signed blocks/wal from the swarm. Edits would go into a distributed mempool to be signed by the key owner.
uniqueuid 5y ago
Yes that would pretty much fit my requirements. Feel free to email me at p <at> atrifle <dot> net.

XMPPwocky 5y ago

Depending on what you want to do with it, there's always https://github.com/cannadayr/git-sqlite https://github.com/cannadayr/git-sqlite

kjeetgill 5y ago
This is really cool, but I was a little disappointed. I clicked on it thinking it'd be the exact opposite: An implementation of git with all the version control stored in a sqlite DB. I've always wanted git-like branching, forking, commit, access to my datasets.

cryptonector 5y ago

Using CRDTs requires that you define all the relevant operations as monoids. A lot of work has to go into the choices of monoids to yield usable results. Monoids work fantastically well for things like upvotes on videos or whatever (so you'll notice that's always the first example given), but they don't really work for things like primary / unique keys, yet primary/unique keys are a critical requirement of most RDBMSes and schemas. Ergo, CRDTs don't work that well for relational databases. The situation is very different for collaborative text editing as there's no primary keys nor schema there, and humans are able to detect and correct nonsensical results. Mind you, I want CRDTs to work for RDBMSes -- the promise of CRDTs is fantastic. But do they? I suspect they do not.