3 ms·

Damn it, I was going to work on exactly this -- my goals were slightly different but basically the same paxos/raft/gossip + sqlite and see how far I could take

Damn it, I was going to work on exactly this -- my goals were slightly different but basically the same paxos/raft/gossip + sqlite and see how far I could take it. Note for those who are interested in cool-stuff-with-SQLite, there's a similar project called RQLite (mentioned in the README): https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite The main differences are laid out in the README, but IMO it really all boils down to the fact that since it's single-writer quorum'd there's only one node actually doing writes, and they've just chosen to replicate right-before (right after?) the WAL commit (I think this is what they're calling frames, basically one chunk of the WAL content), not at the point of receiving a query. This is more like having a read secondary more than anything (I also assume writes are redirected to master). It says in the readme that it should be expected to be slower than RQlite, I'd love to see some numbers. I do really like this though -- can't wait to pour through the code and see if there's anything I can learn.

bluejekyll 9y ago
Just because somethings already been done, doesn’t mean you can do it too, or even better.
- hardwaresofton 9y ago
  Yeah, but it's probably not smart to re-invent the wheel too many times. Especially when the projects that came before don't look badly written or unreasonable. I think I do have something to offer -- mostly trying gossip and making use of aggressive automatic sharding -- but I'm also more interested in what I wanted to build on top of distributed SQLite as well. No need to get down in the weeds if someone's already done the hard work for me :)
  - bluejekyll 9y ago
    All great points. I was just responding to what seemed like a defeated perspective. By all means contribute to this project. Everything starts somewhere!
    - free-ekanayaka 9y ago
      I implemented dqlite in Go for convenience (a production-ready raft library was available, github.com/hashicorp/raft). It'd be interesting to implement something similar in C or event batter in rust, so you can link the library from languages than just Go. But not sure there'd be good use cases for that.
      - hardwaresofton 9y ago
        Rust is exactly what I wanted to implement mine in -- I figure any performance savings can go towards making SQLite look more performant :) I've been looking at https://github.com/pingcap/raft-rs https://github.com/pingcap/raft-rs
      - twic 9y ago
        There's a Raft implementation in Rust to help get started: https://github.com/pingcap/raft-rs https://github.com/pingcap/raft-rs
xenadu02 9y ago
The WAL frames are by definition the result of the executed transactions so replicating the frames is a pretty smart way of ensuring you correctly distribute the results. You're correct: only the master accepts writes. During a failover clients can get an error and are expected to retry against the new master.
- hardwaresofton 9y ago
  Oh this is what I figured it was, it's the only thing left! I just have never come acros the teminology of a WAL "frame" before, so I wasn't completley sure. since it's a log I would have expected it to be called a record. Loud and clear on the failure modes, the README was pretty clear about it
free-ekanayaka 9y ago
Hello, author of dqlite here. Things are still in flux, but yes, I plan to publish benchmarks before making a 1.0 release, as well as improving documentations and introduce some more abstractions to make it easier to use. The reason it might be slower is some cases is that RQlite replicates statements and dqlite replicates WAL frames (where "frames" roughly means "disk pages"), which are typically bigger. I didn't run benchmarks against RQlite yet, but I'd expect performance differences to be negligible for most use cases.
- hardwaresofton 9y ago
  Hey thanks for putting this out there, it's a pretty cool project. I see what you meant by it being slower, but that's only replication speed right? maybe that should be pointed out in the documentation when you find time to update it. Does it seem like the replication patch is going to land in Sqlite? I sure would like it to...
  - free-ekanayaka 9y ago
    "slower" means this: when you perform a SQLite write, SQLite writes a new page on disk to the its write-ahead log. dqlite needs to replicate that page write across all nodes. A page is typically 4kb, so when you do a write on the leader node you need to transfer 4kb across the wire to all other follower nodes and wait for a quorum of them to come back and say "I got it" (which includes writing the page to disk). On the contrary rqlite just needs to transfer and store the SQL text, which is typically less than 4kb. In practice it's still pretty performant, but I'll publish benchmarks later on. I plan to submit the patch to SQLite upstream too, yes.
thinkersilver 9y ago
I'm in the same boat mate. Kudos for getting this far this quickly
otoolep 9y ago
rqlite is also a full application, whereas dqlite is more like the SQLite library -- you must wrap an application layer around it. In principle rqlite could use dqlite.

bluejekyll 9y ago

Just because somethings already been done, doesn’t mean you can do it too, or even better.

hardwaresofton 9y ago
Yeah, but it's probably not smart to re-invent the wheel too many times. Especially when the projects that came before don't look badly written or unreasonable. I think I do have something to offer -- mostly trying gossip and making use of aggressive automatic sharding -- but I'm also more interested in what I wanted to build on top of distributed SQLite as well. No need to get down in the weeds if someone's already done the hard work for me :)
- bluejekyll 9y ago
  All great points. I was just responding to what seemed like a defeated perspective. By all means contribute to this project. Everything starts somewhere!
  - free-ekanayaka 9y ago
    I implemented dqlite in Go for convenience (a production-ready raft library was available, github.com/hashicorp/raft). It'd be interesting to implement something similar in C or event batter in rust, so you can link the library from languages than just Go. But not sure there'd be good use cases for that.
    - hardwaresofton 9y ago
      Rust is exactly what I wanted to implement mine in -- I figure any performance savings can go towards making SQLite look more performant :) I've been looking at https://github.com/pingcap/raft-rs https://github.com/pingcap/raft-rs
    - twic 9y ago
      There's a Raft implementation in Rust to help get started: https://github.com/pingcap/raft-rs https://github.com/pingcap/raft-rs

xenadu02 9y ago

The WAL frames are by definition the result of the executed transactions so replicating the frames is a pretty smart way of ensuring you correctly distribute the results. You're correct: only the master accepts writes. During a failover clients can get an error and are expected to retry against the new master.

hardwaresofton 9y ago
Oh this is what I figured it was, it's the only thing left! I just have never come acros the teminology of a WAL "frame" before, so I wasn't completley sure. since it's a log I would have expected it to be called a record. Loud and clear on the failure modes, the README was pretty clear about it

free-ekanayaka 9y ago

Hello, author of dqlite here. Things are still in flux, but yes, I plan to publish benchmarks before making a 1.0 release, as well as improving documentations and introduce some more abstractions to make it easier to use. The reason it might be slower is some cases is that RQlite replicates statements and dqlite replicates WAL frames (where "frames" roughly means "disk pages"), which are typically bigger. I didn't run benchmarks against RQlite yet, but I'd expect performance differences to be negligible for most use cases.

hardwaresofton 9y ago
Hey thanks for putting this out there, it's a pretty cool project. I see what you meant by it being slower, but that's only replication speed right? maybe that should be pointed out in the documentation when you find time to update it. Does it seem like the replication patch is going to land in Sqlite? I sure would like it to...
- free-ekanayaka 9y ago
  "slower" means this: when you perform a SQLite write, SQLite writes a new page on disk to the its write-ahead log. dqlite needs to replicate that page write across all nodes. A page is typically 4kb, so when you do a write on the leader node you need to transfer 4kb across the wire to all other follower nodes and wait for a quorum of them to come back and say "I got it" (which includes writing the page to disk). On the contrary rqlite just needs to transfer and store the SQL text, which is typically less than 4kb. In practice it's still pretty performant, but I'll publish benchmarks later on. I plan to submit the patch to SQLite upstream too, yes.

thinkersilver 9y ago

I'm in the same boat mate. Kudos for getting this far this quickly

otoolep 9y ago

rqlite is also a full application, whereas dqlite is more like the SQLite library -- you must wrap an application layer around it. In principle rqlite could use dqlite.