6 ms·

Rqlite 6.0: the evolution of a distributed database design

180 points by otoolep 5y ago

sigmonsays 5y ago
isn't knowing every member of a cluster and the leader a core function of raft? I feel like this post is leaving out critical pieces of information, like why the URL can't be deterministic or data about comparisons of different approaches. At what cluster size and concurrency does asking every node break down? I have been meaning to take a closer look at rqlite and want to understand more about it.
- otoolep 5y ago
  rqlite author here. Yes, you're right every node knows the Raft network address of every other node. But Raft network addresses are not the network address used by clients to query the cluster. Instead every node also exposes a HTTP API for queries. So code needs exist to share information -- in this case the HTTP API addresses -- between nodes that the Raft layer doesn't handle.
  - otoolep 5y ago
    Also the HTTP API URL isn't deterministic because a) the operator sets it for any given node, and b) over the lifetime of the cluster the entire set of nodes could change as nodes fail, are replaced, etc.
    - otoolep 5y ago
      >At what cluster size and concurrency does asking every node break down? None, a follower only needs to ask the leader. So regardless of the size of the cluster, in 6.0 querying a follower only introduces a single hop to the leader before responding to the client. While this hop was not required in earlier versions, earlier versions had to maintain state -- and stateful systems are generally more prone to bugs.
      - vlowther 5y ago
        I am curious about where things broke down with the 301 based solution y'all used earlier.
        otoolep 5y ago
        I included details in the blog post, the 3.x to 5.x design had the following issues: - stateful system, with extra data stored in Raft. Always a chance for bugs with stateful systems. - some corner cases whereby the state rqlite was storing got out of sync with the some other cluster configuration. Finding the root cause of these bugs could have been very time-consuming. - certain failure cases happened during automatic cluster operations, meaning an operator mightn't notice and be able to deal with them. Now those failures cases -- while still very rare -- happen at query time. The operators know immediately sometime is up, and can deal with the problem there and then, usually by just re-issuing the query.
c17r 5y ago
As I was reading I was thinking "Why doesn't the Follower proxy the request to the Leader?" which I see was covered later on in "Transparent request forwarding". Good stuff! I remember building a janky version of sqlite+raft for Stripe's 2014 CTF. I'm sure others here have made a similar comment when rqlite gets posted to HN.
- otoolep 5y ago
  rqlite author here. Yes, it's coming in a future release and is much easier to do now. One key principle of rqlite has always been quality, clean design, and simplicity of operation. So I've been reluctant to add a feature -- in this case Request Forwarding -- until I was sure it would be a clear win and not make rqlite less robust. After years of experience with the system now, I'm happy it can be added in a high-quality manner.
  - robertlagrant 5y ago
    It definitely makes working with round robin proxies (e.g. k8s services) much, much simpler. Having proxies be aware of the leader, or having clients being able to access nodes directly instead of behind said proxies, is a lot more complexity.
- sroussey 5y ago
  It does add quite a bit to network traffic though, which may or may not be an issue.
convivialdingo 5y ago
Very interesting... Does it support full text search? I scanned a little bit but didn’t find any info either way.
- otoolep 5y ago
  Whatever SQLite exposes is available in rqlite.
rektide 5y ago
Not to be mistaken for high-availability Dqlite[1], which is one of the options one can run the k3s kubernetes distribution on (instead of etcd), via the Kine etcd shim[2]. Ultimately though the K3s team replaced Dqlite with an embedded etcd to get high-availability[3]. [1] https://dqlite.io/ https://dqlite.io/ [2] https://github.com/k3s-io/kine https://github.com/k3s-io/kine [3] https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/ https://rancher.com/docs/k3s/latest/en/installation/ha-embed...
- solarengineer 5y ago
  For anyone else facing a website crash error message: I had to turn off “Reader Mode” on Safari iOS to be able to view the DQLite website
deleted 5y ago
[deleted]
maxpert 5y ago
Thanks for the quality work here. Have been considering using this over infinicache or hazelcast for locally replicated caching scenario. Are there any battle testing stories/case-studies out there?
- otoolep 5y ago
  I haven't, I'm not super familiar with those systems TBH.
- geoka9 5y ago
  Have you considered using olric[0]? Just asking so that I can piggyback on your research :) [0] https://github.com/buraksezer/olric https://github.com/buraksezer/olric
  - maxpert 5y ago
    Olric did come up in my search but I never payed attention to it due to lack of production usage examples. I usually try to piggy back on existing well tested tools. The exception for rqlite because I understand both SQLite and Raft really well. So even if something goes wrong I will know what needs to be done.
Willson50 5y ago
https://web.archive.org/web/20210610124826/https://www.philipotoole.com/rqlite-6-0-0-building-for-the-future/ https://web.archive.org/web/20210610124826/https://www.phili...
killingtime74 5y ago
May I ask how users generally use Rqlite? Distributed db for web apps? Embedded? Thanks
- otoolep 5y ago
  I'm not sure how folks use it, but I think the sweet spot is for simple-to-run relational storage for a smallish set of data. Some people don't use it for the distribution, but just like a HTTP API in front of SQLite. https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-would-i-use-this-versus-some-other-replicated-database https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...
  - thomasmarcelis 5y ago
    Thats close to our use case. We use it to sync a small amount of configuration data across a small amount of servers (2-50, depending client). RQlite was perfect as our software is an addon for a legacy platform. We needed an easy low-access way of installing a distributed datastore.
    - otoolep 5y ago
      Cool. Did you use read-only nodes, by any chance? https://github.com/rqlite/rqlite/blob/master/DOC/READ_ONLY_NODES.md https://github.com/rqlite/rqlite/blob/master/DOC/READ_ONLY_N...
  - simonw 5y ago
    "I'm not sure how folks use it" Suggestion: have you considered running office hours and inviting your users to chat to you about what they're doing with it? I've been doing that for six months for my Datasette project and I've had over 60 conversations now, it's been a revelation - it almost completely solved the "I don't know how people are using this" problem for me, and gave me a ton of ideas for future directions for the project. I wrote more about that here: https://simonwillison.net/2021/Feb/19/office-hours/ https://simonwillison.net/2021/Feb/19/office-hours/
abotsis 5y ago
Someone needs to officiate a marriage between this and duckdb.
mirekrusin 5y ago
Very interesting, I wonder how it’d hold against Aphyr’s db claim invalidating machine.
- sesm 5y ago
  Related Github ticket: https://github.com/rqlite/rqlite/issues/94 https://github.com/rqlite/rqlite/issues/94 (labeled as 'help wanted')
- skyde 5y ago
  its a very simple request log in front of SQLite so unless there is a problem with the (Paxos/raft) algorithm used for replicating the logs it should hold very well.
  - skyde 5y ago
    actually it's using "github.com/hashicorp/raft" and not its own version. So since Hashicorp survived Aphyr test, it should be fine.
    - otoolep 5y ago
      Not quite as simple as that. I could have still used the Hashicorp code wrongly. And once I did: https://github.com/rqlite/rqlite/issues/5 https://github.com/rqlite/rqlite/issues/5 aphyr himself chimed in.
      - otoolep 5y ago
        Relevant doc: https://github.com/rqlite/rqlite/blob/master/DOC/CONSISTENCY.md https://github.com/rqlite/rqlite/blob/master/DOC/CONSISTENCY...
macawfish 5y ago
Would it be possible to run this in the browser? Via WASM or something?
- eudoxus 5y ago
  It wouldn't make too much sense to run a distributed raft-based SQLite system in its entirety in a browser (via WASM). However, you can run an individual SQLite instance in the browser (via WASM) using this: https://sql.js.org/#/ https://sql.js.org/#/
  - macawfish 5y ago
    Why wouldn't it make sense in your view? By the way, when I ask about "the browser" I'm also thinking of stuff like electron apps and progressive web apps.
    - kanwisher 5y ago
      Cause that's a single node, this is made for clusters. Maybe what you are looking for is a replication of another data? What use case would you have for multiple browsers syncing over a network?

180 points by otoolep 5y ago

isn't knowing every member of a cluster and the leader a core function of raft? I feel like this post is leaving out critical pieces of information, like why the URL can't be deterministic or data about comparisons of different approaches. At what cluster size and concurrency does asking every node break down? I have been meaning to take a closer look at rqlite and want to understand more about it.

otoolep 5y ago
rqlite author here. Yes, you're right every node knows the Raft network address of every other node. But Raft network addresses are not the network address used by clients to query the cluster. Instead every node also exposes a HTTP API for queries. So code needs exist to share information -- in this case the HTTP API addresses -- between nodes that the Raft layer doesn't handle.
- otoolep 5y ago
  Also the HTTP API URL isn't deterministic because a) the operator sets it for any given node, and b) over the lifetime of the cluster the entire set of nodes could change as nodes fail, are replaced, etc.
  - otoolep 5y ago
    >At what cluster size and concurrency does asking every node break down? None, a follower only needs to ask the leader. So regardless of the size of the cluster, in 6.0 querying a follower only introduces a single hop to the leader before responding to the client. While this hop was not required in earlier versions, earlier versions had to maintain state -- and stateful systems are generally more prone to bugs.
    - vlowther 5y ago
      I am curious about where things broke down with the 301 based solution y'all used earlier.
      - otoolep 5y ago
        I included details in the blog post, the 3.x to 5.x design had the following issues: - stateful system, with extra data stored in Raft. Always a chance for bugs with stateful systems. - some corner cases whereby the state rqlite was storing got out of sync with the some other cluster configuration. Finding the root cause of these bugs could have been very time-consuming. - certain failure cases happened during automatic cluster operations, meaning an operator mightn't notice and be able to deal with them. Now those failures cases -- while still very rare -- happen at query time. The operators know immediately sometime is up, and can deal with the problem there and then, usually by just re-issuing the query.

c17r 5y ago

As I was reading I was thinking "Why doesn't the Follower proxy the request to the Leader?" which I see was covered later on in "Transparent request forwarding". Good stuff! I remember building a janky version of sqlite+raft for Stripe's 2014 CTF. I'm sure others here have made a similar comment when rqlite gets posted to HN.

otoolep 5y ago
rqlite author here. Yes, it's coming in a future release and is much easier to do now. One key principle of rqlite has always been quality, clean design, and simplicity of operation. So I've been reluctant to add a feature -- in this case Request Forwarding -- until I was sure it would be a clear win and not make rqlite less robust. After years of experience with the system now, I'm happy it can be added in a high-quality manner.
- robertlagrant 5y ago
  It definitely makes working with round robin proxies (e.g. k8s services) much, much simpler. Having proxies be aware of the leader, or having clients being able to access nodes directly instead of behind said proxies, is a lot more complexity.
sroussey 5y ago
It does add quite a bit to network traffic though, which may or may not be an issue.

convivialdingo 5y ago

Very interesting... Does it support full text search? I scanned a little bit but didn’t find any info either way.

otoolep 5y ago
Whatever SQLite exposes is available in rqlite.

rektide 5y ago

Not to be mistaken for high-availability Dqlite[1], which is one of the options one can run the k3s kubernetes distribution on (instead of etcd), via the Kine etcd shim[2]. Ultimately though the K3s team replaced Dqlite with an embedded etcd to get high-availability[3]. [1] https://dqlite.io/ https://dqlite.io/ [2] https://github.com/k3s-io/kine https://github.com/k3s-io/kine [3] https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/ https://rancher.com/docs/k3s/latest/en/installation/ha-embed...

solarengineer 5y ago
For anyone else facing a website crash error message: I had to turn off “Reader Mode” on Safari iOS to be able to view the DQLite website

deleted 5y ago

[deleted]

maxpert 5y ago

Thanks for the quality work here. Have been considering using this over infinicache or hazelcast for locally replicated caching scenario. Are there any battle testing stories/case-studies out there?

otoolep 5y ago
I haven't, I'm not super familiar with those systems TBH.
geoka9 5y ago
Have you considered using olric[0]? Just asking so that I can piggyback on your research :) [0] https://github.com/buraksezer/olric https://github.com/buraksezer/olric
- maxpert 5y ago
  Olric did come up in my search but I never payed attention to it due to lack of production usage examples. I usually try to piggy back on existing well tested tools. The exception for rqlite because I understand both SQLite and Raft really well. So even if something goes wrong I will know what needs to be done.

Willson50 5y ago

https://web.archive.org/web/20210610124826/https://www.philipotoole.com/rqlite-6-0-0-building-for-the-future/ https://web.archive.org/web/20210610124826/https://www.phili...

killingtime74 5y ago

May I ask how users generally use Rqlite? Distributed db for web apps? Embedded? Thanks

otoolep 5y ago
I'm not sure how folks use it, but I think the sweet spot is for simple-to-run relational storage for a smallish set of data. Some people don't use it for the distribution, but just like a HTTP API in front of SQLite. https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-would-i-use-this-versus-some-other-replicated-database https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...
- thomasmarcelis 5y ago
  Thats close to our use case. We use it to sync a small amount of configuration data across a small amount of servers (2-50, depending client). RQlite was perfect as our software is an addon for a legacy platform. We needed an easy low-access way of installing a distributed datastore.
  - otoolep 5y ago
    Cool. Did you use read-only nodes, by any chance? https://github.com/rqlite/rqlite/blob/master/DOC/READ_ONLY_NODES.md https://github.com/rqlite/rqlite/blob/master/DOC/READ_ONLY_N...
- simonw 5y ago
  "I'm not sure how folks use it" Suggestion: have you considered running office hours and inviting your users to chat to you about what they're doing with it? I've been doing that for six months for my Datasette project and I've had over 60 conversations now, it's been a revelation - it almost completely solved the "I don't know how people are using this" problem for me, and gave me a ton of ideas for future directions for the project. I wrote more about that here: https://simonwillison.net/2021/Feb/19/office-hours/ https://simonwillison.net/2021/Feb/19/office-hours/

abotsis 5y ago

Someone needs to officiate a marriage between this and duckdb.

mirekrusin 5y ago

Very interesting, I wonder how it’d hold against Aphyr’s db claim invalidating machine.

sesm 5y ago
Related Github ticket: https://github.com/rqlite/rqlite/issues/94 https://github.com/rqlite/rqlite/issues/94 (labeled as 'help wanted')
skyde 5y ago
its a very simple request log in front of SQLite so unless there is a problem with the (Paxos/raft) algorithm used for replicating the logs it should hold very well.
- skyde 5y ago
  actually it's using "github.com/hashicorp/raft" and not its own version. So since Hashicorp survived Aphyr test, it should be fine.
  - otoolep 5y ago
    Not quite as simple as that. I could have still used the Hashicorp code wrongly. And once I did: https://github.com/rqlite/rqlite/issues/5 https://github.com/rqlite/rqlite/issues/5 aphyr himself chimed in.
    - otoolep 5y ago
      Relevant doc: https://github.com/rqlite/rqlite/blob/master/DOC/CONSISTENCY.md https://github.com/rqlite/rqlite/blob/master/DOC/CONSISTENCY...

macawfish 5y ago

Would it be possible to run this in the browser? Via WASM or something?

eudoxus 5y ago
It wouldn't make too much sense to run a distributed raft-based SQLite system in its entirety in a browser (via WASM). However, you can run an individual SQLite instance in the browser (via WASM) using this: https://sql.js.org/#/ https://sql.js.org/#/
- macawfish 5y ago
  Why wouldn't it make sense in your view? By the way, when I ask about "the browser" I'm also thinking of stuff like electron apps and progressive web apps.
  - kanwisher 5y ago
    Cause that's a single node, this is made for clusters. Maybe what you are looking for is a replication of another data? What use case would you have for multiple browsers syncing over a network?