5 ms·

Building rqlite 9.0: Cutting disk usage by half

197 points by otoolep 2y ago

west0n 2y ago
I'm curious about how rqlite's performance compares to other distributed databases developed in Go, such as CockroachDB, Vitess, and TiDB.
- jitl 2y ago
  It’s going to have much lower write throughput, since SQLite is single-writer and on top of that you need to do Raft consensus. TiDB and CockroachDB can handle concurrent writes easily. Cockroach runs raft per “range” of 128mb of the key space, I’m not as familiar with TiDB. Vitess is an orchestration layer over MySQL, and MySQL handles concurrent writes easily.
  - otoolep 2y ago
    rqlite creator here. That's correct, there is a write-performance hit for the reasons you say. All Raft systems will take the same hit, and SQLite is intrinsically single-writer -- nothing about rqlite changes that[2]. That said, there are approaches to increasing write-performance substantially. See [1] for much more information. Write-performance is not the only thing to consider though (assuming one has sufficient performance in that dimension). Ease of deployment and operation are also important, and that's an area in which rqlite excels[3] (at least I think so, but I'm biased). [1] https://rqlite.io/docs/guides/performance/ https://rqlite.io/docs/guides/performance/ [2] https://rqlite.io/docs/faq/#rqlite-is-distributed-does-that-mean-it-can-increase-sqlite-performance https://rqlite.io/docs/faq/#rqlite-is-distributed-does-that-... [3] https://rqlite.io/docs/faq/#why-would-i-use-this-versus-some-other-distributed-database https://rqlite.io/docs/faq/#why-would-i-use-this-versus-some...
    - otoolep 2y ago
      Oh, I also presented some performance numbers in a presentation to a CMU a couple of years back. A little out-of-date, but gives a order-of-magnitude sense. https://youtu.be/JLlIAWjvHxM?t=2690 https://youtu.be/JLlIAWjvHxM?t=2690 The biggest performance improvement since is due to the introduction of Queued Writes. See https://rqlite.io/docs/api/queued-writes/ https://rqlite.io/docs/api/queued-writes/
    - spmurrayzzz 2y ago
      Relevant to the original inquiry — I really admire that you bring up the etcd and consul comparison right up front in the readme. For my own comprehension at least, it makes obvious the type of workloads for which you're optimizing and I appreciate that context as a past user of both of those stacks.
    - joostdecock 2y ago
      > Ease of deployment and operation are also important, and that's an area in which rqlite excels Amen. I've been building something appliance-like where I want to support clustering but I don't want to manage a database cluster inside the project. Rqlite is so easy to run either stand-alone or clustered. It's a godsend. And when people want postgres or whatever, I let them bring their own database. It's not hard to abstract a database storage layet if you plan ahead. But if you want it to 'just work' rqlite is doing that with flying colors.
- ClumsyPilot 2y ago
  Maybe ETCD is a more appropriate comparison?
  - protosam 2y ago
    Depending on what you’re using these tools for. If you want a locking manager and some meta data storage to help your distributed system maintain state, etcd is better for the job than rqlite for that. It’s a better zookeeper. With etcd you can hold a lock and defer unlocking if the connection is disrupted. Rqlite is not a good option for this.
    - otoolep 2y ago
      Agreed, in the sense that while rqlite has a lot in common with etcd (and Consul too -- Consul and rqlite share the same Raft implementation[1]) rqlite's primary use case is not about making it easy to build other distributed systems on top of it. [1] https://github.com/hashicorp/raft https://github.com/hashicorp/raft
      - protosam 2y ago
        Every time I've looked at rqlite, it just falls short features-wise in what I would want to do with it. A single raft group does not scale horizontally, so to me rqlite is a toy rather than a tool worth using (because someone might mistake the toy as production grade software).
        otoolep 2y ago
        rqlite creator here. That's clearly a mistaken attitude because both Consul and etcd also use a single "Raft group" and they are production-grade software. Ruling out a piece of software simply because it doesn't "scale horizontally" (and only writes don't scale horizontally in practice) is a naive attitude.
        protosam 2y ago
commercialnix 2y ago
Looking forward to a Rust implementation.
usr1106 2y ago
It's been many years that I haven't been working actively with databases anymore, but I have never heard about Raft or rq before. Is that system used by many / by significant players?
- hencoappel 2y ago
  I've not heard of rqlite, but raft is a popular consensus algorithm used by several quite a few notable systems including CockroachDB, MongoDB, RabbitMQ. https://en.m.wikipedia.org/wiki/Raft_(algorithm) https://en.m.wikipedia.org/wiki/Raft_(algorithm)
  - kitd 2y ago
    And now Kafka, without Zookeeper.
- otoolep 2y ago
  rqlite creator here. One notable production user is Replicated: https://www.philipotoole.com/replicated-postgres-to-rqlite/ https://www.philipotoole.com/replicated-postgres-to-rqlite/
djbusby 2y ago
Cool! We use this but never had too much problems with disk usage (I/o or size). But, were only using to deploy configs across multiple nodes. And we read the local copy directly!
- mst 2y ago
  Presumably you'll need to either use the correct WAL file or accept some very slight data staleness during the 9.0 sync process (also the direct read trick sounds great to me where permissible - EDIT: question about how much of your warranty does that void removed because the answer's already written up here - https://rqlite.io/docs/faq/#can-i-read-the-sqlite-file-directly https://rqlite.io/docs/faq/#can-i-read-the-sqlite-file-direc...). I would -imagine- that 'slight staleness' won't matter for your use case (I'm pretty confident that for the sort of configs I'm considering rqlite for that'll be the case, at least) but it's probably worth triple checking when configs are involved.
  - otoolep 2y ago
    Thanks for the question, but I don't follow it -- I don't see any data staleness if you query rqlite during snapshotting. Granted the blog post doesn't go into every single detail, so this might be hard to follow. Can you expand a bit more on your concern? What scenario do you have in mind?
    - mst 2y ago
      They're querying the sqlite database directly so I was thinking, possibly mistakenly, that when rqlite spawns a new WAL file to snapshot, the connection they have to said database outside of rqlite might not see the changes only in the snapshotting operation's WAL. I could easily be imagining a problem that won't exist.
      - otoolep 2y ago
        Ah yes. If you are accessing the SQLite files directly I cannot be sure what you will see during the snapshotting. I haven't tested that, since accessing the SQLite files underneath rqlite is not officially supported.
yNeolh 2y ago
A little off-topic, but I love that the first paragraph describes the project. Usually, posts exclude that information, and the landing is not of more help.
- deleted 2y ago
  [deleted]
ku1ik 2y ago
Version 9.0 already? To me this unfortunately signals lack of focus and/or disregard for backward compatibility, which means unnecessary churn for me as user of this project. Hard pass.
- otoolep 2y ago
  rqlite creator here. That's a mischaracterization. rqlite has been in development for 10 years[1], it's a long-running project and its design goals have never changed. The API hasn't changed in a breaking fashion since 2016 and rqlite has supported seamless upgrades for years now. In other words rqlite users have been upgrading from version to version for over 8 years, without having to change a single line of their code. [1] https://rqlite.io/docs/design/ https://rqlite.io/docs/design/
  - OskarS 2y ago
    I so much more appreciate this style of version numbering compared to being eternally 0.something.
  - ku1ik 2y ago
    Thanks, and sorry for mischaracterization. Glad to hear it’s different than I (wrongly) assumed. I’m curious though, if the design goals never changed and there was very little breaking changes, why the version is that high then? Not that you can’t use any numbering scheme you like, and not that semver is mandatory for every project. In hindsight I think my reaction came from assuming you use semver.
    - otoolep 2y ago
      I usually bump the major version number anytime I introduce important new functionality, major performance improvements, or a major new design change. While the API hasn't changed in years, the underlying implementation and file layout can change a lot between major versions. I want to communicate that. Also rqlite doesn't support seamless downgrades between major versions, only seamless upgrades. I want to communicate that too (I've put a lot of work into the backup-and-restore system[1] so users can protect themselves if they are concerned about the seamless upgrade failing on them). So by bumping the major version it helps people understand that they are upgrading to a substantially different version of rqlite, even if their client code doesn't have to change at all. [1] https://rqlite.io/docs/guides/backup/ https://rqlite.io/docs/guides/backup/
simplify 2y ago
Well-written article. Introduces what the library is/does, gives background context, includes a high-level overview of the system, and demonstrates how it solves the problem.
- otoolep 2y ago
  Just to be clear, rqlite is not a library. It's a complete RDBMS. rqlite has everything you need to read and write data, and backup, maintain, and monitor the database itself. It's not just a library (unlike, say, dqlite).
malaysanghi 2y ago
[flagged]
zxilly 2y ago
well, still didn't support embed as library.
niux 2y ago
Absolutely amazing news! Thank you Philip!
Lord_Zero 2y ago
How does this compare to litestream?
- scottyeager 2y ago
  Litestream is a solution for realtime replication of a SQLite database to S3 storage. The application uses SQLite as usual. It allows for recovery with less opportunity for data loss than with periodic backups. rqlite is a database server that uses SQLite as a backend and Raft consensus for clustering. It provides an API for clients to access over a network connection, rather than clients using SQLite directly. With a cluster you get data replication and high availability. There's also LiteFS now from the creator of Litestream. It's also a clustered approach but the app still uses SQLite directly. LiteFS depends on a Consul cluster. See also: https://litestream.io/alternatives/ https://litestream.io/alternatives/ And: https://rqlite.io/docs/faq/#how-is-it-different-than-litestream https://rqlite.io/docs/faq/#how-is-it-different-than-litestr...

197 points by otoolep 2y ago

I'm curious about how rqlite's performance compares to other distributed databases developed in Go, such as CockroachDB, Vitess, and TiDB.

jitl 2y ago
It’s going to have much lower write throughput, since SQLite is single-writer and on top of that you need to do Raft consensus. TiDB and CockroachDB can handle concurrent writes easily. Cockroach runs raft per “range” of 128mb of the key space, I’m not as familiar with TiDB. Vitess is an orchestration layer over MySQL, and MySQL handles concurrent writes easily.
- otoolep 2y ago
  rqlite creator here. That's correct, there is a write-performance hit for the reasons you say. All Raft systems will take the same hit, and SQLite is intrinsically single-writer -- nothing about rqlite changes that[2]. That said, there are approaches to increasing write-performance substantially. See [1] for much more information. Write-performance is not the only thing to consider though (assuming one has sufficient performance in that dimension). Ease of deployment and operation are also important, and that's an area in which rqlite excels[3] (at least I think so, but I'm biased). [1] https://rqlite.io/docs/guides/performance/ https://rqlite.io/docs/guides/performance/ [2] https://rqlite.io/docs/faq/#rqlite-is-distributed-does-that-mean-it-can-increase-sqlite-performance https://rqlite.io/docs/faq/#rqlite-is-distributed-does-that-... [3] https://rqlite.io/docs/faq/#why-would-i-use-this-versus-some-other-distributed-database https://rqlite.io/docs/faq/#why-would-i-use-this-versus-some...
  - otoolep 2y ago
    Oh, I also presented some performance numbers in a presentation to a CMU a couple of years back. A little out-of-date, but gives a order-of-magnitude sense. https://youtu.be/JLlIAWjvHxM?t=2690 https://youtu.be/JLlIAWjvHxM?t=2690 The biggest performance improvement since is due to the introduction of Queued Writes. See https://rqlite.io/docs/api/queued-writes/ https://rqlite.io/docs/api/queued-writes/
  - spmurrayzzz 2y ago
    Relevant to the original inquiry — I really admire that you bring up the etcd and consul comparison right up front in the readme. For my own comprehension at least, it makes obvious the type of workloads for which you're optimizing and I appreciate that context as a past user of both of those stacks.
  - joostdecock 2y ago
    > Ease of deployment and operation are also important, and that's an area in which rqlite excels Amen. I've been building something appliance-like where I want to support clustering but I don't want to manage a database cluster inside the project. Rqlite is so easy to run either stand-alone or clustered. It's a godsend. And when people want postgres or whatever, I let them bring their own database. It's not hard to abstract a database storage layet if you plan ahead. But if you want it to 'just work' rqlite is doing that with flying colors.
ClumsyPilot 2y ago
Maybe ETCD is a more appropriate comparison?
- protosam 2y ago
  Depending on what you’re using these tools for. If you want a locking manager and some meta data storage to help your distributed system maintain state, etcd is better for the job than rqlite for that. It’s a better zookeeper. With etcd you can hold a lock and defer unlocking if the connection is disrupted. Rqlite is not a good option for this.
  - otoolep 2y ago
    Agreed, in the sense that while rqlite has a lot in common with etcd (and Consul too -- Consul and rqlite share the same Raft implementation[1]) rqlite's primary use case is not about making it easy to build other distributed systems on top of it. [1] https://github.com/hashicorp/raft https://github.com/hashicorp/raft
    - protosam 2y ago
      Every time I've looked at rqlite, it just falls short features-wise in what I would want to do with it. A single raft group does not scale horizontally, so to me rqlite is a toy rather than a tool worth using (because someone might mistake the toy as production grade software).
      - otoolep 2y ago
        rqlite creator here. That's clearly a mistaken attitude because both Consul and etcd also use a single "Raft group" and they are production-grade software. Ruling out a piece of software simply because it doesn't "scale horizontally" (and only writes don't scale horizontally in practice) is a naive attitude.
        protosam 2y ago

commercialnix 2y ago

Looking forward to a Rust implementation.

usr1106 2y ago

It's been many years that I haven't been working actively with databases anymore, but I have never heard about Raft or rq before. Is that system used by many / by significant players?

hencoappel 2y ago
I've not heard of rqlite, but raft is a popular consensus algorithm used by several quite a few notable systems including CockroachDB, MongoDB, RabbitMQ. https://en.m.wikipedia.org/wiki/Raft_(algorithm) https://en.m.wikipedia.org/wiki/Raft_(algorithm)
- kitd 2y ago
  And now Kafka, without Zookeeper.
otoolep 2y ago
rqlite creator here. One notable production user is Replicated: https://www.philipotoole.com/replicated-postgres-to-rqlite/ https://www.philipotoole.com/replicated-postgres-to-rqlite/

djbusby 2y ago

Cool! We use this but never had too much problems with disk usage (I/o or size). But, were only using to deploy configs across multiple nodes. And we read the local copy directly!

mst 2y ago
Presumably you'll need to either use the correct WAL file or accept some very slight data staleness during the 9.0 sync process (also the direct read trick sounds great to me where permissible - EDIT: question about how much of your warranty does that void removed because the answer's already written up here - https://rqlite.io/docs/faq/#can-i-read-the-sqlite-file-directly https://rqlite.io/docs/faq/#can-i-read-the-sqlite-file-direc...). I would -imagine- that 'slight staleness' won't matter for your use case (I'm pretty confident that for the sort of configs I'm considering rqlite for that'll be the case, at least) but it's probably worth triple checking when configs are involved.
- otoolep 2y ago
  Thanks for the question, but I don't follow it -- I don't see any data staleness if you query rqlite during snapshotting. Granted the blog post doesn't go into every single detail, so this might be hard to follow. Can you expand a bit more on your concern? What scenario do you have in mind?
  - mst 2y ago
    They're querying the sqlite database directly so I was thinking, possibly mistakenly, that when rqlite spawns a new WAL file to snapshot, the connection they have to said database outside of rqlite might not see the changes only in the snapshotting operation's WAL. I could easily be imagining a problem that won't exist.
    - otoolep 2y ago
      Ah yes. If you are accessing the SQLite files directly I cannot be sure what you will see during the snapshotting. I haven't tested that, since accessing the SQLite files underneath rqlite is not officially supported.

yNeolh 2y ago

A little off-topic, but I love that the first paragraph describes the project. Usually, posts exclude that information, and the landing is not of more help.

deleted 2y ago
[deleted]

ku1ik 2y ago

Version 9.0 already? To me this unfortunately signals lack of focus and/or disregard for backward compatibility, which means unnecessary churn for me as user of this project. Hard pass.

otoolep 2y ago
rqlite creator here. That's a mischaracterization. rqlite has been in development for 10 years[1], it's a long-running project and its design goals have never changed. The API hasn't changed in a breaking fashion since 2016 and rqlite has supported seamless upgrades for years now. In other words rqlite users have been upgrading from version to version for over 8 years, without having to change a single line of their code. [1] https://rqlite.io/docs/design/ https://rqlite.io/docs/design/
- OskarS 2y ago
  I so much more appreciate this style of version numbering compared to being eternally 0.something.
- ku1ik 2y ago
  Thanks, and sorry for mischaracterization. Glad to hear it’s different than I (wrongly) assumed. I’m curious though, if the design goals never changed and there was very little breaking changes, why the version is that high then? Not that you can’t use any numbering scheme you like, and not that semver is mandatory for every project. In hindsight I think my reaction came from assuming you use semver.
  - otoolep 2y ago
    I usually bump the major version number anytime I introduce important new functionality, major performance improvements, or a major new design change. While the API hasn't changed in years, the underlying implementation and file layout can change a lot between major versions. I want to communicate that. Also rqlite doesn't support seamless downgrades between major versions, only seamless upgrades. I want to communicate that too (I've put a lot of work into the backup-and-restore system[1] so users can protect themselves if they are concerned about the seamless upgrade failing on them). So by bumping the major version it helps people understand that they are upgrading to a substantially different version of rqlite, even if their client code doesn't have to change at all. [1] https://rqlite.io/docs/guides/backup/ https://rqlite.io/docs/guides/backup/

simplify 2y ago

Well-written article. Introduces what the library is/does, gives background context, includes a high-level overview of the system, and demonstrates how it solves the problem.

otoolep 2y ago
Just to be clear, rqlite is not a library. It's a complete RDBMS. rqlite has everything you need to read and write data, and backup, maintain, and monitor the database itself. It's not just a library (unlike, say, dqlite).

malaysanghi 2y ago

[flagged]

zxilly 2y ago

well, still didn't support embed as library.

niux 2y ago

Absolutely amazing news! Thank you Philip!

Lord_Zero 2y ago

How does this compare to litestream?

scottyeager 2y ago
Litestream is a solution for realtime replication of a SQLite database to S3 storage. The application uses SQLite as usual. It allows for recovery with less opportunity for data loss than with periodic backups. rqlite is a database server that uses SQLite as a backend and Raft consensus for clustering. It provides an API for clients to access over a network connection, rather than clients using SQLite directly. With a cluster you get data replication and high availability. There's also LiteFS now from the creator of Litestream. It's also a clustered approach but the app still uses SQLite directly. LiteFS depends on a Consul cluster. See also: https://litestream.io/alternatives/ https://litestream.io/alternatives/ And: https://rqlite.io/docs/faq/#how-is-it-different-than-litestream https://rqlite.io/docs/faq/#how-is-it-different-than-litestr...