17 ms·
Rqlite: The lightweight, distributed relational database built on SQLite
- szszrk 6y agorqlite is mentioned here quite often, multiple times last year. I don't think this entry brings anything new.
- blackbear_ 6y agoI know nothing of consensus algorithms and distributed systems so bear with me please. > rqlite uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all. What I understood from this sentence is that, if we have three instances, rqlite will make sure that every change is written to at least two. But what if two changes are written to two different pairs of instances? Then the three instances will have three different versions of the data. For example, change X is written to instances A and B, and change Y is written to B and C. Now A has X, B has X and Y, and C has Y only. How do you decide who is right?
- jasonjayr 6y agoIIRC, a monatomic counter is involved. The odd one out will realize it's behind the highest sequence number and discard it's updates to resync with the majority consensus. Edit: http://thesecretlivesofdata.com/raft/ http://thesecretlivesofdata.com/raft/ if you have some time seems to be a good step by step explanation on how it works in detail.
- adsharma 6y agoThe visual is indeed cool. I also thought it'd be nice to use a chat like interface to learn raft. Alice: /set a 1 Alice: /set b 2 Bob: /status Etc https://github.com/adsharma/zre_raft https://github.com/adsharma/zre_raft https://github.com/adsharma/raft https://github.com/adsharma/raft Bug reports welcome
- edoceo 6y agoRaft consensus: https://raft.github.io/ https://raft.github.io/ Surprisingly easy to understand, and a cool viaual.
- teraflop 6y agoIn brief: at any point in time, one of the replicas is a "leader" which controls the order in which operations are committed. The changes occur in a defined sequence, and other replicas may lag behind the leader, but cannot be inconsistent with it. Your example can't happen, because if (for instance) A is the leader, then C will not apply change Y without contacting the leader, which will tell it to apply X first. If you want more details about how this handles all the edge cases -- for instance, what happens if the leader crashes -- the Raft paper is quite accessible: https://raft.github.io/raft.pdf https://raft.github.io/raft.pdf
- hinkley 6y agoTL;DR: Raft updates are serialized (as in sequential).
- whizzter 6y agoThe semantics of Raft has a "simple" (compared to the harder to understand Paxos) forward motion of events that is supposed to guarantee that you won't get into weird states regardless of if any particular node(s) goes down (I think it can surive (N/2)-1 dead machines in a cluster of N). Raft is based on having a leader decide what the next COMMIT is going to be, so B could never have X and Y at the same time (they could both be queued but other mechanisms could reject them). Also data is not considered committed until more than half the cluster has acknowledged it (at which point the leader will know it and handle going forward), leader election also works in a similar way iirc. As others mentioned, the visualization on https://raft.github.io/ https://raft.github.io/ is really good (You can affect it to create commits and control downtime of machines)
- hinkley 6y agoIt’s 1/2 + 1 isn’t it? So if the leader goes down at the exact moment of quorum, you you can still get quorum again. That would mean in 3 servers you need 2.5 aka 3 machines to commit a change. Then 4/5, 5/7, 6/9, 7/11. And I think it’s a wash anyway, because as the servers go up the fraction you need for quorum goes down, but the odds of falling behind or failing outright go up too. Not to mention the time during which 1/n machines are down due to an upgrade gets longer and longer the more machines you have, increasing the chances of double fault.
- simtel20 6y ago> It’s 1/2 + 1 isn’t it? The parent post is talking about the number that can go down while maintaining quorum, and you're talking about the number that need to remain up to maintain quorum. So you're both correct. However: > That would mean in 3 servers you need 2.5 aka 3 machines to commit a change. That seems wrong. You need N//2 +1 where "//" is floor division, so in a 3 node cluster, you need 3//2 +1, or 1+1 or 2 nodes to commit a change.
- hinkley 6y agoI think I see the problem. 'Simple majority' is based on the number of the machines that the leader knows about. You can only change the membership by issuing a write. Write quorum and leadership quorum are two different things, and if I've got it right, they can diverge after a partition. I'm also thinking of double faults, because the point of Raft is to get past single fault tolerance. [edit: shortened] After a permanent fault (broken hardware) in a cluster of 5, the replacement quorum member can't vote for writes until it has caught up. It can vote for leaders, but it can't nominate itself. Catching up leaves a window for additional faults. It's always 3/5 for writes and elections, the difference is that the ratio of original machines that have to confirm a write can go to 100% of survivors, instead of the 3/4 of reachable machines. Meaning network jitter and packet loss, slows down writes until it recovers, and an additional partition can block writes altogether, even with 3/5 surviving the partition.
- ClumsyPilot 6y agoI think microk8s uses this to form a cluster, and k3s used to use it but moved back to etc. Would be good to hear from someone who used it what are the pros and cons of such a setup
- fasteo 6y agoAFAIK, microk8s uses a similar - but not this - form of distributed sqlite. Specifically, it uses dqlite[1] "a C library that implements an embeddable and replicated SQL database engine with high-availability and automatic failover." [1] https://github.com/canonical/dqlite https://github.com/canonical/dqlite
- tyingq 6y agoProbably worth mentioning that Canonical initially made dqlite to be the backing store for LXD. It uses the sqlite VFS as the client entry point, so it's a very easy transition for an existing sqlite app, just recompile with the new header.
- hermitcrab 6y agoSo if: Alice, Bob and Charlie have a synced copy of the same database Charlie goes on a plane and adds a loads of records without a connection to the other databases Alice and Bob make no changes Charlie comes home and syncs Will Charlie lose all his changes, as his database is different to Alice and Bob's? What happens if Alice, Bob and Charlie all makes changes offline then resync?
- mrkurt 6y agoIt doesn't work that way. rqlite is effectively a leader/follower model that uses raft for leader consensus. Writes can only happen online, and only to the leader.
- hermitcrab 6y agoOk, thanks!
- teraflop 6y agoAs per the description, all updates must be replicated to a quorum of instances. If Charlie is on a plane without the ability to contact a quorum, he can't add records in the first place. This is a fundamentally necessary tradeoff to provide strong consistency, as described by the CAP theorem.
- unnouinceput 6y agoIn which case Charlie will have an additional local DB to record those records and when he gets back will use another protocol/method/system/whatever to add those new records? How about if everybody goes and adds records on same table? Here is a real life scenario that I had to deal with in the past. Technician (carpenters) goes to clients home to repair furniture in the middle of nowhere, so no internet. Adds the necessary paperwork which is pictures, declarations, contract (signed and scanned) to the Clients table. This company was employing hundreds of such technicians throughout the many counties of Germany, each with a laptop on them running this app which was the backbone for getting paid/do the work. And was not uncommon to have more than one carpenter go to client home and do the repairs. Since each carpenter was paid according to its own work, each of them would create entries in their local Clients table and when getting back to HQ their work was manually uploaded to central DB, and only after that they got paid. I automated that (that was the job, to eliminate the thousands of hours that carpenters were wasting manually). So given the above scenario, how is this system going to achieve that? Same table, and same client details even in table Clients, just different rows for different carpenters (foreign key to table Carpenters).
- foolinaround 6y agoWe currently use browsers on several devices (both laptops and android) and rely on google sync currently. Maybe this could be used to sync bookmarks, history etc across my devices but still keep my data local to me?
- JoachimSchipper 6y agoThis uses Raft, so a quorum of devices would need to be online at the same time. That's not what you want for browser sync.
- ericlewis 6y agoExpensify had a version of something like this back in like 2013/14 I think.
- moderation 6y agoThe project is BedrockDB [0] and has been previously discussed [1]. 0. https://bedrockdb.com/ https://bedrockdb.com/ 1. https://news.ycombinator.com/item?id=12739771 https://news.ycombinator.com/item?id=12739771
- ericlewis 6y agoNice! Though the blockchain part is new to me. Interesting they kept growing this.
- fnord123 6y agoFoundationDB and Comdb2 also use sqlite as a storage engine. Curious that they decided to implement yet another one. https://www.foundationdb.org/ https://www.foundationdb.org/ http://comdb2.org/ http://comdb2.org/
- jchrisa 6y agoI'm curious how this relates to the Calvin protocol as implemented by FaunaDB. They both use Raft, but FaunaDB and Calvin have additional details about how transactions are retried and aborted. https://fauna.com/blog/consistency-without-clocks-faunadb-transaction-protocol https://fauna.com/blog/consistency-without-clocks-faunadb-tr...
- scottlamb 6y agoIt looks like this layers Raft on top of SQLite. I don't like when systems replicate high-level changes like "update users set salary = salary + 1000 where ...;" Instead, I prefer they replicate low-level changes like "replace key/block X, which should have contents C_x, with C_x'". Why? Imagine you're doing a rolling update. Some of your replica are running the newer version of SQLite and some are running the older version. They may not execute the high-level query in exactly the same way. For example, in the absence of an "order by" clause, select results' order is unstated. So imagine someone makes a mutation that depends on this: "insert ... select ... limit". (Maybe a dumb example but it can happen anyway.) Now the databases start to diverge, not only in underlying bytes and implementation-defined ordering but in actual row data. I worked on a major distributed system that originally replicated high-level changes and switched to replicating low-level changes for this reason. We had a system for detecting when replicas didn't match, and replication of high-level changes was the biggest reason for diffs. (Hardware error was the second biggest reason; we added a lot of checksumming because of that.)
- ttul 6y agoSounds like you need to submit a patch!
- scottlamb 6y agolol, this is an unhelpful reflex answer to any criticism of open source software. What I'm describing is a redesign. There's no point in submitting a patch for that. It'd be much more effort than a few lines of code, and it wouldn't be accepted. Open source means that anyone can fork. It doesn't mean that maintainers will automatically merge patches replacing their software with completely different software. The only way that will happen is if the maintainers decide for themselves it needs to be redesigned, and that starts with discussion rather than a patch. It's also a long shot compared to just finding different software that already has the design I prefer. If I want replicated SQLite, I'll look at dqlite or litestream instead, which sound more compatible with my design sensibilities. (Thanks, tyingq and benbjohnson!)
- 6y ago
- mshenfield 6y agoGoing from a local db to one over a network has at least one risk. The SQLite docs gives developers the okay to write "n+1" style queries (https://www.sqlite.org/np1queryprob.html https://www.sqlite.org/np1queryprob.html). When the db is on the same file system as the application this pattern is fine. But as soon a you add a network call it becomes a potential bottleneck.
- mech422 6y agoSo it looks like you can now distribute SQLite at the: Stmt level: https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite VFS Level: https://github.com/canonical/dqlite https://github.com/canonical/dqlite Block Level: https://github.com/benbjohnson/litestream https://github.com/benbjohnson/litestream Really cool enhancements to an awesome project!
- Conlectus 6y agoOne thing that jumps out at me after reading a lot of Jepsen analyses - does Rqlite assume that partitions form equality relations? That is, that all nodes belong to one and only partition group? This is not always the case in practice.
- yjftsjthsd-h 6y agoSo the case of A can talk to B, B can talk to C, but A can't talk to C? (Making sure that I understand how you can be in multiple partitions)
- Conlectus 6y agoThat's right, yes. This is apparently possible (if not common) given the networking setups of a lot of data centers.
- peter_d_sherman 6y agoFirst of all, great idea, and a brilliant and highly laudable effort! Favorited! One minor caveat ("Here be Dragons") I have (with respect to my own future adoption/production use), however: https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md >"Does rqlite support transactions? It supports a form of transactions. You can wrap a bulk update in a transaction such that all the statements in the bulk request will succeed, or none of them will. However the behaviour or rqlite is undefined if you send explicit BEGIN, COMMIT, or ROLLBACK statements. This is not because they won't work -- they will -- but if your node (or cluster) fails while a transaction is in progress, the system may be left in a hard-to-use state. So until rqlite can offer strict guarantees about its behaviour if it fails during a transaction, using BEGIN, COMMIT, and ROLLBACK is officially unsupported. Unfortunately this does mean that rqlite may not be suitable for some applications." PDS: Distributed transactions are extremely difficult to get exactly right -- so I'm not trying to criticize all of the hard work and effort that everyone has put into this (again, it's a great idea, and I think it has a terrific future). But Distributed Transactions -- are what differentiate something like rsqlite from say, something like CockroachDB (https://www.cockroachlabs.com/docs/stable/architecture/life-of-a-distributed-transaction.html https://www.cockroachlabs.com/docs/stable/architecture/life-...). Of course, CockroachDB is a pay-for product with an actual company with many years of experience backing it, whereas rqlite, as far as I can intuit, at this point in time (someone correct me if I am wrong), appears to be a volunteer effort... Still, I think that rqlite despite this -- has a glorious and wonderful future! Again, a brilliant and laudable effort, suitable for many use cases presently, and I can't wait to see what the future holds for this Open Source project! Maybe in the future some code-ninja will step up to the plate and add fully guaranteed, safe, distributed transactions! Until then, it looks like a great idea coupled with a great software engineering effort! As I said, "Favorited!".
- f430 6y agoCould you use this to build a decentralized p2p app? If so, what gotchas and limitations are there?
- otoolep 6y agoNo, rqlite is not suitable for that kind of application. All writes must go through the leader.
- alberth 6y ago@otoolep SQLite has a great post on “When to Use” (and not use) SQLite. Would be great if you included these same use cases in the ReamMe docs and make it clear if Rqlite can address them. https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html