11 ms·
Distributed SQLite for Go applications
- notamy 9y agoAfter looking through the repo, I'm still not sure exactly why you would want / need this. What exactly is the use-case here?
- free-ekanayaka 9y agoIf your application is: 1) Distributed (e.g. n equal processes running on n machines) 2) Needs some shared state across nodes 3) Would like that state to have SQL semantics (relations, transactions, etc.) 4) Not to heavy on writes to this shared state (I might provide some ballpark numbers at some point) 5) Wants to avoid the operational overhead of an external storage system (mysql, postgresql) 6) Wants to be fault-tolerant and have transparent failovers then dqlite might be a good choice.
- notamy 9y agoInteresting. Thank you for the explanation!
- hamandcheese 9y agoI would imagine it would be used similarly to etcd, zookeeper, consul, etc. as a distributed configuration store, with the advantage of being SQL and the schema goodness that comes along with that.
- hardwaresofton 9y agoDamn it, I was going to work on exactly this -- my goals were slightly different but basically the same paxos/raft/gossip + sqlite and see how far I could take it. Note for those who are interested in cool-stuff-with-SQLite, there's a similar project called RQLite (mentioned in the README): https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite The main differences are laid out in the README, but IMO it really all boils down to the fact that since it's single-writer quorum'd there's only one node actually doing writes, and they've just chosen to replicate right-before (right after?) the WAL commit (I think this is what they're calling frames, basically one chunk of the WAL content), not at the point of receiving a query. This is more like having a read secondary more than anything (I also assume writes are redirected to master). It says in the readme that it should be expected to be slower than RQlite, I'd love to see some numbers. I do really like this though -- can't wait to pour through the code and see if there's anything I can learn.
- bluejekyll 9y agoJust because somethings already been done, doesn’t mean you can do it too, or even better.
- hardwaresofton 9y agoYeah, but it's probably not smart to re-invent the wheel too many times. Especially when the projects that came before don't look badly written or unreasonable. I think I do have something to offer -- mostly trying gossip and making use of aggressive automatic sharding -- but I'm also more interested in what I wanted to build on top of distributed SQLite as well. No need to get down in the weeds if someone's already done the hard work for me :)
- bluejekyll 9y agoAll great points. I was just responding to what seemed like a defeated perspective. By all means contribute to this project. Everything starts somewhere!
- free-ekanayaka 9y agoI implemented dqlite in Go for convenience (a production-ready raft library was available, github.com/hashicorp/raft). It'd be interesting to implement something similar in C or event batter in rust, so you can link the library from languages than just Go. But not sure there'd be good use cases for that.
- hardwaresofton 9y agoRust is exactly what I wanted to implement mine in -- I figure any performance savings can go towards making SQLite look more performant :) I've been looking at https://github.com/pingcap/raft-rs https://github.com/pingcap/raft-rs
- twic 9y agoThere's a Raft implementation in Rust to help get started: https://github.com/pingcap/raft-rs https://github.com/pingcap/raft-rs
- throwaway5752 9y agoHow does it compare to RocksDB? edit: I'm sorry, I was thinking of BedrockDB.
- gregwebs 9y agoRocksDB is a storage engine (a non-distriuted ordered Key-Value store with transactions). SQLite is a non-distributed SQL database that uses a storage engine (see SQLite4). The linked tool here makes SQLite distributed.
- free-ekanayaka 9y agoBedrockDB requires to operate a separate process, whereas you can embed dqlite in your Go application (pretty much in the SQLite philosophy). Also, afaik BedrockDB patches upstream SQLite with some more intrusive changes than dqlite (e.g. for supporting concurrent writes). The SQLite patch that dqlite requires is pretty minimal and just adds hooks to internal WAL events.
- anentropic 9y agoso the use case is: you're building a distributed Go application that needs some shared state in the form of an SQL db?
- free-ekanayaka 9y agoYes, dqlite is currently used for clustering support in LXD 3.0: https://github.com/lxc/lxd https://github.com/lxc/lxd
- PirateBay 9y agoProjects like this(sparsely documented, tiny) always seem to be just internal tools that the company just threw out into the world just because they can. Not that I'm ungrateful, but without any sort of explanation as to why I'd use this over a clustered MySQL or Postgres it's a little hard to make heads or tails about whether I should care.
- thotaway 9y agoYou would probably integrate this into a self contained application binary.
- strken 9y agoIt looks like a dependency of lxd, the successor to lxc, which is being developed by Canonical (of Ubuntu fame). You can see how they're using it in https://github.com/lxc/lxd/tree/master/lxd/cluster https://github.com/lxc/lxd/tree/master/lxd/cluster. Dqlite and rqlite are not in the same space as postgres and mysql, they're more comparable to tools like Zookeeper, etcd, or Consul[0] - low throughput CP data stores that are generally used for coordinating distributed systems. In this case, it saves users from having to manage an external data store to run lxd clustered. [0] in fact, dqlite uses the same raft implementation as Consul, which is cool
- contingencies 9y agoJust what the world needs, yet another would-you-like-my-version-of-clustering-with-that virtual environments tool...
- strken 9y agoHopefully someone more acquainted with the ecosystem can respond, but docker is built on lxc, and it appears that the intention of lxd is to do something similar - provide a low-level tool for docker and related tools to use, with some extra tools and security guarantees.
- free-ekanayaka 9y ago
- bbayer 9y agosqlite is fantastic piece of software. I am using it in every single side projects of mine. I can not describe how I satisfied with results even in relatively high load (300K pages/day on $5 Linode VPS). I found that most of time using a full featured RDBMS is not necessary depending on use case.(I am doing mostly reads)
- majewsky 9y ago> 300K pages/day Does that qualify as "high load"? That's just below 4 pages/second. Given how beefy server CPUs and SSDs are, I would consider something like 100+ requests/second "high load".
- bbayer 9y agoYou are right. That is why I said "relatively". It is relatively high for my setup.($5 linode with 2gb ram that runs 4 instance gunicorn server.)
- cbluth 9y agoTry accounting for peak load times.
- monsieurbanana 9y agoIf only it worked like that, we would avoid so many problems. Sadly you cannot extrapolate at all the load of a server with their numbers of pages seen per day, depending on the use you can easily have peaks of more than 10x that.
- riku_iki 9y ago> using a full featured RDBMS is not necessary But what kind of overhead postgress adds comparing to sqlite? Advantages are clear: - more advanced functionality is available when you need it. - pgsql likely has better performance comparing to sqlite
- atombender 9y agoIt's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library. What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where every node is guaranteed to be identical, but it won't scale writes and won't distribute storage across your cluster. You can build sharding on top of Raft, but in an SQL setting this would require distributed transactions, something that's difficult enough that relatively few projects have tackled it so far, the notable ones being CockroachDB and TiDB. (Google Spanner, being proprietary, is not relevant in this context.)
- eis 9y ago> It's fascinating how Raft has democratized distributed computing First I thought you probably ment commoditized but then I realized how democratized actually applies perfectly well to consensus algorithms because by definition they apply decisions via voting by all participants :) But anyways, I agree very much with you that Raft has transformed the industry and we can't be thankful enough for it. But like you said, it's just one of the core building blocks of a distributed system that also shards data. Sharding is more use-case specific and so it can be valid for different databases to have different sharding logics.
- lobster_johnson 9y agoIt's the second meaning of democratize, meaning "to make something accessible to everyone", that I intended, but I suppose Raft also has commoditized the technology, since there are now off-the-shelf libraries you can pick and choose from somewhat interchangeably. FWIW, the CockroachDB people developed a modified version they call MultiRaft [1] that deals with the scaling challenges they have around shards. [1] https://www.cockroachlabs.com/blog/scaling-raft/ https://www.cockroachlabs.com/blog/scaling-raft/
- free-ekanayaka 9y agoIf you require SQL, sharding and concurrent writes, yes do use CockroachDB or other equivalent solution. If you don't have those requirements, dqlite or rqlite might get the job done with less moving parts and operational overhead. It's pretty much the same argument of using SQLite vs (say) PostgreSQL, but translated into distributed systems. See: https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
- nudpiedo 9y agoI wonder how this kind of project can be tested in order to ensure the correctness of the use cases and error cases. does anyone know a resource to learn more about testing in this kind of situation? I hope there is an alternative to just try all cases in different real hardware setups (which is the mantra at my company)
- lifty 9y agoOne very good resource are the writings of Aphyr where he tests various distributed databases: https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen . Jepsen is the framework he wrote to test them. Next would be formal verification but you get very far by understanding the distributed db semantics and applying practical tests.
- free-ekanayaka 9y agoUnit and fuzzy-based test coverage that includes all possible edge cases is being put in place. I'd like to write a Jepsen test suite too. Hardware does not matter that much, since SQLite is pretty hardened for handling any possible hardware failure (out of memory, out of disk space, memory/disk corruption) and propagate that to client code (including dqlite). Regarding network failures (hardware-related or not), raft guards you against them, and it's formally proved.
- c17r 9y agoReminds me of the last Stripe CTF. I miss those.