5 ms·

LiteFS/Litestream author here. You bring up a lot of great points that I'll try to address. > What I'd like to have seen is how this compares to things like rq

LiteFS/Litestream author here. You bring up a lot of great points that I'll try to address. > What I'd like to have seen is how this compares to things like rqlite or Cloudflare's D1 addressed directly in the article I think a post comparing the different options is a great idea. I'll try to summarize a bit here though. LiteFS aims to be an analogue to Postgres replication but with built-in failover. Postgres uses log shipping to copy database state from a primary to its replicas, as does LiteFS. LiteReplica is probably the closest thing to LiteFS although it uses a dual GPL/commercial license model. LiteFS uses Apache 2. There are Raft-based tools like rqlite or dqlite. These have higher consistency guarantees, however, they tend to be more complex to set up -- especially in an ephemeral environment like Kubernetes. LiteFS has a more relaxed membership model for this reason. As for Cloudflare's D1, they haven't released many details and I assume it's closed source. It also requires using a custom JavaScript client instead of using a native SQLite client. There are also several eventually consistent stores built on SQLite such as Mycelial[1]. These work great for applications with loose consistency needs. LiteFS still maintains serializable isolation within a transaction, although, it has looser guarantees across nodes than something like rqlite. > Most workloads are already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common) Yes, that's absolutely true. I don't expect anyone to port their application from Postgres/MySQL to SQLite so they can use LiteFS. Databases and database tools have a long road to becoming mainstream and it follows the traditional adoption curve. I wrote a Go database library called BoltDB about 10 years ago and it had the same adoption concerns early on. Typically, folks try it out with toy applications and play around with it. Once they get more comfortable, then they create new applications on top of it. As more people use it, then late adopters get more comfortable and it further builds trust. We're committed to LiteFS for the long term so we'll be making updates, fixing bugs, and we'll keep trying to build trust with the community. [1]: https://mycelial.com/ https://mycelial.com/

benbjohnson 4y ago
> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream for you. Oh, I forgot to touch on this point. LiteFS uses some concepts to Litestream (e.g. log shipping), however, it doesn't use Litestream internally. It has much stricter requirements in terms of ensuring consistency since it's distributed so it performs an incremental checksum of the database on every transaction. It has additional benefits with its internal storage format called LTX. These storage files can be compacted together which will allow point-in-time restores that are nearly instant.
no_wizard 4y ago
Note: I apologize if this is overstepping, its hard to tell! I think a strong - extremely strong - selling point is the point I made about "prebaked" data for your APIs, since the entire strength of these SQLite based systems reside in their fast read capacity (as mentioned elsewhere and in this article, its very fast for read heavy applications, which is most) you could take on an angle around that to get people "in the door" by showing a pathway of how this fits inside your existing data warehouse / storage model. We found we liked the SQLite durability to do this. It was a bit smarter than just a plain cache (such as Redis) with better durability and (for our needs) comparable enough performance (I think in absolute terms, a tuned Redis instance will always be faster, but up to a certain point, speed isn't everything, especially when factoring cost). We found it was cheaper - by a good margin - to do this over caching everything AOT in a redis cluster, and we could therefore much more cheaply go multi-region and have DB's sitting next to our customers that acted as a nearline cache. The complexity - which is an area where this might help in the future, and why I'm mentioning it - is shipping changes back. What we ended up doing is setting up a write Redis cluster that clients write to, and we take those writes and trigger a propogation job back to the database. This allowed us to run a much slimmer redis cluster and made us feel more comfortable doing cache eviction since we could verify writes pretty easily. You could do this with memcache or whatever too. Sounds convoluted, but it worked really well. It allowed us to keep our centralized database intact without having to spin up expensive instances to be multi-region or commit to ever growing Redis cluster(s). the SQLite flat file model + history of durability made things the perfect tradeoff for this use case. Of course, YMMV, however it was a novel solution that used "enterprise grade" parts all the way down, which made it an easy selling point. You might find it worth exploring this more. As far as the comparisons go, I think it'd be cool to see a deep dive, and run a test suite against each of the major SQLite as a distributed database model. For that, I don't think it has to be open source to do a reasonable comparison?
- benbjohnson 4y ago
  > I think a strong - extremely strong - selling point is the point I made about "prebaked" data for your APIs. I think caches are an excellent use case for LiteFS early on. Sorry I didn't make that point in my previous reply. It's a good way to get benefits out of LiteFS without committing to it as your source of truth. Also related, Segment built a custom SQLite-based solution[1] for distributing out cached data that worked well for them. [1]: https://segment.com/blog/separating-our-data-and-control-planes-with-ctlstore/ https://segment.com/blog/separating-our-data-and-control-pla... > We found it was cheaper - by a good margin - to do this over caching everything AOT in a redis cluster Do you remember specifics of the cost difference? I can imagine it'd be pretty significant since you don't need to spin up servers with a bunch of RAM. > Note: I apologize if this is overstepping, its hard to tell! Not overstepping at all! It's great to hear folks' feedback.
ignoramous 4y ago
Thanks. > LiteFS still maintains serializable isolation within a transaction, although, it has looser guarantees across nodes than something like rqlite. Picking up a term from the consistency map here [0], what guarantees LiteFS makes across nodes? [0] https://jepsen.io/consistency https://jepsen.io/consistency
- benbjohnson 4y ago
  Good question. Right now, LiteFS operates with async replication so its possible to have a transaction written to the primary get lost if the primary fails before it's replicated out. For that short window, you could read a transaction that no longer exists. From that standpoint, I believe it would technically be Read Uncommitted. However, during normal operation it'll function more like Snapshot Isolation. LiteFS does provide a transaction ID so requests could wait for a replica to catch up before issuing a transaction to get something closer to Serializable.
  - Spivak 4y ago
    I think you’re being too harsh on yourself, isolation levels don’t typically account for replication lag and network partitioning into account. For example on MySQL you can have async replication and serializable isolation. Replicas in this mode might never receive updates.
    - benbjohnson 4y ago
      Isolation levels in ACID are typically used for single instances. However, if you're talking about a distributed system then consistency diagram on Jepsen is more applicable[1]. Raft, for example, can ensure read consistency across a cluster by running reads through the consensus mechanism. LiteFS aims to be the analogue to Postgres/MySQL replication and those generally work great for most applications. [1]: https://jepsen.io/consistency https://jepsen.io/consistency
threatofrain 4y ago
Cloudflare D1 is in closed beta so everything is presumably tentative, but atm they don't support transactions (!) and it doesn't appear to be on their near-term product roadmap. That one really surprised me.
mch82 4y ago
Are you adopting the TH3 approach for your codebase? One of the distinguishing things about SQLite is the DO-178B TH3. Not sure if any other open source database have that. http://www3.sqlite.org/th3.html http://www3.sqlite.org/th3.html
- benbjohnson 4y ago
  The TH3 test is proprietary and, IIRC, it might only be available to SQLite Consortium members and that's $125k/year. We do pay for SQLite support at Fly.io but not quite to that level. We do have plans to run against the Tcl test suite[1] although most of that test suite is not applicable to LiteFS since it tests higher level constructs. Since LiteFS acts on the raw pages, it really just functions similar to a VFS. Out of the 130K source lines of code in SQLite, only 4.8k are for the Unix VFS. As such, the testing coverage from the SQLite test suite mostly tests non-VFS code. [1]: https://github.com/superfly/litefs/issues/17 https://github.com/superfly/litefs/issues/17

benbjohnson 4y ago

> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream for you. Oh, I forgot to touch on this point. LiteFS uses some concepts to Litestream (e.g. log shipping), however, it doesn't use Litestream internally. It has much stricter requirements in terms of ensuring consistency since it's distributed so it performs an incremental checksum of the database on every transaction. It has additional benefits with its internal storage format called LTX. These storage files can be compacted together which will allow point-in-time restores that are nearly instant.

no_wizard 4y ago

Note: I apologize if this is overstepping, its hard to tell! I think a strong - extremely strong - selling point is the point I made about "prebaked" data for your APIs, since the entire strength of these SQLite based systems reside in their fast read capacity (as mentioned elsewhere and in this article, its very fast for read heavy applications, which is most) you could take on an angle around that to get people "in the door" by showing a pathway of how this fits inside your existing data warehouse / storage model. We found we liked the SQLite durability to do this. It was a bit smarter than just a plain cache (such as Redis) with better durability and (for our needs) comparable enough performance (I think in absolute terms, a tuned Redis instance will always be faster, but up to a certain point, speed isn't everything, especially when factoring cost). We found it was cheaper - by a good margin - to do this over caching everything AOT in a redis cluster, and we could therefore much more cheaply go multi-region and have DB's sitting next to our customers that acted as a nearline cache. The complexity - which is an area where this might help in the future, and why I'm mentioning it - is shipping changes back. What we ended up doing is setting up a write Redis cluster that clients write to, and we take those writes and trigger a propogation job back to the database. This allowed us to run a much slimmer redis cluster and made us feel more comfortable doing cache eviction since we could verify writes pretty easily. You could do this with memcache or whatever too. Sounds convoluted, but it worked really well. It allowed us to keep our centralized database intact without having to spin up expensive instances to be multi-region or commit to ever growing Redis cluster(s). the SQLite flat file model + history of durability made things the perfect tradeoff for this use case. Of course, YMMV, however it was a novel solution that used "enterprise grade" parts all the way down, which made it an easy selling point. You might find it worth exploring this more. As far as the comparisons go, I think it'd be cool to see a deep dive, and run a test suite against each of the major SQLite as a distributed database model. For that, I don't think it has to be open source to do a reasonable comparison?

benbjohnson 4y ago
> I think a strong - extremely strong - selling point is the point I made about "prebaked" data for your APIs. I think caches are an excellent use case for LiteFS early on. Sorry I didn't make that point in my previous reply. It's a good way to get benefits out of LiteFS without committing to it as your source of truth. Also related, Segment built a custom SQLite-based solution[1] for distributing out cached data that worked well for them. [1]: https://segment.com/blog/separating-our-data-and-control-planes-with-ctlstore/ https://segment.com/blog/separating-our-data-and-control-pla... > We found it was cheaper - by a good margin - to do this over caching everything AOT in a redis cluster Do you remember specifics of the cost difference? I can imagine it'd be pretty significant since you don't need to spin up servers with a bunch of RAM. > Note: I apologize if this is overstepping, its hard to tell! Not overstepping at all! It's great to hear folks' feedback.

ignoramous 4y ago

Thanks. > LiteFS still maintains serializable isolation within a transaction, although, it has looser guarantees across nodes than something like rqlite. Picking up a term from the consistency map here [0], what guarantees LiteFS makes across nodes? [0] https://jepsen.io/consistency https://jepsen.io/consistency

benbjohnson 4y ago
Good question. Right now, LiteFS operates with async replication so its possible to have a transaction written to the primary get lost if the primary fails before it's replicated out. For that short window, you could read a transaction that no longer exists. From that standpoint, I believe it would technically be Read Uncommitted. However, during normal operation it'll function more like Snapshot Isolation. LiteFS does provide a transaction ID so requests could wait for a replica to catch up before issuing a transaction to get something closer to Serializable.
- Spivak 4y ago
  I think you’re being too harsh on yourself, isolation levels don’t typically account for replication lag and network partitioning into account. For example on MySQL you can have async replication and serializable isolation. Replicas in this mode might never receive updates.
  - benbjohnson 4y ago
    Isolation levels in ACID are typically used for single instances. However, if you're talking about a distributed system then consistency diagram on Jepsen is more applicable[1]. Raft, for example, can ensure read consistency across a cluster by running reads through the consensus mechanism. LiteFS aims to be the analogue to Postgres/MySQL replication and those generally work great for most applications. [1]: https://jepsen.io/consistency https://jepsen.io/consistency

threatofrain 4y ago

Cloudflare D1 is in closed beta so everything is presumably tentative, but atm they don't support transactions (!) and it doesn't appear to be on their near-term product roadmap. That one really surprised me.

mch82 4y ago

Are you adopting the TH3 approach for your codebase? One of the distinguishing things about SQLite is the DO-178B TH3. Not sure if any other open source database have that. http://www3.sqlite.org/th3.html http://www3.sqlite.org/th3.html

benbjohnson 4y ago
The TH3 test is proprietary and, IIRC, it might only be available to SQLite Consortium members and that's $125k/year. We do pay for SQLite support at Fly.io but not quite to that level. We do have plans to run against the Tcl test suite[1] although most of that test suite is not applicable to LiteFS since it tests higher level constructs. Since LiteFS acts on the raw pages, it really just functions similar to a VFS. Out of the 130K source lines of code in SQLite, only 4.8k are for the Unix VFS. As such, the testing coverage from the SQLite test suite mostly tests non-VFS code. [1]: https://github.com/superfly/litefs/issues/17 https://github.com/superfly/litefs/issues/17