4 ms·

This is distributed SQLite 3, running (I assume at least partially managed?) LiteFS[5] for you. Which is pretty cool! What I'd like to have seen is how this co

This is distributed SQLite 3, running (I assume at least partially managed?) LiteFS[5] for you. Which is pretty cool! What I'd like to have seen is how this compares to things like rqlite[1] or Cloudflare's D1[2] addressed directly in the article That said, I think this is pretty good for things like read replica's. I know the sales pitch here is as a full database, and I don't disagree with it, and if I was starting from scratch today and could use this, I totally would give it a try and benchmark / test accordingly, however I can't speak to that use case directly. What I find however and what I can speak to, is that most workloads already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common). This is a great way to make very - insanely, really - fast read replica's across regions of your data. You can use an independent raft[3][4] implementation to do this on write. If your database supports it, you can even trigger a replication directly from a write to the database itself (I think Aurora has this ability, and I think - don't quote me! - PostgreSQL can do this natively via an extension to kick off a background job) To that point, in my experience one thing SQLite is actually really good at is storing JSON blobs. I have successfully used it for replicating JSON representations of read only data in the past to great success, cutting down on read times significantly for APIs as the data is "pre-baked" and the lightweight nature of SQLite allows you to - if you wanted to naively do this - just spawn a new database for each customer and transform their data accordingly ahead of time. Its like AOT compilation for your data. if you want to avoid some complexity with sharding (you can't always avoid it outright, but this can help cap its complexity) this approach helps enormously in my experience. Do try before you buy! EDIT: Looks like its running LiteFS[5] not LiteStream[0]. This is my error of understanding. [0]: https://litestream.io/ https://litestream.io/ [1]: https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite [2]: https://blog.cloudflare.com/introducing-d1/ https://blog.cloudflare.com/introducing-d1/ [3]: https://raft.github.io/ https://raft.github.io/ [4]: https://raft.github.io/#implementations https://raft.github.io/#implementations [5]: https://github.com/superfly/litefs https://github.com/superfly/litefs

deleted 4y ago
[deleted]
Gys 4y ago
> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream[0] for you. Litestream and LiteFS are by the same author and serve different purposes. https://litestream.io/alternatives/ https://litestream.io/alternatives/ goes into this. It includes LiteFS and rqlite. [Edited for clarification]
- no_wizard 4y ago
  I did see this, its a little light on details in the way I was thinking about this, however I appreciate the call out, especially for others! It is missing the D1 comparison though.
  - benbjohnson 4y ago
    I'm happy to add D1 onto the alternatives list once it's released. I believe it's still in closed beta though.
benbjohnson 4y ago
LiteFS/Litestream author here. You bring up a lot of great points that I'll try to address. > What I'd like to have seen is how this compares to things like rqlite or Cloudflare's D1 addressed directly in the article I think a post comparing the different options is a great idea. I'll try to summarize a bit here though. LiteFS aims to be an analogue to Postgres replication but with built-in failover. Postgres uses log shipping to copy database state from a primary to its replicas, as does LiteFS. LiteReplica is probably the closest thing to LiteFS although it uses a dual GPL/commercial license model. LiteFS uses Apache 2. There are Raft-based tools like rqlite or dqlite. These have higher consistency guarantees, however, they tend to be more complex to set up -- especially in an ephemeral environment like Kubernetes. LiteFS has a more relaxed membership model for this reason. As for Cloudflare's D1, they haven't released many details and I assume it's closed source. It also requires using a custom JavaScript client instead of using a native SQLite client. There are also several eventually consistent stores built on SQLite such as Mycelial[1]. These work great for applications with loose consistency needs. LiteFS still maintains serializable isolation within a transaction, although, it has looser guarantees across nodes than something like rqlite. > Most workloads are already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common) Yes, that's absolutely true. I don't expect anyone to port their application from Postgres/MySQL to SQLite so they can use LiteFS. Databases and database tools have a long road to becoming mainstream and it follows the traditional adoption curve. I wrote a Go database library called BoltDB about 10 years ago and it had the same adoption concerns early on. Typically, folks try it out with toy applications and play around with it. Once they get more comfortable, then they create new applications on top of it. As more people use it, then late adopters get more comfortable and it further builds trust. We're committed to LiteFS for the long term so we'll be making updates, fixing bugs, and we'll keep trying to build trust with the community. [1]: https://mycelial.com/ https://mycelial.com/
- benbjohnson 4y ago
  > This is distributed SQLite 3, running (I assume at least partially managed?) Litestream for you. Oh, I forgot to touch on this point. LiteFS uses some concepts to Litestream (e.g. log shipping), however, it doesn't use Litestream internally. It has much stricter requirements in terms of ensuring consistency since it's distributed so it performs an incremental checksum of the database on every transaction. It has additional benefits with its internal storage format called LTX. These storage files can be compacted together which will allow point-in-time restores that are nearly instant.

deleted 4y ago

[deleted]

Gys 4y ago

> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream[0] for you. Litestream and LiteFS are by the same author and serve different purposes. https://litestream.io/alternatives/ https://litestream.io/alternatives/ goes into this. It includes LiteFS and rqlite. [Edited for clarification]

no_wizard 4y ago
I did see this, its a little light on details in the way I was thinking about this, however I appreciate the call out, especially for others! It is missing the D1 comparison though.
- benbjohnson 4y ago
  I'm happy to add D1 onto the alternatives list once it's released. I believe it's still in closed beta though.

benbjohnson 4y ago

LiteFS/Litestream author here. You bring up a lot of great points that I'll try to address. > What I'd like to have seen is how this compares to things like rqlite or Cloudflare's D1 addressed directly in the article I think a post comparing the different options is a great idea. I'll try to summarize a bit here though. LiteFS aims to be an analogue to Postgres replication but with built-in failover. Postgres uses log shipping to copy database state from a primary to its replicas, as does LiteFS. LiteReplica is probably the closest thing to LiteFS although it uses a dual GPL/commercial license model. LiteFS uses Apache 2. There are Raft-based tools like rqlite or dqlite. These have higher consistency guarantees, however, they tend to be more complex to set up -- especially in an ephemeral environment like Kubernetes. LiteFS has a more relaxed membership model for this reason. As for Cloudflare's D1, they haven't released many details and I assume it's closed source. It also requires using a custom JavaScript client instead of using a native SQLite client. There are also several eventually consistent stores built on SQLite such as Mycelial[1]. These work great for applications with loose consistency needs. LiteFS still maintains serializable isolation within a transaction, although, it has looser guarantees across nodes than something like rqlite. > Most workloads are already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common) Yes, that's absolutely true. I don't expect anyone to port their application from Postgres/MySQL to SQLite so they can use LiteFS. Databases and database tools have a long road to becoming mainstream and it follows the traditional adoption curve. I wrote a Go database library called BoltDB about 10 years ago and it had the same adoption concerns early on. Typically, folks try it out with toy applications and play around with it. Once they get more comfortable, then they create new applications on top of it. As more people use it, then late adopters get more comfortable and it further builds trust. We're committed to LiteFS for the long term so we'll be making updates, fixing bugs, and we'll keep trying to build trust with the community. [1]: https://mycelial.com/ https://mycelial.com/

benbjohnson 4y ago
> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream for you. Oh, I forgot to touch on this point. LiteFS uses some concepts to Litestream (e.g. log shipping), however, it doesn't use Litestream internally. It has much stricter requirements in terms of ensuring consistency since it's distributed so it performs an incremental checksum of the database on every transaction. It has additional benefits with its internal storage format called LTX. These storage files can be compacted together which will allow point-in-time restores that are nearly instant.