benbjohnson

10 ms·

31.
by benbjohnson 3y ago
Author here. Cool to see the post make it up on HN again. I'm still as excited as ever about the SQLite space. So much great work going on from rqlite, cr-sqlite, & Turso, and we're still plugging away on LiteFS. I'm happ
32.
by benbjohnson 3y ago
Author here. My goal with the article was to write about an use of LiteFS that I found to be useful and show the benefits and trade-offs. I don't think it's a general purpose technique for everyone but I think it has its place in
33.
by benbjohnson 3y ago
hey Ryan, thanks! As for Corrosion, I can't say anything publicly but something may be announced in the near future wink wink :)
34.
by benbjohnson 3y ago
I think a spectrum is a great way to look at it. If you're adding one service then it's no big deal but as you add more and more then you get a better sense of the requirements of the API and I think you're in a better place
35.
by benbjohnson 3y ago
LiteFS works similarly to async replication you'd find in Postgres or MySQL so it doesn't try to be as strict as something running a distributed consensus protocol like Raft. The guarantees for async replication are fairly loose s
36.
by benbjohnson 3y ago
Yes, Postgres replication would work as well. I agree that the "getting to know the schema" part is an issue. I think there's use cases out there where you have power users that would gladly invest extra time in exchange for
37.
by benbjohnson 3y ago
Author here. We do this with some internal applications that share internal state -- no customer data. I would expect stricter restrictions depending on the type of data shared and how bureaucratic the company is.
38.
by benbjohnson 3y ago
Author here. It's not meant as a one-size-fits-all approach. It was an interesting side effect that we noticed that we used internally so I wanted to share my experience. The database mentioned in the post is about 8GB and it replicati
39.
by benbjohnson 3y ago
Author here. I hadn't heard of Sequin until today. "Skip the API" was just a snappy title so I went with it.
40.
by benbjohnson 3y ago
Author here. I don't see this is a general practice to be used for most applications. It was a side effect that we came across and it allowed us to share data between internal applications in some interesting ways. Internal apps are id
41.
by benbjohnson 3y ago
Author here. We're using LiteFS to replicate SQLite databases in real time so the changes sent are only incremental. I think there are several ideal use cases for sharing databases across applications: 1. Internal tooling: you're
42.
by benbjohnson 3y ago
Author here. I think we could have set better expectations with our Postgres docs. It wasn't meant to be a managed service but rather some tooling to help streamline setting up a database and replicas. I'm sorry about the troubles
43.
by benbjohnson 3y ago
That makes sense! Let me know if you have any questions. There's a Litestream Slack you can find on the home page or email me at benbjohnson@yahoo.com
44.
by benbjohnson 3y ago
Litestream author here. The streaming replication was moved to a different project called LiteFS[1]. That may be a fit for you or, at the very least, it could be a helpful reference for your Rust port. [1]: https://github.com&#x2
45.
by benbjohnson 3y ago
Sorry, I should have been more clear. HA = High Availability. Essentially, folks wanted the ability to automatically failover to a secondary node if the current primary failed.
46.
by benbjohnson 3y ago
LiteFS author here. I think single-node deployments are great and it's a model that fits many (or even most) apps out there. However, the two most common feature requests when I was developing Litestream were HA & replication. Whil
47.
by benbjohnson 3y ago
Litestream & LiteFS author here. I think single-node, easily-recoverable systems are great and it fits a lot of people's use cases. VPS providers are pretty reliable too so even using a regular hourly backup can be good enough for
48.
by benbjohnson 3y ago
That's not true. LiteFS itself is a distributed database so you have redundancy within your cluster outside of LiteFS Cloud. A typical setup is to run two candidate nodes within a single region so they have low replication latency and
49.
by benbjohnson 3y ago
Author here. We run LiteFS in production for several services internally. The recent issue was pretty rare and it took a while just to be able to reproduce it. There's always going to be bugs in all software. Hopefully they just become
50.
by benbjohnson 3y ago
WAL solves the high concurrency read situation. Not the writes. SQLite can do thousands of writes per second in WAL mode which is more than enough for the vast majority of applications out there. It's not like most businesses could ful
51.
by benbjohnson 3y ago
No, LiteFS just does physical page replication. We don't really have a way to do merge conflict resolution between two nodes. You may want to look at either vlcn[1] or Mycelite[2] as options for doing that approach. [1]: https:/&
52.
by benbjohnson 3y ago
It's hard to say definitively. If you're issuing multiple database queries per request and you aren't using really complicated Postgres SQL then I would guess that you'll see a performance boost. In tests I've done
53.
by benbjohnson 3y ago
I agree with everything the OP said above. Typically if you need to scale writes in SQLite, you'll want to look at sharding. The "single writer" restriction is per database so you can split your SaaS customers across multiple
54.
by benbjohnson 3y ago
It's hard to say. If Postgres is working for you then it might not be worth the trouble. Not everything migrates one-to-one between database vendors. Fly.io takes daily snapshots of your server volumes so that's one approach to ba
55.
by benbjohnson 3y ago
Yes, you have the current model correct. To support Lambda, we'll need to move the lock to LiteFS Cloud and allow writes directly to it. The write performance won't be as fast as a local LiteFS instance that is always the primary
56.
by benbjohnson 3y ago
Good question. LTX uses a rolling checksum of the entire database at every transaction. Actually, it includes two checksums: one for the state of the database before the transaction and one for after the transaction. It's incremental s
57.
by benbjohnson 3y ago
Good question. LiteFS is just a replication layer so it's probably better to answer your question just from a SQLite standpoint. One of the biggest limitations of SQLite is that it only allows a single writer at a time so you can'
58.
by benbjohnson 3y ago
I don't think the issue is unique to SQLite. Postgres & MySQL both have async replication options that are commonly used. There's always going to be a latency and throughput trade-off when you use synchronous replication and m
59.
by benbjohnson 3y ago
Thanks! I thought DonutDB was an awesome approach too. We do have plans for supporting ephemeral serverless (e.g. Lambda, Vercel) by paging in data on-demand and caching it in the temp space but that work is still a little ways out. I'
60.
by benbjohnson 3y ago
Author here. I'm planning on writing up a blog post about FUSE performance. I get a lot of questions about it. In practical terms, it mostly affects write performance. If you have a write-heavy application then it's probably not a

More ›