benbjohnson

11 ms·

91.
by benbjohnson 4y ago
Currently, you need to SSH in and use the sqlite3 CLI on the server. There has been some work in this area but it's all still rough around the edges. I wrote a server called Postlite[1] that exposes remote SQLite databases over the Pos
92.
by benbjohnson 4y ago
It depends on what you're trying to do. If you're trying to query revenue then that's usually stored in a third-party system like Stripe. If you're tracking metrics then those can be aggregated into a time series databas
93.
by benbjohnson 4y ago
The SYNCHRONOUS pragma is great but I'll mention that there is a durability trade-off. In "NORMAL" mode, there is not an fsync() after every transaction so you could lose recent transactions if you unexpectedly shutdown. The
94.
by benbjohnson 4y ago
We're targeting databases of 1 to 10 GB right now and those seem to work pretty well overall. I'm sure we'll expand that target over time as LiteFS matures though. Most of the requests I've seen have been to support a lo
95.
by benbjohnson 4y ago
Litestream & LiteFS author here. Tailscale has been using Litestream in production for a while and blogged about it in April 2022[1]. I've had a number of folks contact me out-of-band and tell me they're using in production at
96.
by benbjohnson 4y ago
One implementation detail about QUIC that I was surprised by was that it requires TLS. That’s great for improving the security on the public Internet but it seems like it adds complexity and CPU overhead if you’re running on something like
97.
by benbjohnson 4y ago
Thanks!
98.
by benbjohnson 4y ago
A request can have a large range of queries within it so I was trying to account for that. If your requests are lightweight and mostly reads, you can do 1,000+ req/sec on a 256MB instance. YMMV.
99.
by benbjohnson 4y ago
There aren’t many apps that have 100,000+ req/sec and you certainly can’t run them on modest hardware. If your app averages 100 req/sec then that’s 8.4M requests per day. That’s more than most applications out there.
100.
by benbjohnson 4y ago
Litestream/LiteFS author here. I agree that "cloud" is a bit ambiguous but Go & SQLite are quite powerful together and I don't think it's only for beginners. Both are fast and have low overhead. In addition to l
101.
by benbjohnson 4y ago
I've hired 2 bootcamp grads from Turing in Denver. They were great and have both gone on to senior roles. I mentored another woman in a bootcamp about ten years ago and she later went on to be a senior engineer at HashiCorp. She then g
102.
by benbjohnson 4y ago
No problem. Jepsen is something we're interested in once the project reaches a certain level of maturity. The replication is pretty basic primary/replica async streaming replication so I'm not sure there are too many surprise
103.
by benbjohnson 4y ago
Thanks! WAL support is in the `main` branch and we're just doing some testing and bug squashing right now. I'm hoping we can release v0.3.0 next week with WAL support.
104.
by benbjohnson 4y ago
Litestream/LiteFS author here. Litestream still exists and it works as a disaster recovery system. That hasn't changed. LiteFS (aka bizarro FUSE filesystem) is for distributing databases geographically. It's admittedly still
105.
by benbjohnson 4y ago
FUSE is a better developer experience since the end user doesn't have to worry about loading the extension every time they use a SQLite driver or the sqlite3 CLI. LiteFS runs an HTTP API server to communicate between nodes as well and
106.
by benbjohnson 4y ago
LiteFS author here. I haven't tested it on Kubernetes yet but it is meant to be deployed anywhere. The only dependency is Consul although you could get around that by using a static leader[1]. [1]: https://github.com/su
107.
by benbjohnson 4y ago
LiteFS author here. You can also set up LiteFS to have a single, static leader instead of using Consul if you don't want that dependency. I need to write up some docs but there's some info on the PR itself: https://gith
108.
by benbjohnson 4y ago
It's not great. tv42 (who maintains the FUSE implementation we use) commented on it recently[1]. macOS support will be via a VFS extension in the future. [1]: https://github.com/superfly/litefs/issues/119
109.
by benbjohnson 4y ago
> What's the overhead of using FUSE to implement LiteFS? It's a tricky question to answer. Most of the noticeable overhead is on the write side. Initial benchmarks of overhead that I've seen locally are about 250µs for the
110.
by benbjohnson 4y ago
How is there vendor lock-in? The LiteFS & Litestream code is all open source under Apache 2.
111.
by benbjohnson 4y ago
LiteFS author here. That's a good explanation, Simon. Preventing writes on the replica is a nice benefit to FUSE versus an external process. Another benefit of the control is that we can maintain a rolling checksum of the entire databa
112.
by benbjohnson 4y ago
Litestream/LiteFS author here. I agree that synchronous replication is important and we have plans to implement it in LiteFS. Because LiteFS supports a loose membership model, quorum-based acknowledgement doesn't really work as we
113.
by benbjohnson 4y ago
LiteFS author here. It depends on the application. You can check the replication position on the replicas and simply wait until the replica catches up. That requires maintaining that position on the client though. An easier approach that wo
114.
by benbjohnson 4y ago
Yes, the app can read it whenever. There's a brief lock obtained on the current database position internally to read it out. The position only gets updated at the end of the transaction.
115.
by benbjohnson 4y ago
The TH3 test is proprietary and, IIRC, it might only be available to SQLite Consortium members and that's $125k/year. We do pay for SQLite support at Fly.io but not quite to that level. We do have plans to run against the Tcl test
116.
by benbjohnson 4y ago
Good catch. LiteFS has an option to mark nodes as candidates or not. For example, you can run a cluster with a couple candidate nodes in one region (e.g Chicago) and every other region may just run a single non-electable replica. We don’t s
117.
by benbjohnson 4y ago
Isolation levels in ACID are typically used for single instances. However, if you're talking about a distributed system then consistency diagram on Jepsen is more applicable[1]. Raft, for example, can ensure read consistency across a c
118.
by benbjohnson 4y ago
> There's a potential performance win if the data you're reading is already in the VFS cache, since that would bypass having to through the FUSE filesystem. That's a typical case for a lot of databases. Most of the "h
119.
by benbjohnson 4y ago
Litestream does log shipping so it just takes a chunk of the write-ahead log (WAL) and copies it out a destination. That can be S3, GCP, SFTP, etc. LiteFS will do the same although I'm only targeting S3 initially. That seemed to be the
120.
by benbjohnson 4y ago
It’s available as a file handle. If you have a database file named “db” then you can read “db-pos” to read the replication position. The position is a tuple of the TXID and the database checksum.

More ›