15 ms·

LiteFS

648 points by danielskogly 4y ago

mwcampbell 4y ago
I wonder if using FUSE has had any appreciable impact on performance, particularly read performance. I ask because FUSE has historically had a reputation for being slow, e.g. with the old FUSE port of ZFS.
- yencabulator 4y ago
  Often, the SQLite database would fit in RAM, and reads would be served straight from the page cache, with no overhead. Disclaimer: I wrote the FUSE framework LiteFS uses, https://bazil.org/fuse https://bazil.org/fuse -- and I also have some pending performance-related work to finish, there...
- miduil 4y ago
  I think FUSE performance has significantly improved since 2010, the year in which ZFS-On-Linux became available I believe. This is just one of many examples: https://www.phoronix.com/news/MTI1MzM https://www.phoronix.com/news/MTI1MzM In contrary, in some cases FUSE is even faster than doing a regular kernel mount(). There is an experimental research distribution called distri that is exclusively relying on fuse mounting and figured out that FUSE was faster for doing mounts. https://michael.stapelberg.ch/posts/tags/distri/ https://michael.stapelberg.ch/posts/tags/distri/
  - seabrookmx 4y ago
    You're conflating zfs-fuse and zfs-on-linux (ZoL). The first one is as it's name suggests.. a user space implementation of ZFS. ZoL (now unified with OpenZFS) is implemented as a kernel module and as such does _not_ run in user space. It performs significantly better as a result. FUSE is still slow, which is why there's ongoing effort to replace things like NTFS-3G (the default NTFS implementation in most linux distros) with an in-kernel implementation: https://news.ycombinator.com/item?id=28418674 https://news.ycombinator.com/item?id=28418674
    - miduil 4y ago
      How do I conflate zfs-fuse with ZoL? I was trying to highlight that if someones memory of "poor fuse performance" is because of zfs-fuse, then someones memory must be quite old, as ZoL has been available for quite some time and since then also fuse itself has received quite some performance gains. Edit: Also I don't want to imply that FUSE is near in-kernel filesystems, but it is certainly performing much better than 12 years ago.
no_wizard 4y ago
This is distributed SQLite 3, running (I assume at least partially managed?) LiteFS[5] for you. Which is pretty cool! What I'd like to have seen is how this compares to things like rqlite[1] or Cloudflare's D1[2] addressed directly in the article That said, I think this is pretty good for things like read replica's. I know the sales pitch here is as a full database, and I don't disagree with it, and if I was starting from scratch today and could use this, I totally would give it a try and benchmark / test accordingly, however I can't speak to that use case directly. What I find however and what I can speak to, is that most workloads already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common). This is a great way to make very - insanely, really - fast read replica's across regions of your data. You can use an independent raft[3][4] implementation to do this on write. If your database supports it, you can even trigger a replication directly from a write to the database itself (I think Aurora has this ability, and I think - don't quote me! - PostgreSQL can do this natively via an extension to kick off a background job) To that point, in my experience one thing SQLite is actually really good at is storing JSON blobs. I have successfully used it for replicating JSON representations of read only data in the past to great success, cutting down on read times significantly for APIs as the data is "pre-baked" and the lightweight nature of SQLite allows you to - if you wanted to naively do this - just spawn a new database for each customer and transform their data accordingly ahead of time. Its like AOT compilation for your data. if you want to avoid some complexity with sharding (you can't always avoid it outright, but this can help cap its complexity) this approach helps enormously in my experience. Do try before you buy! EDIT: Looks like its running LiteFS[5] not LiteStream[0]. This is my error of understanding. [0]: https://litestream.io/ https://litestream.io/ [1]: https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite [2]: https://blog.cloudflare.com/introducing-d1/ https://blog.cloudflare.com/introducing-d1/ [3]: https://raft.github.io/ https://raft.github.io/ [4]: https://raft.github.io/#implementations https://raft.github.io/#implementations [5]: https://github.com/superfly/litefs https://github.com/superfly/litefs
- deleted 4y ago
  [deleted]
- Gys 4y ago
  > This is distributed SQLite 3, running (I assume at least partially managed?) Litestream[0] for you. Litestream and LiteFS are by the same author and serve different purposes. https://litestream.io/alternatives/ https://litestream.io/alternatives/ goes into this. It includes LiteFS and rqlite. [Edited for clarification]
lijogdfljk 4y ago
This is really cool! Unfortunately i primarily am interested in offline databases so perhaps i'm just not the target audience. However i have to ask, on that note, does this have any application in the offline space? Ie i wonder if there's a way to can write your applications such that they have less/minimal contention, and then allow the databases to merge when back online? Of course, what happens when there inevitably _is_ contention? etc Not sure that idea would have a benefit over many SQLite DBs with userland schemas mirroring CRDT principles though. But a boy can dream. Regardless, very cool work being done here.
- benbjohnson 4y ago
  Thanks! It depends on your particular use case. If you need all nodes to allow writes then, unfortunately, no it doesn't support that. LiteFS uses physical replication so there's no way to merge pages from separate writers. You're probably better off looking at eventually consistent SQLite tools like Mycelial[1]. However, if you have a single writer and just need replicas to get updates when they periodically connect then yes, LiteFS could fit that use case. [1]: https://mycelial.com/ https://mycelial.com/
- bigfish24 4y ago
  Shameless plug but would love for you to try out Ditto[1]! CRDT-based replicated database that works peer-to-peer including automatically with WiFi and Bluetooth. [1] https://www.ditto.live https://www.ditto.live
asim 4y ago
10 years ago fly.io is the company I wanted to build. Something with massive technical depth that becomes a developer product. They're doing an incredible job and part of that comes down to how they evangelise the product outside of all the technical hackery. This requires so much continued effort. AND THEN to actually run a business on top of all that. Kudos to you guys. I struggled so much with this. Wish you nothing but continued success.
- ignoramous 4y ago
  5 years ago, fly.io was basically poor man's deno.com / oven.sh [0]. In my (incorrect) opinion, tptacek changed fly.io's trajectory single-handedly. [0] https://ghostarchive.org/varchive/r-1hXDvOoHA https://ghostarchive.org/varchive/r-1hXDvOoHA
  - stingraycharles 4y ago
    What is @tptacek’s relation with fly.io, and how did he change it?
    - ignoramous 4y ago
      tptacek is a security researcher and principal at fly.io [0]. And as an outsider looking in, seems to have been the eng behind some of their defining features [1]: - udp support - container2vm overhaul - private networks aka 6pn - some key flyctl (cli) commands like flyctl ssh, flyctl proxy - metrics - litefs - perhaps, the imminent overhaul of the orchestration layer (?) - the upcoming authz layer [0] https://archive.is/Zwzlh https://archive.is/Zwzlh [1] https://community.fly.io/u/thomas/activity/topics https://community.fly.io/u/thomas/activity/topics | https://fly.io/blog/author/thomas/ https://fly.io/blog/author/thomas/
      - yencabulator 4y ago
        > seems to have been the eng behind some of their defining features > - litefs That glory goes to https://news.ycombinator.com/user?id=benbjohnson https://news.ycombinator.com/user?id=benbjohnson
theomega 4y ago
Does anyone else bump into the issue, that the fly.io website does not load if requested via IPv6 on Mac? I tried Safari, Chrome and curl and neither work: $ curl -v https://fly.io/blog/introducing-litefs/ * Trying 2a09:8280:1::a:791:443... * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer Requesting via ipv4 works $ curl -4v https://fly.io/blog/introducing-litefs/ * Trying 37.16.18.81:443... * Connected to fly.io (37.16.18.81) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=fly.io * start date: Jul 25 11:20:01 2022 GMT * expire date: Oct 23 11:20:00 2022 GMT * subjectAltName: host "fly.io" matched cert's "fly.io" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. * Using HTTP2, server supports multiplexing * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x135011c00) > GET /blog/introducing-litefs/ HTTP/2 > Host: fly.io > user-agent: curl/7.79.1 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 32)! < HTTP/2 200 < accept-ranges: bytes < cache-control: max-age=0, private, must-revalidate < content-type: text/html < date: Wed, 21 Sep 2022 16:50:16 GMT < etag: "632b20f0-1bdc1" < fly-request-id: 01GDGFA3RPZPRDV9M3AQ3159ZK-fra < last-modified: Wed, 21 Sep 2022 14:34:24 GMT < server: Fly/51ee4ef9 (2022-09-20) < via: 1.1 fly.io, 2 fly.io < <!doctype html> ...
- tptacek 4y ago
  Works fine over IPv6 from this Mac.
- toast0 4y ago
  Could easily be a path MTU issue? I don't currently have working IPv6, but on TCP/IPv4, the server->client handshake packets I get back from fly.io are full length packets. If your tcpv6 MSS is set incorrectly, and some other not great things are happening in the path, then you might not be able to receive large packets.
ranger_danger 4y ago
ELI5?
- benbjohnson 4y ago
  LiteFS makes it so that you can have a SQLite database that is transparently replicated to a cluster of machines. The use case is if you have your app running on a server running in Chicago, then users in Europe will have 100ms latency to your server and users in Asia will have a 250ms latency. That's on top of the time it takes for your server to process the request. Many people target sub-100ms latency to make their applications feel snappy but that's impossible if users are waiting on their request to be sent halfway around the world. Traditionally, it's complicated to replicate your database to different regions of the world using something like Postgres. However, LiteFS aims to make it as simple as just spinning up more server instances around the world. It connects them automatically and ships changes in a transactionally safe way between them.
  - boltzmann-brain 4y ago
    > Traditionally, it's complicated to replicate your database to different regions of the world using something like Postgres what makes it complicated?
- twobitshifter 4y ago
  It’s a tool to allow you to keep your database on the app server and backed up on s3. This means you don’t need a separate database server. Having a database on the same machine as the app is very fast, but it is usually risky. The backups made by LiteFS lessen the risk.
  - hobo_mark 4y ago
    That's Litestream, not LiteFS.
pphysch 4y ago
> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing the individual query overhead due to TCP roundtrip. SQLite is a great technology, but keep in mind that you can also deploy Postgres right next to your app as well. If your app is something like a company backend that could evolve a lot and benefit from Postgres's advanced features, this may be the right choice.
- tptacek 4y ago
  Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?
  - pphysch 4y ago
    I would wager that the median IT shop/dept is running off a single node RDBMS, the majority of which also have the spare resources to support running the primary application (which has 1-100 DAU). No practical way to test this, of course.
  - whalesalad 4y ago
    is it cheating when pgbouncer is being accessed via socket? lol
    - tptacek 4y ago
      Yes, that is cheating.
      - kjeetgill 4y ago
        I don't have any context around this: why is that cheating?
hobo_mark 4y ago
Well that was fast... [1] Are the readable replicas supposed to be long-lived (as in, I don't know, hours)? Or does consul happily converge even with ephemeral instances coming and going every few minutes (thinking of something like Cloud Run and the like, not sure if Fly works the same way)? And do they need to make a copy of the entire DB when they "boot" or do they stream pages in on demand? [1] https://news.ycombinator.com/item?id=32240230 https://news.ycombinator.com/item?id=32240230
- benbjohnson 4y ago
  > Are the readable replicas supposed to be long-lived Right now, yes, they should be relatively long-lived (e.g. hours). Each node keeps a full copy of the database so it's recommended to use persistent volumes with it so it doesn't need to re-snapshot on boot. We do have plans to make this work with very short-lived requests for platforms like Lambda or Vercel. That'll use transactionally-aware paging on-demand. You could even run it in a browser, although I'm not sure why you would want to. :)
whoisjuan 4y ago
Unrelated, but why does the map on their homepage show a region in Cuba? That must be wrong.
- dcre 4y ago
  I think it's in Miami and the icon is confusingly centered over it. The icons look like pins but I think they're supposed to be hot-air balloons. This explains why some of them appear to point to spots in the ocean. https://fly.io/docs/reference/regions/ https://fly.io/docs/reference/regions/
  - whoisjuan 4y ago
    Ahh. Great catch. Their balloon definitely looks like a pin.
nicoburns 4y ago
Where is the data actually being stored in this setup? A copy on each machine running the application? If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? Also, what happens if the Consul instance goes down? If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. If it completely abstracts that away somehow then I suppose that'd be pretty cool. Currently finding it hard to get on board with the idea that adding a distributed system here actually makes things simpler.
- benbjohnson 4y ago
  > Where is the data actually being stored in this setup? A copy on each machine running the application? Yes, each node has a full copy of the database locally. > If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? S3 replication support is coming[1]. Probably in the next month or so. Until then, it's recommended that you run a persistent volume with your nodes. > What happens if the Consul instance goes down? If Consul goes down then you'll lose write availability but your nodes will still be able to perform read queries. > If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. Support for pure ephemeral nodes is coming. If you're comparing LiteFS to a single node Postgres/MySQL then you're right, it's harder to operate. However, distributing out Postgres/MySQL to regions around the world and handling automatic failovers is likely more difficult to operate than LiteFS.
  - omnimus 4y ago
    Will this replicate the filesystem/volume between instances? Can for example use text files or different kind of filebased database? Or it requires me to use sqlite?
    - simonw 4y ago
      This is only for SQLite. But you can put text files (or even binary files - see https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sqlite/ https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sq...) in a SQLite database and you may find you get better performance than reading files from disk: https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html
infogulch 4y ago
> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?
- sk55 4y ago
  They don't have ACID guarantees with this setup.
  - benbjohnson 4y ago
    ACID is typically used to describe single database instances and from that perspective, LiteFS doesn't change the ACID semantics that SQLite has. For distributed consistency models[1], it gets more complicated as LiteFS currently just supports async replication. We'll introduce stronger guarantees in the future although we currently provide some information such as transaction IDs so you can manage replication lag if you need stronger consistency. [1]: https://jepsen.io/consistency https://jepsen.io/consistency
- hinkley 4y ago
  That's a fun one. A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction. Ultimately that's a very odd flavor of sharding. What I don't recall is how they propagated that shard information efficiently, since with sharding and consistent hashing the whole idea is that the lookup algorithm is deterministic, and therefore can be run anywhere and get the same result. WAL lag information is dynamic, so... Raft?
  - simonw 4y ago
    Another solution that I've used successfully in the past for web apps is to set a 10s cookie every time a user performs a write, and then route their read requests to the lead server until that cookie expires. That way they're sure to see the impact of the write they just made.
clord 4y ago
I can imagine a database which tries to solve both of these domains. A centralized database handles consistency, and vends data closures to distributed applications for in-process querying (and those closures reconcile via something like CRDT back to the core db). Does this exist?
- ignoramous 4y ago
  (except for the "reconcile back" to primary) Yes? https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html https://martin.kleppmann.com/2015/11/05/database-inside-out-... See also: materialize.com, readyset.io, aws elastic-views (forever in preview).
deleted 4y ago
[deleted]
vcryan 4y ago
Currently, I am running multiple application servers (and lambda functions) using AWS Fargate that access sqlite files (databases) on an EFS share. So far so good, although my use cases are fairly simple.
vcryan 4y ago
This approach is very appealing to me :) curious about how people handle schema migrations when using this approach. I segment sqlite files (databases) that have the same schema into the same folder. I haven't really had a case where migrations was really a concern, but I could see it happening soon. Seems like in my deployment, I'm going to need an approach to loop over dbs to apply this change... I currently have a step of app deployment that attempts to apply migrations... but it is more simplistic because the primary RDBMS (postgresql) just appears to the application as a single entity which is the normative use-case for db-migrate-runners.
- simonw 4y ago
  Yeah schema migrations are going to be interesting. Presumably those end up making a HUGE change to the WAL (since they could affect every stored row in a large table), which means LiteFS then has to send a truly giant chunk of data out to the replicas. I wonder how this would present itself in real-world usage? Would the replicas go out-of-date for 10-30s but continue serving read-only traffic, or could there be some element of cluster downtime caused by this?
  - benbjohnson 4y ago
    > Presumably those end up making a HUGE change to the WAL (since they could affect every stored row in a large table), which means LiteFS then has to send a truly giant chunk of data out to the replicas. Yes, a large migration could end up sending a lot of data out to replicas. One way to mitigate this is to shard your data into separate SQLite databases so you're only migrating a small subset at a time. Of course, that doesn't work for every application. > Would the replicas go out-of-date for 10-30s but continue serving read-only traffic, or could there be some element of cluster downtime caused by this? Once WAL support is in LiteFS, it will be able to replicate this out without causing any read down time. SQLite is still a single-writer database though so writes would be blocked during a migration.
jensneuse 4y ago
Sound like a drop in solution to add high availability to WunderBase (https://github.com/wundergraph/wunderbase https://github.com/wundergraph/wunderbase). Can we combine LiteFS with Litestream for Backups, or how would you do HA + Backups together?
- benbjohnson 4y ago
  Yes, it should fit with WunderBase to provide HA. I'm adding S3 support which will work the same (in principle) as Litestream. I'm hoping to have that in about a month or so.
  - rsync 4y ago
    Can you comment on how you added SFTP support to litestream so that it would work with rsync.net[1][2] ? How does that compare/contrast with what my grandparent is alluding to ? [1] https://github.com/benbjohnson/litestream/issues/140 https://github.com/benbjohnson/litestream/issues/140 [2] https://www.rsync.net/resources/notes/2021-q3-rsync.net_technotes.html https://www.rsync.net/resources/notes/2021-q3-rsync.net_tech...
    - benbjohnson 4y ago
      Litestream does log shipping so it just takes a chunk of the write-ahead log (WAL) and copies it out a destination. That can be S3, GCP, SFTP, etc. LiteFS will do the same although I'm only targeting S3 initially. That seemed to be the bulk of what people used.
      - rsync 4y ago
        OK, thank you. As with the litestream component, it would be wonderful to have an SFTP transport for LiteFS. When you say 'S3' I assume you mean "S3 compatible API" so that would be fairly open and portable but SFTP would be even more so. Thanks again.
endisneigh 4y ago
Seems neat, until you try to do schema migrations. Unless they can guarantee that all containers’ SQLite instances have the same scheme without locking I’m not sure how doesn’t run into the same issues as many NoSQL. CouchDB had this same issue with its database per user model and eventually consistent writes.
- yencabulator 4y ago
  I don't see how this is different to a world where you use Postgres over the network, with lots of DB clients. You "just" do the schema migrations like everyone who can't shut down the business does: incrementally and in a backwards-compatible manner. After the migration, deploy new code that takes advantage of the new schema. Repeat until done.
- slively 4y ago
  In my experience distributed database migrations tend to be two phase though so it can be compatible during the migration. The first migration to add something, release the apps to use it, then a second migration to cleanup.
fny 4y ago
This reads like a professor who's so steeped in research that he's forgotten how to communicate to his students! What exactly are we talking about here? A WebSQL thats actually synced to a proper RDBMS? Synced across devices? I'm not clear about an end to end use case. Edit: Honestly, this line from the LiteFS docs[0] needs to be added to the top of the article: > LiteFS is a distributed file system that transparently replicates SQLite databases. This lets you run your application like it's running against a local on-disk SQLite database but behind the scenes the database is replicated to all the nodes in your cluster. This lets you run your database right next to your application on the edge. I had no idea what was being talked about otherwise. [0]: https://fly.io/docs/litefs/ https://fly.io/docs/litefs/
hinkley 4y ago
> Second, your application can only serve requests from that one server. If you fired up your server in Dallas then that'll be snappy for Texans. But your users in Chennai will be cursing your sluggish response times since there's a 250ms ping time between Texas & India. > To improve availability, it uses leases to determine the primary node in your cluster. By default, it uses Hashicorp's Consul. Having a satellite office become leader of a cluster is one of the classic blunders in distributed computing. There are variants of Raft where you can have quorum members that won't nominate themselves for election, but out of the box this is a bad plan. If you have a Dallas, Chennai, Chicago, and Cleveland office and Dallas goes dark (ie, the tunnel gets fucked up for the fifth time this year), you want Chicago to become the leader, Cleveland if you're desperate. But if Chennai gets elected then everyone has a bad time, including Dallas when it comes back online.
- benbjohnson 4y ago
  Good catch. LiteFS has an option to mark nodes as candidates or not. For example, you can run a cluster with a couple candidate nodes in one region (e.g Chicago) and every other region may just run a single non-electable replica. We don’t support any kind of tiering for candidates in different regions. That’s not a bad idea though.
azlyrics 4y ago
I actually gave fly.io a whirl over the weekend. was not fun. spent a lot of time on the forums and its pretty clear it has some way to go before it can give AWS or Linode a run for their money. For instance, we run kubernetes on multiple VPS providers, including public clouds with serverless onramp/offramps deployed on edge location. Anything under 15 minutes are processed by serverless. Anything longer is offloaded to one of the VPS containers available in every part of the world. I have some more feedbacks ready if you are interested, its a neat idea but not exactly as seamless and easy as the idea proposed since public clouds already offer a way to do this.
morenoh149 4y ago
still wondering why projects like https://www.reactivated.io/ https://www.reactivated.io/ are using fly.io
Existenceblinks 4y ago
The pain is that this approach is suitable on VPS/IaaS where disk volume is supported. As a solo dev, I only use PaaS kind of infra, there are just a few PaaS i'm aware of that support attachable disk. Fly, Render, .. nothing else?
presentation 4y ago
My dream would be if this supported geo-partitioning. In my field people are pretty sensitive about GDPR so would love to box in EU PII in EU servers.
- yencabulator 4y ago
  Since the scope is a SQLite database, that means you just run a EU database, with no replicas outside of EU. This level of database replication isn't going to put individual rows in different geographies.

648 points by danielskogly 4y ago

I wonder if using FUSE has had any appreciable impact on performance, particularly read performance. I ask because FUSE has historically had a reputation for being slow, e.g. with the old FUSE port of ZFS.

yencabulator 4y ago
Often, the SQLite database would fit in RAM, and reads would be served straight from the page cache, with no overhead. Disclaimer: I wrote the FUSE framework LiteFS uses, https://bazil.org/fuse https://bazil.org/fuse -- and I also have some pending performance-related work to finish, there...
miduil 4y ago
I think FUSE performance has significantly improved since 2010, the year in which ZFS-On-Linux became available I believe. This is just one of many examples: https://www.phoronix.com/news/MTI1MzM https://www.phoronix.com/news/MTI1MzM In contrary, in some cases FUSE is even faster than doing a regular kernel mount(). There is an experimental research distribution called distri that is exclusively relying on fuse mounting and figured out that FUSE was faster for doing mounts. https://michael.stapelberg.ch/posts/tags/distri/ https://michael.stapelberg.ch/posts/tags/distri/
- seabrookmx 4y ago
  You're conflating zfs-fuse and zfs-on-linux (ZoL). The first one is as it's name suggests.. a user space implementation of ZFS. ZoL (now unified with OpenZFS) is implemented as a kernel module and as such does _not_ run in user space. It performs significantly better as a result. FUSE is still slow, which is why there's ongoing effort to replace things like NTFS-3G (the default NTFS implementation in most linux distros) with an in-kernel implementation: https://news.ycombinator.com/item?id=28418674 https://news.ycombinator.com/item?id=28418674
  - miduil 4y ago
    How do I conflate zfs-fuse with ZoL? I was trying to highlight that if someones memory of "poor fuse performance" is because of zfs-fuse, then someones memory must be quite old, as ZoL has been available for quite some time and since then also fuse itself has received quite some performance gains. Edit: Also I don't want to imply that FUSE is near in-kernel filesystems, but it is certainly performing much better than 12 years ago.

no_wizard 4y ago

This is distributed SQLite 3, running (I assume at least partially managed?) LiteFS[5] for you. Which is pretty cool! What I'd like to have seen is how this compares to things like rqlite[1] or Cloudflare's D1[2] addressed directly in the article That said, I think this is pretty good for things like read replica's. I know the sales pitch here is as a full database, and I don't disagree with it, and if I was starting from scratch today and could use this, I totally would give it a try and benchmark / test accordingly, however I can't speak to that use case directly. What I find however and what I can speak to, is that most workloads already have database of some kind setup, typically not SQLite as their main database (MySQL or PostgreSQL seem most common). This is a great way to make very - insanely, really - fast read replica's across regions of your data. You can use an independent raft[3][4] implementation to do this on write. If your database supports it, you can even trigger a replication directly from a write to the database itself (I think Aurora has this ability, and I think - don't quote me! - PostgreSQL can do this natively via an extension to kick off a background job) To that point, in my experience one thing SQLite is actually really good at is storing JSON blobs. I have successfully used it for replicating JSON representations of read only data in the past to great success, cutting down on read times significantly for APIs as the data is "pre-baked" and the lightweight nature of SQLite allows you to - if you wanted to naively do this - just spawn a new database for each customer and transform their data accordingly ahead of time. Its like AOT compilation for your data. if you want to avoid some complexity with sharding (you can't always avoid it outright, but this can help cap its complexity) this approach helps enormously in my experience. Do try before you buy! EDIT: Looks like its running LiteFS[5] not LiteStream[0]. This is my error of understanding. [0]: https://litestream.io/ https://litestream.io/ [1]: https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite [2]: https://blog.cloudflare.com/introducing-d1/ https://blog.cloudflare.com/introducing-d1/ [3]: https://raft.github.io/ https://raft.github.io/ [4]: https://raft.github.io/#implementations https://raft.github.io/#implementations [5]: https://github.com/superfly/litefs https://github.com/superfly/litefs

deleted 4y ago
[deleted]
Gys 4y ago
> This is distributed SQLite 3, running (I assume at least partially managed?) Litestream[0] for you. Litestream and LiteFS are by the same author and serve different purposes. https://litestream.io/alternatives/ https://litestream.io/alternatives/ goes into this. It includes LiteFS and rqlite. [Edited for clarification]

lijogdfljk 4y ago

This is really cool! Unfortunately i primarily am interested in offline databases so perhaps i'm just not the target audience. However i have to ask, on that note, does this have any application in the offline space? Ie i wonder if there's a way to can write your applications such that they have less/minimal contention, and then allow the databases to merge when back online? Of course, what happens when there inevitably _is_ contention? etc Not sure that idea would have a benefit over many SQLite DBs with userland schemas mirroring CRDT principles though. But a boy can dream. Regardless, very cool work being done here.

benbjohnson 4y ago
Thanks! It depends on your particular use case. If you need all nodes to allow writes then, unfortunately, no it doesn't support that. LiteFS uses physical replication so there's no way to merge pages from separate writers. You're probably better off looking at eventually consistent SQLite tools like Mycelial[1]. However, if you have a single writer and just need replicas to get updates when they periodically connect then yes, LiteFS could fit that use case. [1]: https://mycelial.com/ https://mycelial.com/
bigfish24 4y ago
Shameless plug but would love for you to try out Ditto[1]! CRDT-based replicated database that works peer-to-peer including automatically with WiFi and Bluetooth. [1] https://www.ditto.live https://www.ditto.live

asim 4y ago

10 years ago fly.io is the company I wanted to build. Something with massive technical depth that becomes a developer product. They're doing an incredible job and part of that comes down to how they evangelise the product outside of all the technical hackery. This requires so much continued effort. AND THEN to actually run a business on top of all that. Kudos to you guys. I struggled so much with this. Wish you nothing but continued success.

ignoramous 4y ago
5 years ago, fly.io was basically poor man's deno.com / oven.sh [0]. In my (incorrect) opinion, tptacek changed fly.io's trajectory single-handedly. [0] https://ghostarchive.org/varchive/r-1hXDvOoHA https://ghostarchive.org/varchive/r-1hXDvOoHA
- stingraycharles 4y ago
  What is @tptacek’s relation with fly.io, and how did he change it?
  - ignoramous 4y ago
    tptacek is a security researcher and principal at fly.io [0]. And as an outsider looking in, seems to have been the eng behind some of their defining features [1]: - udp support - container2vm overhaul - private networks aka 6pn - some key flyctl (cli) commands like flyctl ssh, flyctl proxy - metrics - litefs - perhaps, the imminent overhaul of the orchestration layer (?) - the upcoming authz layer [0] https://archive.is/Zwzlh https://archive.is/Zwzlh [1] https://community.fly.io/u/thomas/activity/topics https://community.fly.io/u/thomas/activity/topics | https://fly.io/blog/author/thomas/ https://fly.io/blog/author/thomas/
    - yencabulator 4y ago
      > seems to have been the eng behind some of their defining features > - litefs That glory goes to https://news.ycombinator.com/user?id=benbjohnson https://news.ycombinator.com/user?id=benbjohnson

theomega 4y ago

Does anyone else bump into the issue, that the fly.io website does not load if requested via IPv6 on Mac? I tried Safari, Chrome and curl and neither work: $ curl -v https://fly.io/blog/introducing-litefs/ * Trying 2a09:8280:1::a:791:443... * Connected to fly.io (2a09:8280:1::a:791) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer Requesting via ipv4 works $ curl -4v https://fly.io/blog/introducing-litefs/ * Trying 37.16.18.81:443... * Connected to fly.io (37.16.18.81) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (OUT), TLS handshake, Client hello (1): * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=fly.io * start date: Jul 25 11:20:01 2022 GMT * expire date: Oct 23 11:20:00 2022 GMT * subjectAltName: host "fly.io" matched cert's "fly.io" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. * Using HTTP2, server supports multiplexing * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x135011c00) > GET /blog/introducing-litefs/ HTTP/2 > Host: fly.io > user-agent: curl/7.79.1 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 32)! < HTTP/2 200 < accept-ranges: bytes < cache-control: max-age=0, private, must-revalidate < content-type: text/html < date: Wed, 21 Sep 2022 16:50:16 GMT < etag: "632b20f0-1bdc1" < fly-request-id: 01GDGFA3RPZPRDV9M3AQ3159ZK-fra < last-modified: Wed, 21 Sep 2022 14:34:24 GMT < server: Fly/51ee4ef9 (2022-09-20) < via: 1.1 fly.io, 2 fly.io < <!doctype html> ...

tptacek 4y ago
Works fine over IPv6 from this Mac.
toast0 4y ago
Could easily be a path MTU issue? I don't currently have working IPv6, but on TCP/IPv4, the server->client handshake packets I get back from fly.io are full length packets. If your tcpv6 MSS is set incorrectly, and some other not great things are happening in the path, then you might not be able to receive large packets.

ranger_danger 4y ago

ELI5?

benbjohnson 4y ago
LiteFS makes it so that you can have a SQLite database that is transparently replicated to a cluster of machines. The use case is if you have your app running on a server running in Chicago, then users in Europe will have 100ms latency to your server and users in Asia will have a 250ms latency. That's on top of the time it takes for your server to process the request. Many people target sub-100ms latency to make their applications feel snappy but that's impossible if users are waiting on their request to be sent halfway around the world. Traditionally, it's complicated to replicate your database to different regions of the world using something like Postgres. However, LiteFS aims to make it as simple as just spinning up more server instances around the world. It connects them automatically and ships changes in a transactionally safe way between them.
- boltzmann-brain 4y ago
  > Traditionally, it's complicated to replicate your database to different regions of the world using something like Postgres what makes it complicated?
twobitshifter 4y ago
It’s a tool to allow you to keep your database on the app server and backed up on s3. This means you don’t need a separate database server. Having a database on the same machine as the app is very fast, but it is usually risky. The backups made by LiteFS lessen the risk.
- hobo_mark 4y ago
  That's Litestream, not LiteFS.

pphysch 4y ago

> Developing against a relational database requires devs to watch out for "N+1" query patterns, where a query leads to a loop that leads to more queries. N+1 queries against Postgres and MySQL can be lethal to performance. Not so much for SQLite. This is misleading AFAICT. The article(s) is actually comparing remote RDBMS to local RDBMS, not Postgres to SQLite. Postgres can also be served over a UNIX socket, removing the individual query overhead due to TCP roundtrip. SQLite is a great technology, but keep in mind that you can also deploy Postgres right next to your app as well. If your app is something like a company backend that could evolve a lot and benefit from Postgres's advanced features, this may be the right choice.

tptacek 4y ago
Anyone want to chime in with the largest app they've deployed where prod Postgres was reachable over a Unix domain socket?
- pphysch 4y ago
  I would wager that the median IT shop/dept is running off a single node RDBMS, the majority of which also have the spare resources to support running the primary application (which has 1-100 DAU). No practical way to test this, of course.
- whalesalad 4y ago
  is it cheating when pgbouncer is being accessed via socket? lol
  - tptacek 4y ago
    Yes, that is cheating.
    - kjeetgill 4y ago
      I don't have any context around this: why is that cheating?

hobo_mark 4y ago

Well that was fast... [1] Are the readable replicas supposed to be long-lived (as in, I don't know, hours)? Or does consul happily converge even with ephemeral instances coming and going every few minutes (thinking of something like Cloud Run and the like, not sure if Fly works the same way)? And do they need to make a copy of the entire DB when they "boot" or do they stream pages in on demand? [1] https://news.ycombinator.com/item?id=32240230 https://news.ycombinator.com/item?id=32240230

benbjohnson 4y ago
> Are the readable replicas supposed to be long-lived Right now, yes, they should be relatively long-lived (e.g. hours). Each node keeps a full copy of the database so it's recommended to use persistent volumes with it so it doesn't need to re-snapshot on boot. We do have plans to make this work with very short-lived requests for platforms like Lambda or Vercel. That'll use transactionally-aware paging on-demand. You could even run it in a browser, although I'm not sure why you would want to. :)

whoisjuan 4y ago

Unrelated, but why does the map on their homepage show a region in Cuba? That must be wrong.

dcre 4y ago
I think it's in Miami and the icon is confusingly centered over it. The icons look like pins but I think they're supposed to be hot-air balloons. This explains why some of them appear to point to spots in the ocean. https://fly.io/docs/reference/regions/ https://fly.io/docs/reference/regions/
- whoisjuan 4y ago
  Ahh. Great catch. Their balloon definitely looks like a pin.

nicoburns 4y ago

Where is the data actually being stored in this setup? A copy on each machine running the application? If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? Also, what happens if the Consul instance goes down? If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. If it completely abstracts that away somehow then I suppose that'd be pretty cool. Currently finding it hard to get on board with the idea that adding a distributed system here actually makes things simpler.

benbjohnson 4y ago
> Where is the data actually being stored in this setup? A copy on each machine running the application? Yes, each node has a full copy of the database locally. > If so, is there another copy somewhere else (e.g. S3) in case all nodes go down? S3 replication support is coming[1]. Probably in the next month or so. Until then, it's recommended that you run a persistent volume with your nodes. > What happens if the Consul instance goes down? If Consul goes down then you'll lose write availability but your nodes will still be able to perform read queries. > If my application nodes can't be ephemeral then this seems like it would be harder to operate than Postgres or MySQL in practice. Support for pure ephemeral nodes is coming. If you're comparing LiteFS to a single node Postgres/MySQL then you're right, it's harder to operate. However, distributing out Postgres/MySQL to regions around the world and handling automatic failovers is likely more difficult to operate than LiteFS.
- omnimus 4y ago
  Will this replicate the filesystem/volume between instances? Can for example use text files or different kind of filebased database? Or it requires me to use sqlite?
  - simonw 4y ago
    This is only for SQLite. But you can put text files (or even binary files - see https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sqlite/ https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sq...) in a SQLite database and you may find you get better performance than reading files from disk: https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html

infogulch 4y ago

> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies. How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?

sk55 4y ago
They don't have ACID guarantees with this setup.
- benbjohnson 4y ago
  ACID is typically used to describe single database instances and from that perspective, LiteFS doesn't change the ACID semantics that SQLite has. For distributed consistency models[1], it gets more complicated as LiteFS currently just supports async replication. We'll introduce stronger guarantees in the future although we currently provide some information such as transaction IDs so you can manage replication lag if you need stronger consistency. [1]: https://jepsen.io/consistency https://jepsen.io/consistency
hinkley 4y ago
That's a fun one. A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction. Ultimately that's a very odd flavor of sharding. What I don't recall is how they propagated that shard information efficiently, since with sharding and consistent hashing the whole idea is that the lookup algorithm is deterministic, and therefore can be run anywhere and get the same result. WAL lag information is dynamic, so... Raft?
- simonw 4y ago
  Another solution that I've used successfully in the past for web apps is to set a 10s cookie every time a user performs a write, and then route their read requests to the lead server until that cookie expires. That way they're sure to see the impact of the write they just made.

clord 4y ago

I can imagine a database which tries to solve both of these domains. A centralized database handles consistency, and vends data closures to distributed applications for in-process querying (and those closures reconcile via something like CRDT back to the core db). Does this exist?

ignoramous 4y ago
(except for the "reconcile back" to primary) Yes? https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html https://martin.kleppmann.com/2015/11/05/database-inside-out-... See also: materialize.com, readyset.io, aws elastic-views (forever in preview).

deleted 4y ago

[deleted]

vcryan 4y ago

Currently, I am running multiple application servers (and lambda functions) using AWS Fargate that access sqlite files (databases) on an EFS share. So far so good, although my use cases are fairly simple.

vcryan 4y ago

This approach is very appealing to me :) curious about how people handle schema migrations when using this approach. I segment sqlite files (databases) that have the same schema into the same folder. I haven't really had a case where migrations was really a concern, but I could see it happening soon. Seems like in my deployment, I'm going to need an approach to loop over dbs to apply this change... I currently have a step of app deployment that attempts to apply migrations... but it is more simplistic because the primary RDBMS (postgresql) just appears to the application as a single entity which is the normative use-case for db-migrate-runners.

simonw 4y ago
Yeah schema migrations are going to be interesting. Presumably those end up making a HUGE change to the WAL (since they could affect every stored row in a large table), which means LiteFS then has to send a truly giant chunk of data out to the replicas. I wonder how this would present itself in real-world usage? Would the replicas go out-of-date for 10-30s but continue serving read-only traffic, or could there be some element of cluster downtime caused by this?
- benbjohnson 4y ago
  > Presumably those end up making a HUGE change to the WAL (since they could affect every stored row in a large table), which means LiteFS then has to send a truly giant chunk of data out to the replicas. Yes, a large migration could end up sending a lot of data out to replicas. One way to mitigate this is to shard your data into separate SQLite databases so you're only migrating a small subset at a time. Of course, that doesn't work for every application. > Would the replicas go out-of-date for 10-30s but continue serving read-only traffic, or could there be some element of cluster downtime caused by this? Once WAL support is in LiteFS, it will be able to replicate this out without causing any read down time. SQLite is still a single-writer database though so writes would be blocked during a migration.

jensneuse 4y ago

Sound like a drop in solution to add high availability to WunderBase (https://github.com/wundergraph/wunderbase https://github.com/wundergraph/wunderbase). Can we combine LiteFS with Litestream for Backups, or how would you do HA + Backups together?

benbjohnson 4y ago
Yes, it should fit with WunderBase to provide HA. I'm adding S3 support which will work the same (in principle) as Litestream. I'm hoping to have that in about a month or so.
- rsync 4y ago
  Can you comment on how you added SFTP support to litestream so that it would work with rsync.net[1][2] ? How does that compare/contrast with what my grandparent is alluding to ? [1] https://github.com/benbjohnson/litestream/issues/140 https://github.com/benbjohnson/litestream/issues/140 [2] https://www.rsync.net/resources/notes/2021-q3-rsync.net_technotes.html https://www.rsync.net/resources/notes/2021-q3-rsync.net_tech...
  - benbjohnson 4y ago
    Litestream does log shipping so it just takes a chunk of the write-ahead log (WAL) and copies it out a destination. That can be S3, GCP, SFTP, etc. LiteFS will do the same although I'm only targeting S3 initially. That seemed to be the bulk of what people used.
    - rsync 4y ago
      OK, thank you. As with the litestream component, it would be wonderful to have an SFTP transport for LiteFS. When you say 'S3' I assume you mean "S3 compatible API" so that would be fairly open and portable but SFTP would be even more so. Thanks again.

endisneigh 4y ago

Seems neat, until you try to do schema migrations. Unless they can guarantee that all containers’ SQLite instances have the same scheme without locking I’m not sure how doesn’t run into the same issues as many NoSQL. CouchDB had this same issue with its database per user model and eventually consistent writes.

yencabulator 4y ago
I don't see how this is different to a world where you use Postgres over the network, with lots of DB clients. You "just" do the schema migrations like everyone who can't shut down the business does: incrementally and in a backwards-compatible manner. After the migration, deploy new code that takes advantage of the new schema. Repeat until done.
slively 4y ago
In my experience distributed database migrations tend to be two phase though so it can be compatible during the migration. The first migration to add something, release the apps to use it, then a second migration to cleanup.

fny 4y ago

This reads like a professor who's so steeped in research that he's forgotten how to communicate to his students! What exactly are we talking about here? A WebSQL thats actually synced to a proper RDBMS? Synced across devices? I'm not clear about an end to end use case. Edit: Honestly, this line from the LiteFS docs[0] needs to be added to the top of the article: > LiteFS is a distributed file system that transparently replicates SQLite databases. This lets you run your application like it's running against a local on-disk SQLite database but behind the scenes the database is replicated to all the nodes in your cluster. This lets you run your database right next to your application on the edge. I had no idea what was being talked about otherwise. [0]: https://fly.io/docs/litefs/ https://fly.io/docs/litefs/

hinkley 4y ago

> Second, your application can only serve requests from that one server. If you fired up your server in Dallas then that'll be snappy for Texans. But your users in Chennai will be cursing your sluggish response times since there's a 250ms ping time between Texas & India. > To improve availability, it uses leases to determine the primary node in your cluster. By default, it uses Hashicorp's Consul. Having a satellite office become leader of a cluster is one of the classic blunders in distributed computing. There are variants of Raft where you can have quorum members that won't nominate themselves for election, but out of the box this is a bad plan. If you have a Dallas, Chennai, Chicago, and Cleveland office and Dallas goes dark (ie, the tunnel gets fucked up for the fifth time this year), you want Chicago to become the leader, Cleveland if you're desperate. But if Chennai gets elected then everyone has a bad time, including Dallas when it comes back online.

benbjohnson 4y ago
Good catch. LiteFS has an option to mark nodes as candidates or not. For example, you can run a cluster with a couple candidate nodes in one region (e.g Chicago) and every other region may just run a single non-electable replica. We don’t support any kind of tiering for candidates in different regions. That’s not a bad idea though.

azlyrics 4y ago

I actually gave fly.io a whirl over the weekend. was not fun. spent a lot of time on the forums and its pretty clear it has some way to go before it can give AWS or Linode a run for their money. For instance, we run kubernetes on multiple VPS providers, including public clouds with serverless onramp/offramps deployed on edge location. Anything under 15 minutes are processed by serverless. Anything longer is offloaded to one of the VPS containers available in every part of the world. I have some more feedbacks ready if you are interested, its a neat idea but not exactly as seamless and easy as the idea proposed since public clouds already offer a way to do this.

morenoh149 4y ago

still wondering why projects like https://www.reactivated.io/ https://www.reactivated.io/ are using fly.io

Existenceblinks 4y ago

The pain is that this approach is suitable on VPS/IaaS where disk volume is supported. As a solo dev, I only use PaaS kind of infra, there are just a few PaaS i'm aware of that support attachable disk. Fly, Render, .. nothing else?

presentation 4y ago

My dream would be if this supported geo-partitioning. In my field people are pretty sensitive about GDPR so would love to box in EU PII in EU servers.

yencabulator 4y ago
Since the scope is a SQLite database, that means you just run a EU database, with no replicas outside of EU. This level of database replication isn't going to put individual rows in different geographies.