16 ms·
Litestream v0.5.0
- deleted 1y ago[deleted]
- dave78 1y agoInteresting tidbit regarding LiteFS/Litestream: > But the market has spoken! Users prefer Litestream. And honestly, we get it: Litestream is easier to run and to reason about. So we’ve shifted our focus back to it.
- simonw 1y agoThat makes sense to me. LiteFS used FUSE, which meant figuring out how to run and mount a custom filesystem. Litestream is a single compiled Go binary that you point at the SQLite database file (and accompanying WAL file).
- thewisenerd 1y agorelevant: Litestream: Revamped (99 comments) https://fly.io/blog/litestream-revamped/ https://fly.io/blog/litestream-revamped/ https://news.ycombinator.com/item?id=44045292 https://news.ycombinator.com/item?id=44045292
- danw1979 1y agoCan’t wait for Kyle to set Jepsen loose on this.
- tptacek 1y agoLitestream is pretty boring for the testing Kyle does. It's not a distributed consensus system, but something more like a storage engine.
- Aurornis 1y agoJepsen is primarily for testing distributed systems and consensus systems. SQLite has neither of those.
- dastbe 1y agolitestream makes very few consistency guarantees compared to other datastores, and so I would expect most any issues found would be "working as intended". at the end of the day with litestream, when you respond back to a client with a successful write you are only guaranteeing a replication factor of 1.
- simonw 1y agoBy "replication factor of 1" you mean your data is stored on local disk only, right? That matches my understanding: Litestream replication is asynchronous, so there's usually a gap of a seconds or two between your write being accepted and the resulting updated page being pushed off to S3 or similar.
- dastbe 1y agoYes. the acknowledgement you're getting in your application code is that the data was persisted in sqlite on that host. There's no mechanism to delay acknowledgement until the write has been asynchronously persisted elsewhere.
- simonw 1y agoAn interesting comparison here is Cloudflare's Durable Objects, which provide a blocking SQLite write that doesn't return until the data has been written to multiple network replicas: https://blog.cloudflare.com/sqlite-in-durable-objects/#trick-two-relay-through-other-servers-in-our-global-network https://blog.cloudflare.com/sqlite-in-durable-objects/#trick...
- simonw 1y agoI wonder if it would be possible to achieve this using a SQLite VFS extension - maybe that could block acknowledgment of a right until the underlying page has been written to S3?
- sighansen 1y agoReally love litestream. Easy to use and never crashes on me. I still recommend using it as a systemd unit service. I'm not only using it as a backup tool but also to mirror databases. Looking forward to their read-replica feature.
- TemptedMuse 1y agoMaybe I misunderstand what this is, but why would I use this and not MySQL, Postgres, or any other proper database? Seems like a hack to get SQLite to do what those do by design.
- victorbjorklund 1y agoWhy use Postgres if all you need is sqlite? Postgres is way overkill for a simple app with few users and no advanced database functionality.
- TemptedMuse 1y agoI'd argue that anything larger than a desktop app should not use SQLite. If you need Litestream for replication and backup it is probably better to just use Postgres. There are a ton of one-click deployment offerings for proper databases, Fly.io actually offers managed Postgres.
- victorbjorklund 1y agoWhy would you argue that? Do you have some benchmarks backing it up or is it more a personal preference?
- crazygringo 1y agoIt's literally what they're designed for. SQLite is designed for one local client at a time. Client-server relational databases are designed for many clients at a time.
- simonw 1y agoThat's not entirely true. SQLite is designed to support many processes reading the same file on disk at once. It only allows one process to write at a time, using locks - but since most writes finish in less than a ms in most cases having a process wait until another process finishes their write isn't actually a problem. If you have lots of concurrent writes SQLite isn't the right solution. For concurrent reads it's fine. SQLite also isn't a network database out-of-the-box. If you want to be able to access it over the network you need to solve that separately. (Don't try and use NFS. https://sqlite.org/howtocorrupt.html#_filesystems_with_broken_or_missing_lock_implementations https://sqlite.org/howtocorrupt.html#_filesystems_with_broke... )
- lordofmoria 1y agoA small warning for folks. I once was responsible for migrating a legacy business app to Azure, and the app had a local MSSQL server co-running with the app (the same pattern that Litestream is using). As have been mentioned below, the app had been developed assuming the local access (and thus <1ms latency), so it had a ton of N+1 everywhere. This made it almost impossible to migrate/transition to another configuration. So, if this style of app hosting doesn't take off and you're at all worried about this being a dead end storage once you reach a certain scale, I'd recommend not doing this, otherwise your options will be very limited. Then again - I bet you could get very very far on a single box, so maybe it'd be a non factor! :)
- inerte 1y agoI used to work on a product where the app server and database were in the same rack - so similar low latency. But the product was successful, so our N+1 would generate thousands of queries and 1ms would become >500ms or more easily. Every other month we would look at New Relic and find some slow spot. It was a Rails app, therefore easy to get into the N+1 but also somewhat easy to fix.
- 1-more 1y agoFor our rails app we actually added tests asserting no N+1s in our controller tests. Think a test setup with 1 post vs 10 posts (via factorybot) and you could do an assertion that the DB query count was not different between the two. A useful technique for any Railsheads reading this!
- simonw 1y agoThat's a good trick. In Django world I like pytest-django's django_assert_max_num_queries fixture: https://pytest-django.readthedocs.io/en/latest/helpers.html#django-assert-max-num-queries https://pytest-django.readthedocs.io/en/latest/helpers.html#... def test_homepage_queries(django_assert_max_num_queries, client): with django_assert_max_num_queries(10): assert client.get("/").status_code == 200 Or django_assert_num_queries to assert an exact number.
- mtlynch 1y agoVery excited to see Fly restart development on Litestream after a 2ish year freeze! I love Litestream and use it in every app I build now. They advertise it as costing "pennies per day," but it's even less expensive than that. It obviously varies depending on how much storage you need, but I had a real app in production, and Litestream replication to S3 only cost me 2-3 cents ($0.02-$0.03) per month.[0] [0] https://mtlynch.io/litestream/#using-logpaste-in-production https://mtlynch.io/litestream/#using-logpaste-in-production
- atombender 1y agoWhat I'd like to see is a system where a single-writer SQLite database is replicated to object storage so that you can spin up really cheap read replicas. Is anyone working on something like that? Such a system would also require a side channel propagating WAL updates (over Kafka or similar) to replicas, so that the running replicas can update themselves incrementally and stay fresh without loading anything from object storage.
- anentropic 1y agoLitestream is basically that, though the OP article has "The next major feature we’re building out is a Litestream VFS for read replicas" as a What's Next todo at the bottom. https://litestream.io/guides/s3/ https://litestream.io/guides/s3/ I think this is also roughly what Turso is, although it's becoming a SQLite-compatible db rather than vanilla https://docs.turso.tech/features/embedded-replicas/introduction https://docs.turso.tech/features/embedded-replicas/introduct... https://docs.turso.tech/cloud/durability https://docs.turso.tech/cloud/durability
- atombender 1y agoI don't Litestream does that yet? It appears to be for backing up to S3, and you manually "restore" the image to a file. You can't point an SQLite client at the S3 bucket, and there's no provision for getting low latency updates. But it sounds like they're working on this. Turso looks cool and is a project I will keep an eye on, but it's replica mode seems to be designed for apps rather than mechanism to rapidly scale out read replicas in a server cluster. Also, the web site warns that it's not ready for production use.
- simonw 1y agoLiteFS can do that, but you need to run a custom FUSE filesystem for it - hence why Litestream remained more popular. https://fly.io/docs/litefs/how-it-works/#capturing-sqlite-transactions https://fly.io/docs/litefs/how-it-works/#capturing-sqlite-tr... Litestream is working on that now - the code is already in https://github.com/benbjohnson/litestream/blob/v0.5.0/vfs.go https://github.com/benbjohnson/litestream/blob/v0.5.0/vfs.go but it's not yet a working, documented feature.
- mmaunder 1y agoIs this a transcript?
- svat 1y agoI don't even care about the topic (much) (right now), but this is a really well written blog post / announcement.
- JaggerFoo 1y agoInteresting information that they chose modernc.org/sqlite over mattn/go-sqlite3 as a Quality-of-Life improvement. Going forward I guess I'll do the same for new projects. The NATS Jetstream use case is something I'm curious about. Cheers and keep up the great work on Litestream.
- threemux 1y agoAll the public benchmarks (and my own applications) indicate that there is only a small (often unnoticeable) performance penalty associated with modernc.org/sqlite and this is far outweighed by the ability to eliminate CGO. I'd use it on future projects without hesitation.
- wodenokoto 1y agoWe have an in house application installed on a remote fleet with spotty internet access. Because of the spotty internet we have a lot of trouble setting up a reliable system for getting data home. Could how does litestream handle backing up through a spotty connection and can we consolidate the backups into a central db an query against it?
- keeganpoppen 1y agoi am not a big fly.io fan per se, but their blog posts, especially in this sort of sphere, is delightful, and i absolutely commend them for this. fly has a fascinating combination of “dev-forward” and “we made this complicated because it shits us” that continues to confound me, but i must confess i am a big fan of this project, and a number of other projects (e.g. svelte) that they seem to genuinely treat in an arms-length, “let a thousand flowers bloom” sort of way.
- taejavu 1y agoI’m curious what you don’t like about fly.io. Also what’s the connection to svelte?
- tptacek 1y agoJust so we're all clear: Litestream is just an open source project, not a feature of our platform. This isn't about Fly.io.
- keeganpoppen 1y agoi hope that came across in what i was saying too. it was intended purely in a "i am not even a fanboy but this thing they do is really cool" sense.
- keeganpoppen 1y agothey hired some of the svelte people on top of the react people they hired... "embrace, extend, extinguish", for the cynically-bent. i personally wouldn't go anywhere near that far-- i just haven't "dug" the platform the times i have considered it, personally. i wish them nothing but success.
- barefootford 1y agoI look forward to trying this out. Any benchmarks or demos on how long it actually takes to restore? I ended up cooking my own boring S3 backup because previously litestream took 20 minutes to restore something like 1000 rows. It felt extremely unoptimized. How long does restoration take today?
- umajho 1y agoIt seems Litestream will soon support arbitrary s3-compatible destinations.[^1] Neat. So far I’ve stuck with the SFTP solution, since I don’t use any of the cloud object storage services that are hardcoded into the tool.[^2] Big thanks to the developers. [^1]: https://github.com/benbjohnson/litestream/pull/731 https://github.com/benbjohnson/litestream/pull/731 [^2]: https://litestream.io/guides/#replica-guides https://litestream.io/guides/#replica-guides
- geenat 1y agoAdvantages of Litestream over https://sqlite.org/rsync.html https://sqlite.org/rsync.html ?
- thelibrarian 1y agoLitestream gives you point in time recovery - you can restore to any snapshot time, not just have a current replica.
- ncruces 1y agoThe other advantage (or difference) is that you don't need a “server” on the other end, just object storage. Which may come out cheaper.
- gizzlon 1y agoCould you use it locally and the upload the result to object storage? I would probably run both litestream and full backups, to get extra safety
- ncruces 1y agoSure. But you'll be transferring the entire file everytime. The advantage of sqlite3_rsync is that, if on the other end is an SSH server with sqlite3_rsync, you only transfer the changed pages, and still get a “perfect” copy of the file on the other end. The advantage of Litestream is that on the other end does not need to live a “server,” and still only changes are uploaded. If you do it continuously, you get many points to recover from, and many of those cross reference each other, saving storage space too. On the flip side, you need the tool to restore.
- kevincox 1y agoFWIW. We saw this and I started experimenting. A simple case of copying our production database locally a few times. About 1 in 4 times the local database was corrupted after running it. I also experienced very strange command line parsing bugs and different behaviour depending on the name of the local database file. So I would treat sqlite3_rsync as more of a demo than a stable product right now. Litestream provides near-real-time offsite replication and point in time recovery, which sqlite3_rsync won't do on its own. You could probably build a litestream-like product based on sqlite3_rsync but it probably won't be as fast or as efficient on storage.
- wg0 1y agoWould it be a good fit for a application with the user base and size of let's say Fresh books?
- bradgessler 1y agoThe DX for deploying SQLite apps to Fly.io is rough. I'm a few hours into trying to get a production Rails app booting, but running into issues getting the database to initialize, migrate, and become writable. The root of my problem was the eager loading of a gem I wrote, but there were several layers of runners above it that made it hard to diagnose. I wish they'd put a bit more effort into the DX here, but it probably doesn't make much sense from a biz PoV since big customers aren't going to be running these kinds workloads. Curious if anybody here is deploying SQLite apps to production and what host they're using?
- Lord_Zero 1y agoI use in production but it's a console app that lives on a server. The database sits on a file share.
- nop_slide 1y agoI setup a fresh rails 8 app on Fly last year, using PG for the main data store but using SQLite for the ancillary solid stack dbs. Only fuss I remember encountering was with fighting with rails migrating solid queue properly, but this seemed like a rails unit issue and don’t remember it being a Fly issue. I’ve been contemplating migrating my pg primary to SQLite too. Anyways don’t have much else to offer other than an anecdote that I’m happily using fly with partial SQLite.
- hoppp 1y agoI self host on a vps Litestream works for that quite well
- mtlynch 1y ago>The DX for deploying SQLite apps to Fly.io is rough. I'm a few hours into trying to get a production Rails app booting, but running into issues getting the database to initialize, migrate, and become writable. The root of my problem was the eager loading of a gem I wrote, but there were several layers of runners above it that made it hard to diagnose. What's the Fly.io issue here? Aren't the issues you're describing in Rails not Fly.io? I run several Go apps in production on Fly.io[0, 1, 2] and I've never had an issue with the Fly.io + SQLite part of it. SQLite + Litestream makes things slightly more complicated, but again, I've never had issues with Fly.io specifically making that harder. I use a custom launch script in my Dockerfile that starts litestream with the -exec flag to start my app as a child process.[3] [0] https://github.com/mtlynch/logpaste https://github.com/mtlynch/logpaste [1] https://github.com/mtlynch/picoshare https://github.com/mtlynch/picoshare [2] https://github.com/mtlynch/screenjournal https://github.com/mtlynch/screenjournal [3] https://github.com/mtlynch/logpaste/blob/0.3.1/docker-entrypoint#L30-L50 https://github.com/mtlynch/logpaste/blob/0.3.1/docker-entryp...
- sealeck 1y agoIn the previous post on the Fly.io blog (https://fly.io/blog/litestream-revamped/ https://fly.io/blog/litestream-revamped/), a really cool thing was mentioned! > In Litestream, we’re solving the problem a different way. Modern object stores like S3 and Tigris solve this problem for us: they now offer conditional write support. With conditional writes, we can implement a time-based lease. We get essentially the same constraint Consul gave us, but without having to think about it or set up a dependency. Reading this blog post though, I couldn't see any reference to this. Is this supported in Litestream v0.5.0, or will it be for a future release?
- grahar64 1y agoFUSE is such a cool idea but every application always suffers performance problems when stressed. I tried LiteFS and it worked, but litestream was awesome.
- JSR_FDED 1y agoQuestion about using Litestream - I have an app running, and now need to upgrade it to a newer version. Currently I build the new version in a different directory, stop the old app, copy/migrate the database from the old version to the new version (in the new directory). Then finally I delete the old directory, and rename the new directory to be the production one. Will Litestream freak out about the database being replaced underneath it? Will I still be able to restore old versions of the DB?