5 ms·

Litestream Eliminated My Database Server for $0.03/Month

92 points by mtlynch 6y ago

mtlynch 6y ago
Author here. I discovered Litestream here on HN after benbjohnson shared it here a few months back.[0] I wasn't interested at first because I don't ever use SQLite, but it stuck with me. I eventually realized it's a great way to achieve vendor flexibility in hosting apps, so I used Litestream to build a pastebin-style service for my app and wanted to share what I learned. Happy to answer any questions or hear any feedback about this post. [0] https://news.ycombinator.com/item?id=26103776 https://news.ycombinator.com/item?id=26103776
- amzans 6y ago
  Just wanted to say thank you for writing this Michael. You've been putting lots of great content on your blog! Really enjoying the real world examples you include.
  - mtlynch 6y ago
    That's great to hear. Thanks for reading and for the kind words!
- benbjohnson 6y ago
  Litestream author here. Great explanation of how everything works! That's awesome that's only running at $0.03/month. :)
  - mtlynch 6y ago
    Hey Ben! Thanks again for your work on Litestream and for your help with this post. I'm excited to see Litestream continue growing in popularity.
- jfengel 6y ago
  Question from a newbie to cloud stuff: what is the performance like for a database server that's separated from your host like that? In terms of latency alone it sounds like it risks adding tens or even hundreds of milliseconds to even a fairly trivial request: out of my datacenter, across the wire, into theirs, grab the data, and then reverse the path. And perhaps a charge for data going out each way. Am I wrong about that? Am I overestimating the travel time, or perhaps they end up being located in the same datacenter so it's all high speed?
  - benbjohnson 6y ago
    It's typical to see at least millisecond latency for a database server that's located in the same data center (e.g. hosting your own Postgres) and tens or hundreds of milliseconds for cloud database services that run outside your data center. The OP's approach of SQLite & Litestream removes that latency entirely since the database is colocated with the application. Litestream will stream backups to an external service (e.g. S3) and that can have tens or hundreds of milliseconds of latency but that won't affect the application's performance since it is asynchronous replication.
    - jfengel 6y ago
      Ah, I had misread how that worked. That makes a ton of sense. Thank you.
  - webmobdev 6y ago
    what is the performance like for a database server that's separated from your host like that? It's not separated. As this diagram shows - https://mtlynch.io/litestream/diagram.jpg https://mtlynch.io/litestream/diagram.jpg - the SQLite database is on the same server as the web application. Additionally, OP's setup seems to use Litestream to backup the SQLite database to some cloud storage.
tdeck 6y ago
Interesting setup! Does this mean you've got to download your entire DB on every deploy?
- mtlynch 6y ago
  Thanks for reading! Yep, you do have to download the DB on every deploy. It's negligible for my deploys because my database is a few MB, but if it got to be in the GB range, I'd have to find some way around that.
  - tdeck 6y ago
    Is it possible to lose records written between the start of the deploy/DB download and the switchover to the new image?
    - benbjohnson 6y ago
      Litestream will flush outstanding changes on a clean shutdown so you shouldn’t lose any data when deploying.
orf 6y ago
> It’s also important to note that Litestream can’t resolve conflicts between multiple database writes, so each database can have only one application server with write access. It’s cool, don’t get me wrong, but given the above constraint isn’t it just: while True: sleep(1) upload_file(“sqlite.db”) with extra bells and whistles?
- mtlynch 6y ago
  Thanks for reading! Litestream is doing more magic than it seems, so it's much better than a naive while loop. Litestream is only streaming the changes in the database, whereas the naive implementation would have to upload the entire database every second. Also, SQLite has a write-ahead log, so you're not guaranteed that all the data is always in the main .db file. Litestream watches the write-ahead log files as well and immediately streams those changes to cloud storage. A system that only watches the main .db file would have long time windows where data loss is possible.
based2 6y ago
https://dqlite.io/ https://dqlite.io/ High-availability SQLite
- benbjohnson 6y ago
  dqlite and rqlite are both great projects but target different use cases. Those run Raft consensus which requires at least 3 nodes for HA whereas Litestream is meant to run on a single node and trades off with lower durability guarantees.
  - atat7024 6y ago
    Could you elaborate on the durability? Would a multi tenant SaaS with databases on the much smaller size compared to what you all talk about do well to use this, with proper backups?
    - benbjohnson 6y ago
      Litestream flushes out changes to S3 on an interval. By default, it’s every 10 seconds although you could reasonably drop that down to 1 second. You could go lower if you’re replicating to NFS. That interval determines your window for data loss in the event of a catastrophic failure (e.g. disk failure). I’ve run multiple DigitalOcean machines for years and have never had a catastrophic failure so they are rare events but they can happen. Dqlite and rqlite run every write through a distributed consensus algorithm which guarantees that your writes are persisted across at least a majority of nodes in your cluster before returning a success. In that situation, you’d need a catastrophic failure on a majority of nodes to lose data, however, there is generally a large performance trade off when running distributed consensus.
  - beagle3 6y ago
    Is it possible to use dqlite/rqlite for high availability in the same datacenter, while shipping logs using litestream to a different data center for diaster recovery? Or are they incompatible? p.s. Thanks for litestream!
croes 6y ago
Misleading title. First thought that Litestream had deleted the database or that the service was discontinued for the price. Better would be "How Litestream replaces a database server for 0.03ct/month" or "Litestream eliminates the need for a Database Server for 0.03ct/month".
avinassh 6y ago
> No, my app never talks to a remote database server. isn't this wrong? The S3 may not be the remote DB, but it is sorta is, in this case.
- iudqnolq 6y ago
  It's not the app talking to it though. The app talks to sqlite, and separately litestream streams the deltas in the sqlite db to s3. The major thing is the app doesn't have to be coded with s3's latency in mind when it accesses the db.
flagZ 6y ago
Cool little project. If lowest cost is the goal, I am wondering if you could do it too by not having neither sqlite and litestream, and store each log as a separate object on S3.

92 points by mtlynch 6y ago

Author here. I discovered Litestream here on HN after benbjohnson shared it here a few months back.[0] I wasn't interested at first because I don't ever use SQLite, but it stuck with me. I eventually realized it's a great way to achieve vendor flexibility in hosting apps, so I used Litestream to build a pastebin-style service for my app and wanted to share what I learned. Happy to answer any questions or hear any feedback about this post. [0] https://news.ycombinator.com/item?id=26103776 https://news.ycombinator.com/item?id=26103776

amzans 6y ago
Just wanted to say thank you for writing this Michael. You've been putting lots of great content on your blog! Really enjoying the real world examples you include.
- mtlynch 6y ago
  That's great to hear. Thanks for reading and for the kind words!
benbjohnson 6y ago
Litestream author here. Great explanation of how everything works! That's awesome that's only running at $0.03/month. :)
- mtlynch 6y ago
  Hey Ben! Thanks again for your work on Litestream and for your help with this post. I'm excited to see Litestream continue growing in popularity.
jfengel 6y ago
Question from a newbie to cloud stuff: what is the performance like for a database server that's separated from your host like that? In terms of latency alone it sounds like it risks adding tens or even hundreds of milliseconds to even a fairly trivial request: out of my datacenter, across the wire, into theirs, grab the data, and then reverse the path. And perhaps a charge for data going out each way. Am I wrong about that? Am I overestimating the travel time, or perhaps they end up being located in the same datacenter so it's all high speed?
- benbjohnson 6y ago
  It's typical to see at least millisecond latency for a database server that's located in the same data center (e.g. hosting your own Postgres) and tens or hundreds of milliseconds for cloud database services that run outside your data center. The OP's approach of SQLite & Litestream removes that latency entirely since the database is colocated with the application. Litestream will stream backups to an external service (e.g. S3) and that can have tens or hundreds of milliseconds of latency but that won't affect the application's performance since it is asynchronous replication.
  - jfengel 6y ago
    Ah, I had misread how that worked. That makes a ton of sense. Thank you.
- webmobdev 6y ago
  what is the performance like for a database server that's separated from your host like that? It's not separated. As this diagram shows - https://mtlynch.io/litestream/diagram.jpg https://mtlynch.io/litestream/diagram.jpg - the SQLite database is on the same server as the web application. Additionally, OP's setup seems to use Litestream to backup the SQLite database to some cloud storage.

tdeck 6y ago

Interesting setup! Does this mean you've got to download your entire DB on every deploy?

mtlynch 6y ago
Thanks for reading! Yep, you do have to download the DB on every deploy. It's negligible for my deploys because my database is a few MB, but if it got to be in the GB range, I'd have to find some way around that.
- tdeck 6y ago
  Is it possible to lose records written between the start of the deploy/DB download and the switchover to the new image?
  - benbjohnson 6y ago
    Litestream will flush outstanding changes on a clean shutdown so you shouldn’t lose any data when deploying.

orf 6y ago

> It’s also important to note that Litestream can’t resolve conflicts between multiple database writes, so each database can have only one application server with write access. It’s cool, don’t get me wrong, but given the above constraint isn’t it just: while True: sleep(1) upload_file(“sqlite.db”) with extra bells and whistles?

mtlynch 6y ago
Thanks for reading! Litestream is doing more magic than it seems, so it's much better than a naive while loop. Litestream is only streaming the changes in the database, whereas the naive implementation would have to upload the entire database every second. Also, SQLite has a write-ahead log, so you're not guaranteed that all the data is always in the main .db file. Litestream watches the write-ahead log files as well and immediately streams those changes to cloud storage. A system that only watches the main .db file would have long time windows where data loss is possible.

based2 6y ago

https://dqlite.io/ https://dqlite.io/ High-availability SQLite

benbjohnson 6y ago
dqlite and rqlite are both great projects but target different use cases. Those run Raft consensus which requires at least 3 nodes for HA whereas Litestream is meant to run on a single node and trades off with lower durability guarantees.
- atat7024 6y ago
  Could you elaborate on the durability? Would a multi tenant SaaS with databases on the much smaller size compared to what you all talk about do well to use this, with proper backups?
  - benbjohnson 6y ago
    Litestream flushes out changes to S3 on an interval. By default, it’s every 10 seconds although you could reasonably drop that down to 1 second. You could go lower if you’re replicating to NFS. That interval determines your window for data loss in the event of a catastrophic failure (e.g. disk failure). I’ve run multiple DigitalOcean machines for years and have never had a catastrophic failure so they are rare events but they can happen. Dqlite and rqlite run every write through a distributed consensus algorithm which guarantees that your writes are persisted across at least a majority of nodes in your cluster before returning a success. In that situation, you’d need a catastrophic failure on a majority of nodes to lose data, however, there is generally a large performance trade off when running distributed consensus.
- beagle3 6y ago
  Is it possible to use dqlite/rqlite for high availability in the same datacenter, while shipping logs using litestream to a different data center for diaster recovery? Or are they incompatible? p.s. Thanks for litestream!

croes 6y ago

Misleading title. First thought that Litestream had deleted the database or that the service was discontinued for the price. Better would be "How Litestream replaces a database server for 0.03ct/month" or "Litestream eliminates the need for a Database Server for 0.03ct/month".

avinassh 6y ago

> No, my app never talks to a remote database server. isn't this wrong? The S3 may not be the remote DB, but it is sorta is, in this case.

iudqnolq 6y ago
It's not the app talking to it though. The app talks to sqlite, and separately litestream streams the deltas in the sqlite db to s3. The major thing is the app doesn't have to be coded with s3's latency in mind when it accesses the db.

flagZ 6y ago

Cool little project. If lowest cost is the goal, I am wondering if you could do it too by not having neither sqlite and litestream, and store each log as a separate object on S3.