6 ms·

You won't gain much by the combination of golang, sqlite and the cloud. You can't scale out and the cloud layer only makes sqlite slower. I can't see any reason

You won't gain much by the combination of golang, sqlite and the cloud. You can't scale out and the cloud layer only makes sqlite slower. I can't see any reason not using RDS other than it's more expensive. I would only say this is a quick way to run up an application with a cheap VPS for a beginner.

benbjohnson 4y ago
Litestream/LiteFS author here. I agree that "cloud" is a bit ambiguous but Go & SQLite are quite powerful together and I don't think it's only for beginners. Both are fast and have low overhead. In addition to lower cost versus RDS, there's near-zero query latency which eliminates a lot of performance problems. You can comfortably run tens or hundreds of requests per second on minimal hardware (e.g. 256MB or 512MB instances). There's a lot of room for scaling up before you hit a performance ceiling.
- ehutch79 4y ago
  Tens of requests a second?
  - randomdata 4y ago
    He's being generous. Indeed, your service will be lucky if it sees tens of requests per day.
    - ehutch79 4y ago
      :-| Normally I'm on the side of 'do you actually get that much traffic?' but yeah, a dashboard view can generate dozens of requests alone, each with multiple queries.
  - benbjohnson 4y ago
    A request can have a large range of queries within it so I was trying to account for that. If your requests are lightweight and mostly reads, you can do 1,000+ req/sec on a 256MB instance. YMMV.
- tacitusarc 4y ago
  Do you mean tens or hundreds of thousands of requests per second?
  - benbjohnson 4y ago
    There aren’t many apps that have 100,000+ req/sec and you certainly can’t run them on modest hardware. If your app averages 100 req/sec then that’s 8.4M requests per day. That’s more than most applications out there.
- msolberg 4y ago
  I just watched your gophercon video on SQLite in production after seeing it in the article. Great talk. Anyone else who's interested can watch here: https://www.youtube.com/watch?v=XcAYkriuQ1o https://www.youtube.com/watch?v=XcAYkriuQ1o
  - benbjohnson 4y ago
    Thanks!
markusw 4y ago
See my followup-article "Distributed SQLite with LiteFS" for the scale-out part: https://www.golang.dk/articles/distributed-sqlite-with-litefs https://www.golang.dk/articles/distributed-sqlite-with-litef... And IMO, what you lose in the cloud layer you gain by having a deployment really close to your user. Speed of light and all that. Plus SQLite is super cool and fun! :D
simonw 4y ago
At what point would you expect to need to scale out?
infamia 4y ago
Once you need to graduate from one large server (which will take you a long way in many cases), there are tools like [rqlite](https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite) that can handle clustering. With WAL mode enabled, SQLite can handle a surprising amount of traffic that would fit a lot of use cases. If latency is important to you, it's going to be hard to beat SQLite for many workloads.
randomdata 4y ago
You gain low latency, which allows you to write your code to a 'pure' relational model (at least as close as SQL allows), without having to rely on code complicating hacks to deal with round-trip issues. Less complicated code can improve delivery time and reduce faults. While there are abstractions that can be used to help with those hacks, they come with their own tradeoffs. Pick your poison, as always. Conceptually, RDS is little more than SQLite with a clever networking layer built on top. In context, your application is also just a clever networking layer, so the middleman doesn't necessarily add any value. Of course it depends on exactly what you are trying to do and what tradeoffs you are willing to accept. There is no free lunch.
ketralnis 4y ago
This is a pretty strong claim without any numbers, to be honest. You were probably already running your single database instance so being limited to a single sqlite instance isn't terribly different. This does limit you to a single app server, which unless you're CPU bound is also fine. Even if you are, you can get a lot of cores in a single instance these days. Most things will never need to be scaled up
- aynyc 4y ago
  AFAIK, sqlite3 doesn't support multi-core, so you can only do vertical scaling. In cloud, people rather do horizontal scaling. That being said, I ran a django based webapp for internal users and we saw core being pegged by sqlite3 when the queries were complex. However, under normal usage, it was fine for about 100 concurrent users including machine API calls.
  - cldellow 4y ago
    > doesn't support multi-core Can you expand on that? I thought it supported multiple readers, but only a single writer. They do all have to be on the same host, though. Since you mentioned Django - I think there are some complexities where the Python driver can't really do concurrent access if using threads instead of processes, but this is due to Python/GIL limitations, not sqlite limitations.
    - aynyc 4y ago
      sqlite3 is a single core application, so even if you have multiple cores on a box, sqlite3 isn't going to take advantage of that. If you want sqlite3 to go faster, you need better single core box. That being said, I do believe sqlite3 allow multiple processes to read the same database file. Our django app sits behind Gunicorn that will spin up multiple django instances, as long as Sqlite3 has WAL mode on, concurrent access (read) isn't a problem. For write, we basically use a lock.
      - markusw 4y ago
0cf8612b2e1e 4y ago
> I can't see any reason not using RDS other than it's more expensive. We don’t all operate with unlimited VC money.
iveqy 4y ago
Depends on your application. On the opposite I've seen many cases where sqlite3 scales better than postgres, depending on the scale and the application. If you're doing horizontal scaling, sqlite3 is a very good alternative.
mariusor 4y ago
It's a little weird that you consider bringing up an application on bare metal something for a beginner. :) I think manually handling the nitty-gritty of an OPS setup is a more senior task than clicking you way into setting up some AWS infrastructure.

benbjohnson 4y ago

Litestream/LiteFS author here. I agree that "cloud" is a bit ambiguous but Go & SQLite are quite powerful together and I don't think it's only for beginners. Both are fast and have low overhead. In addition to lower cost versus RDS, there's near-zero query latency which eliminates a lot of performance problems. You can comfortably run tens or hundreds of requests per second on minimal hardware (e.g. 256MB or 512MB instances). There's a lot of room for scaling up before you hit a performance ceiling.

ehutch79 4y ago
Tens of requests a second?
- randomdata 4y ago
  He's being generous. Indeed, your service will be lucky if it sees tens of requests per day.
  - ehutch79 4y ago
    :-| Normally I'm on the side of 'do you actually get that much traffic?' but yeah, a dashboard view can generate dozens of requests alone, each with multiple queries.
- benbjohnson 4y ago
  A request can have a large range of queries within it so I was trying to account for that. If your requests are lightweight and mostly reads, you can do 1,000+ req/sec on a 256MB instance. YMMV.
tacitusarc 4y ago
Do you mean tens or hundreds of thousands of requests per second?
- benbjohnson 4y ago
  There aren’t many apps that have 100,000+ req/sec and you certainly can’t run them on modest hardware. If your app averages 100 req/sec then that’s 8.4M requests per day. That’s more than most applications out there.
msolberg 4y ago
I just watched your gophercon video on SQLite in production after seeing it in the article. Great talk. Anyone else who's interested can watch here: https://www.youtube.com/watch?v=XcAYkriuQ1o https://www.youtube.com/watch?v=XcAYkriuQ1o
- benbjohnson 4y ago
  Thanks!

markusw 4y ago

See my followup-article "Distributed SQLite with LiteFS" for the scale-out part: https://www.golang.dk/articles/distributed-sqlite-with-litefs https://www.golang.dk/articles/distributed-sqlite-with-litef... And IMO, what you lose in the cloud layer you gain by having a deployment really close to your user. Speed of light and all that. Plus SQLite is super cool and fun! :D

simonw 4y ago

At what point would you expect to need to scale out?

infamia 4y ago

Once you need to graduate from one large server (which will take you a long way in many cases), there are tools like [rqlite](https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite) that can handle clustering. With WAL mode enabled, SQLite can handle a surprising amount of traffic that would fit a lot of use cases. If latency is important to you, it's going to be hard to beat SQLite for many workloads.

randomdata 4y ago

You gain low latency, which allows you to write your code to a 'pure' relational model (at least as close as SQL allows), without having to rely on code complicating hacks to deal with round-trip issues. Less complicated code can improve delivery time and reduce faults. While there are abstractions that can be used to help with those hacks, they come with their own tradeoffs. Pick your poison, as always. Conceptually, RDS is little more than SQLite with a clever networking layer built on top. In context, your application is also just a clever networking layer, so the middleman doesn't necessarily add any value. Of course it depends on exactly what you are trying to do and what tradeoffs you are willing to accept. There is no free lunch.

ketralnis 4y ago

This is a pretty strong claim without any numbers, to be honest. You were probably already running your single database instance so being limited to a single sqlite instance isn't terribly different. This does limit you to a single app server, which unless you're CPU bound is also fine. Even if you are, you can get a lot of cores in a single instance these days. Most things will never need to be scaled up

aynyc 4y ago
AFAIK, sqlite3 doesn't support multi-core, so you can only do vertical scaling. In cloud, people rather do horizontal scaling. That being said, I ran a django based webapp for internal users and we saw core being pegged by sqlite3 when the queries were complex. However, under normal usage, it was fine for about 100 concurrent users including machine API calls.
- cldellow 4y ago
  > doesn't support multi-core Can you expand on that? I thought it supported multiple readers, but only a single writer. They do all have to be on the same host, though. Since you mentioned Django - I think there are some complexities where the Python driver can't really do concurrent access if using threads instead of processes, but this is due to Python/GIL limitations, not sqlite limitations.
  - aynyc 4y ago
    sqlite3 is a single core application, so even if you have multiple cores on a box, sqlite3 isn't going to take advantage of that. If you want sqlite3 to go faster, you need better single core box. That being said, I do believe sqlite3 allow multiple processes to read the same database file. Our django app sits behind Gunicorn that will spin up multiple django instances, as long as Sqlite3 has WAL mode on, concurrent access (read) isn't a problem. For write, we basically use a lock.
    - markusw 4y ago

0cf8612b2e1e 4y ago

> I can't see any reason not using RDS other than it's more expensive. We don’t all operate with unlimited VC money.

iveqy 4y ago

Depends on your application. On the opposite I've seen many cases where sqlite3 scales better than postgres, depending on the scale and the application. If you're doing horizontal scaling, sqlite3 is a very good alternative.

mariusor 4y ago

It's a little weird that you consider bringing up an application on bare metal something for a beginner. :) I think manually handling the nitty-gritty of an OPS setup is a more senior task than clicking you way into setting up some AWS infrastructure.