3 ms·

Yea, but does it scale? Yes, for 100 users maybe. But you have 100 000 small sqlite db:s I would be a bit worried about how it scales. Electric is super cool.

Yea, but does it scale? Yes, for 100 users maybe. But you have 100 000 small sqlite db:s I would be a bit worried about how it scales. Electric is super cool. Can't wait until it is a bit more mature.

biorach 3y ago
> Yea, but does it scale? Yes, for 100 users maybe. But you have 100 000 small sqlite db:s If it's a case of 1 node per customer (node being vm, lambda, cloudflare worker, whatever) then there should be no limit. Maybe you're thinking of a more "traditional" approach with a server process managing 1 sqlite db per customer, which might make sense in order to keep cloud costs down. Even in that case it should be trivially easy to distribute the load.
- victorbjorklund 3y ago
  So store sqlite on S3. Start a new lambda (or similar) per user, download the db from sqlite and write to it and upload it again. And making sure there can never be more than 1 function running per user (to avoid two lambdas editing the same database). Seems a bit fragil but I might be wrong.
samwillis 3y ago
In broad strokes having 100k SQLite dbs is a bit like having 100k git repositories. It's doable and just about having tooling to manage it. It actually makes a few things simpler, say progressively rolling out visiting upgrades. You need to write schema migrations anyway, this lets you upgrade one customer at a time.
- treis 3y ago
  The problem is that one at a time times 100,000 DBs takes a long time. For every second a migration takes its 1.1 days. You can parallelize it, of course, but that's a challenge in of itself.
  - remlov 3y ago
    > You can parallelize it, of course, but that's a challenge in of itself. Not a challenge at all really.
    - mring33621 3y ago
      GNU Parallel + some Python code?
- victorbjorklund 3y ago
  Yea, pretty much tooling and stuff is what I doubt scales. I'm sure a machine can handle it. It will just be very hard to manage because lack of tooling.

biorach 3y ago

> Yea, but does it scale? Yes, for 100 users maybe. But you have 100 000 small sqlite db:s If it's a case of 1 node per customer (node being vm, lambda, cloudflare worker, whatever) then there should be no limit. Maybe you're thinking of a more "traditional" approach with a server process managing 1 sqlite db per customer, which might make sense in order to keep cloud costs down. Even in that case it should be trivially easy to distribute the load.

victorbjorklund 3y ago
So store sqlite on S3. Start a new lambda (or similar) per user, download the db from sqlite and write to it and upload it again. And making sure there can never be more than 1 function running per user (to avoid two lambdas editing the same database). Seems a bit fragil but I might be wrong.

samwillis 3y ago

In broad strokes having 100k SQLite dbs is a bit like having 100k git repositories. It's doable and just about having tooling to manage it. It actually makes a few things simpler, say progressively rolling out visiting upgrades. You need to write schema migrations anyway, this lets you upgrade one customer at a time.

treis 3y ago
The problem is that one at a time times 100,000 DBs takes a long time. For every second a migration takes its 1.1 days. You can parallelize it, of course, but that's a challenge in of itself.
- remlov 3y ago
  > You can parallelize it, of course, but that's a challenge in of itself. Not a challenge at all really.
  - mring33621 3y ago
    GNU Parallel + some Python code?
victorbjorklund 3y ago
Yea, pretty much tooling and stuff is what I doubt scales. I'm sure a machine can handle it. It will just be very hard to manage because lack of tooling.