4 ms·

No it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html https://www.sqlite.org/releaselog/3_33_0.html [1]

No it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html https://www.sqlite.org/releaselog/3_33_0.html [1] https://www.sqlite.org/limits.html https://www.sqlite.org/limits.html (#12)

sgt 2y ago
You can split up into 10 SQLite DB's on this individual server.
- cdchn 2y ago
  You've now implemented sharding on top of SQLite. Eventually all programs will be able to read email.
  - mrbungie 2y ago
    Any non-trivial complexity codebase eventually implements a mediocre SQL/Lisp/etc.
  - Closi 2y ago
    Exactly! Why take a system not designed for this sort of scale and force it to scale, rather than use systems which are designed and tested for this scale and volume? All you will do is hackily re-invent all the other things that the other databases had to do to scale to this extent. Plus size is only one limit, you would be limited to 1 write every few milliseconds. My napkin maths estimate is that there are at least 1-2m writes per hour going into this thing, so probably 300-600 writes / second (Average) and maybe over 1k writes/second peak. We are going to fall over here! Not sure why some people seem to have a viwe of "There is no scaling problem that can't be solved with a sufficient enough number of SQLite databases".
- zaphirplane 2y ago
  > You can split up into 10 SQLite DB's on this individual server. 1 is a scalable, managed, highly available service, with economies of scale the other is a fixed size, capital expenditure with fixed performance, limited DR, requiring a couple of SRE/DevOps and colo There is also the will it always work question
- mlnj 2y ago
  Just storing petabytes of data is not the issue. Managing and querying it reliably is.
- chasil 2y ago
  Beware of WAL mode, as you sacrifice ACID in this configuration. https://sqlite.org/lang_attach.html https://sqlite.org/lang_attach.html 'Transactions involving multiple attached databases are atomic, assuming that the main database is not ":memory:" and the journal_mode is not WAL. If the main database is ":memory:" or if the journal_mode is WAL, then transactions continue to be atomic within each individual database file. But if the host computer crashes in the middle of a COMMIT where two or more database files are updated, some of those files might get the changes where others might not.'
- nemothekid 2y ago
  Once you are splitting up 10 sqlite dbs you have a bespoke distributed system anyways, and you will find yourself doing all the headache of LedgerStore anyways. Most of the novel work in LedgerStore is probably around managing the headaches of distributed storage, not the persistence layer.

sgt 2y ago

You can split up into 10 SQLite DB's on this individual server.

cdchn 2y ago
You've now implemented sharding on top of SQLite. Eventually all programs will be able to read email.
- mrbungie 2y ago
  Any non-trivial complexity codebase eventually implements a mediocre SQL/Lisp/etc.
- Closi 2y ago
  Exactly! Why take a system not designed for this sort of scale and force it to scale, rather than use systems which are designed and tested for this scale and volume? All you will do is hackily re-invent all the other things that the other databases had to do to scale to this extent. Plus size is only one limit, you would be limited to 1 write every few milliseconds. My napkin maths estimate is that there are at least 1-2m writes per hour going into this thing, so probably 300-600 writes / second (Average) and maybe over 1k writes/second peak. We are going to fall over here! Not sure why some people seem to have a viwe of "There is no scaling problem that can't be solved with a sufficient enough number of SQLite databases".
zaphirplane 2y ago
> You can split up into 10 SQLite DB's on this individual server. 1 is a scalable, managed, highly available service, with economies of scale the other is a fixed size, capital expenditure with fixed performance, limited DR, requiring a couple of SRE/DevOps and colo There is also the will it always work question
mlnj 2y ago
Just storing petabytes of data is not the issue. Managing and querying it reliably is.
chasil 2y ago
Beware of WAL mode, as you sacrifice ACID in this configuration. https://sqlite.org/lang_attach.html https://sqlite.org/lang_attach.html 'Transactions involving multiple attached databases are atomic, assuming that the main database is not ":memory:" and the journal_mode is not WAL. If the main database is ":memory:" or if the journal_mode is WAL, then transactions continue to be atomic within each individual database file. But if the host computer crashes in the middle of a COMMIT where two or more database files are updated, some of those files might get the changes where others might not.'
nemothekid 2y ago
Once you are splitting up 10 sqlite dbs you have a bespoke distributed system anyways, and you will find yourself doing all the headache of LedgerStore anyways. Most of the novel work in LedgerStore is probably around managing the headaches of distributed storage, not the persistence layer.