5 ms·

I love this. This is another in the "Now that we have computers with gazillions of bits of RAM and many cores and gazillions of cycles, what conventional wisdo

I love this. This is another in the "Now that we have computers with gazillions of bits of RAM and many cores and gazillions of cycles, what conventional wisdom is no longer wise?" series :-). I suggested to a client that they use Sqlite3 for their company database rather than a disk based database. And they looked at me shocked, as if I had spoken heresy. I pointed out how many records they could store on a 64GB machine and that is a small data center class machine and it sort of boggled their mind. Add read-only copies, shared from a NAS device gave them perfect scalability in terms of queries per second (their sales folks had their own database that they were updating). It was fun to watch their expression go from horror to wonder to excitement when they finally implemented it.

michaelmcmillan 6y ago
Yup! I'm currently running a b2c solution for a client that's grossing ~$3 million/year using Python + sqlite3 behind nginx. I work for another shop that scolds me for even suggesting using something else than k8s. How can a persistent file system possibly scale? Lol. Ironically they're grossing -$3 million/year.
- tonetheman 6y ago
  I am curious about your case is the traffic mostly read heavy? Ha wish you could share more. Interesting stuff.
dragontamer 6y ago
https://yourdatafitsinram.net/ https://yourdatafitsinram.net/ The IBM 53-bit quantum compute result from a ton of hard drives + compute power was also intriguing (https://www.ibm.com/blogs/research/2019/10/on-quantum-supremacy/ https://www.ibm.com/blogs/research/2019/10/on-quantum-suprem...). 128PiB of disk space needed for a 54-qubit computation on their system (hard drives). Fun to see where the supercomputers are still needed, but smaller qubits can be emulated on normal computers. 40 qubit calculations probably can be emulated on a 16TB team of hard drives in RAID0.
kortilla 6y ago
The trade-off isn’t really size in a lot of cases though. When stuff is stored in RAM you lose the persistence story.
- greggyb 6y ago
  Disk storage for the WAL and periodic checkpoints can get you sub-second recovery targets easily. Ninja edit: or a UPS and a shutdown procedure triggered by it that writes the database to disk. You can write 100GB in a couple minutes over a 10Gbps network. Faster to a local SSD.
  - kortilla 6y ago
    Sub-second recovery and UPS triggered shutdowns are still nowhere near the reliability of a database ensuring fsync has been called when you call “commit”. There are just certain classes of data (e.g. any kind of business transaction) where you can’t afford this type of best effort durability.
svnpenn 6y ago
> I suggested to a client that they use Sqlite3 for their company database rather than a disk based database. Sqlite is a disk based database...
- capableweb 6y ago
  Well, maybe not "disk based" per se. It can be stored on disk, yeah. But it can also run in memory. I think in this case it's clear that ChuckMcM meant a machine with 64GB of RAM (not disk space), therefore they are talking about having SQLite running in memory, not on disk.
  - sillysaurusx 6y ago
    Even if a DB is disk-based, you can throw it in a ramdisk to get the same speedups, often. I would be curious to know if there are any drawbacks or dangers from doing that, apart from the obvious “if your box powers off, the ramdisk is gone” and “each time the box restarts, you have to copy the db to the ramdisk.” The main problem I was uneasy about was, I would simply use the equivalent of rsync -Pa /ramdisk /disk in a loop, which seemed to ensure safety in the event of a poweroff. It never broke, but I wouldn’t say it was a good idea, necessarily. Sure speeds up most things though. The benefits are pretty crazy for I/O bound programs.
    - AI_WAIFU 6y ago
      Does anyone know the risk of radiation bit flips in this case? I don't think it's an issue with ecc ram, but if it's not that could be a problem for certain applications.
      - sillysaurusx 6y ago
        Exactly! I was wondering about that. It seems like it's so rare as not to matter, but that's no guarantee it won't matter.
      - count 6y ago
        A friend of mine wrote a 'chaos engineering' tester for this specific issue: http://github.com/enferex/cme http://github.com/enferex/cme Randomly inject bit flips into a target process memory space, so you can see/test how your software handles it (if at all).

michaelmcmillan 6y ago

Yup! I'm currently running a b2c solution for a client that's grossing ~$3 million/year using Python + sqlite3 behind nginx. I work for another shop that scolds me for even suggesting using something else than k8s. How can a persistent file system possibly scale? Lol. Ironically they're grossing -$3 million/year.

tonetheman 6y ago
I am curious about your case is the traffic mostly read heavy? Ha wish you could share more. Interesting stuff.

dragontamer 6y ago

https://yourdatafitsinram.net/ https://yourdatafitsinram.net/ The IBM 53-bit quantum compute result from a ton of hard drives + compute power was also intriguing (https://www.ibm.com/blogs/research/2019/10/on-quantum-supremacy/ https://www.ibm.com/blogs/research/2019/10/on-quantum-suprem...). 128PiB of disk space needed for a 54-qubit computation on their system (hard drives). Fun to see where the supercomputers are still needed, but smaller qubits can be emulated on normal computers. 40 qubit calculations probably can be emulated on a 16TB team of hard drives in RAID0.

kortilla 6y ago

The trade-off isn’t really size in a lot of cases though. When stuff is stored in RAM you lose the persistence story.

greggyb 6y ago
Disk storage for the WAL and periodic checkpoints can get you sub-second recovery targets easily. Ninja edit: or a UPS and a shutdown procedure triggered by it that writes the database to disk. You can write 100GB in a couple minutes over a 10Gbps network. Faster to a local SSD.
- kortilla 6y ago
  Sub-second recovery and UPS triggered shutdowns are still nowhere near the reliability of a database ensuring fsync has been called when you call “commit”. There are just certain classes of data (e.g. any kind of business transaction) where you can’t afford this type of best effort durability.

svnpenn 6y ago

> I suggested to a client that they use Sqlite3 for their company database rather than a disk based database. Sqlite is a disk based database...

capableweb 6y ago
Well, maybe not "disk based" per se. It can be stored on disk, yeah. But it can also run in memory. I think in this case it's clear that ChuckMcM meant a machine with 64GB of RAM (not disk space), therefore they are talking about having SQLite running in memory, not on disk.
- sillysaurusx 6y ago
  Even if a DB is disk-based, you can throw it in a ramdisk to get the same speedups, often. I would be curious to know if there are any drawbacks or dangers from doing that, apart from the obvious “if your box powers off, the ramdisk is gone” and “each time the box restarts, you have to copy the db to the ramdisk.” The main problem I was uneasy about was, I would simply use the equivalent of rsync -Pa /ramdisk /disk in a loop, which seemed to ensure safety in the event of a poweroff. It never broke, but I wouldn’t say it was a good idea, necessarily. Sure speeds up most things though. The benefits are pretty crazy for I/O bound programs.
  - AI_WAIFU 6y ago
    Does anyone know the risk of radiation bit flips in this case? I don't think it's an issue with ecc ram, but if it's not that could be a problem for certain applications.
    - sillysaurusx 6y ago
      Exactly! I was wondering about that. It seems like it's so rare as not to matter, but that's no guarantee it won't matter.
    - count 6y ago
      A friend of mine wrote a 'chaos engineering' tester for this specific issue: http://github.com/enferex/cme http://github.com/enferex/cme Randomly inject bit flips into a target process memory space, so you can see/test how your software handles it (if at all).