4 ms·
I love the webamp website. But, in the world of multigig RAM and multigbps I/O speeds, 1.2GB isn't that much. Not sure if that's something novel for SQLite3 tho
by sedatk 4y ago
I love the webamp website. But, in the world of multigig RAM and multigbps I/O speeds, 1.2GB isn't that much. Not sure if that's something novel for SQLite3 though.
- walrus01 4y agounlike a webserver setup maybe 20 years ago now it's trivially easy to hold even a 30GB database entirely in RAM and never touch disk, for a vast performance increase. I think I saw a used x86-64 server with 512GB of RAM for sale for $1700 recently.
- metadat 4y ago> I think I saw a used x86-64 server with 512GB of RAM for sale for $1700 recently. Where? Sounds like a terrific bargain, my old supermicro only has 384GB.
- walrus01 4y agoebay, was a dell r630 with two somewhat older 14-core xeons, 28 cores
- tiernano 4y agoYou could pick up an R620 with 768GB Ram and 2 8 core Xeons for around 1400GBP (https://www.bargainhardware.co.uk/dell-poweredge-r620-10-sff-bays-configure-to-order https://www.bargainhardware.co.uk/dell-poweredge-r620-10-sff...) without disks... DDR4 is a bit more expensive... but DDR3 is still fast for a lot of jobs...
- walrus01 4y agoR620 is a problem for anything that needs high bandwidth networking, however, because the pci-e slots are only 2.0. ruling out the use of many 10GbE interfaces or things like two port 100GbE
- midislack 4y agoSince the 1970's Unix has held frequently used pages in cache. Still does!
- deleted 4y ago[deleted]
- walrus01 4y agonot the same thing as tuning the db settings to specifically cache the whole thing in RAM and maybe even making its disk storage live on a RAM drive that's created at boot.
- midislack 4y agoTime to revive the sticky bit!
- astrange 4y agoThat's mlock() and it does already exist. People really like to reinvent things their OS already has because they're one step too far above the kernel to use them - like building a web service to send files between two internal servers when you could use NFS.
- hedora 4y agoIf only this were true for S3 and friends. (It is true for clients of scale out NFS servers, for what it's worth.)
- 6c696e7578 4y agoIf you don't care much about the data, no harm putting it in /dev/shm :)
- apetresc 4y agoNot novel at all. SQLite3 can handle 100GB+ files on fast storage without breaking a sweat.
- pornel 4y agoYeah, I have a project with about this much data, and literally read the whole file into memory, and rewrite the whole file on disk every time I need to modify it. It's hilarious that it works.
- solardev 4y agoWait, is that just how SQlite works? It can't update an entry on disk, it just has to rewrite the whole file...?
- arein3 4y agoIt writes in a file, but only the part needed, not the whole file (like any other DB with persistence)
- adzm 4y agoIt can also use a WAL log file and periodically checkpoint to the actual db
- tinco 4y agoNo, they're saying they've implemented something that's even more simple than an embedded database and it still works fine for them. If you can stand several 100s of milliseconds of latency on every write then writing a 1.2gb flat file is not a problem.
- iforgotpassword 4y agoAt that point, why not just mmap the whole file?
- elteto 4y agoProbably because file truncation/expansion with mmap is tricky.