3 ms·
I have had some bad experiences with Sqlite for local desktop apps with regards to memory usage, especially on MacOs. Insert and delete a few thousands rows per
by Leherenn 1y ago
I have had some bad experiences with Sqlite for local desktop apps with regards to memory usage, especially on MacOs.
Insert and delete a few thousands rows per hour, and over a few days your memory usage has ballooned. It seems to cause a lot of fragmentation.
- dkga 1y agoCurious to hear more about your experience, since my impression from hacking around in native Apple software is that pretty much a bunch of it is based on top of sqlite3. The Photos app is a case in point I remember off the cuff.
- Leherenn 1y agoI think we might be using it in a slightly unusual way: collect data for a while, do one big query once in a while to aggregate/transform the data and clean everything. Rinse and repeat as it's a background app. So lots of allocations/deallocations. If you're only storing a few key/value pairs long term, you won't have any issues.
- echelon 1y agoWhat kind of memory usage are you talking about? That sounds wild. Are you keeping the entire database in memory, or are you flushing to disk? It seems wild that such a critical and pervasive piece of software would behave like that.
- Leherenn 1y agoKeeping it in memory. After a few days, the memory usage reported by the Activity Monitor (so not the actual resident memory, but the one customers see and complain about) grows from maybe a few 10s MB to a few hundreds MBs. But as far as I can tell, it's more an OS issue than really a Sqlite issue, simply doing malloc/free in a loop results in a similar behaviour. And Sqlite does a lot of allocations. We see a similar problem on Windows, but not as pronounced, and there we can force the OS to reclaim memory. It's probably solvable by using a custom allocator, but at this point it's no longer plug and play the way the GP meant.