4 ms·

> Because then on a power failure the file would contain "a stable commit" but not necessarily "the most recent commit" ... And it seems that for most SQLite in

> Because then on a power failure the file would contain "a stable commit" but not necessarily "the most recent commit" ... And it seems that for most SQLite integrations that is what we want? You HAVE to commit the journal to disk in a stable fashion - that requires fsync() because without there no guarantee that the data is actually on disk. Without the journal there’s no way to back out a partially applied write, leaving you with a corrupt database. Could an application maintain the state in memory and periodically flush out to SQLite to get around it? Sure. But many people would be pissed to lose their browser history for the past N time before a crash or power loss after just finding a page they needed for some purpose. Get a better storage solution for your home directories, ZFS with high lifespan Optane drives as a SLOG would dramatically increase the write endurance of standard NAND flash by batching writes while maintaining acceptable fsync performance (since synchronous writes don’t hit the pool itself until the transaction group commits every X seconds).

zbuf 6y ago
> You HAVE to commit the journal to disk in a stable fashion - that requires fsync() because without there no guarantee that the data is actually on disk. But does this indicate the issue; that userland only has fsync() API -- it's a case of full synchronous write, or not. Whereas the SQLite file could contain its own journal and data, and maintain the integrity using a write barrier. I still have no need for my browser history to be fully synchronous to disk. Yet that's what we're doing, and then... > Get a better storage solution for your home directories, ZFS with high lifespan Optane drives as a SLOG would dramatically increase the write endurance of standard NAND flash by batching writes while maintaining acceptable fsync performance ... spending on high grade tech in an attempt to solve the problem we just created. Fundamentally, we're asking these systems to do synchronous writes, when it's not what we actually want. When everything's a priority, nothing's a priority.
theamk 6y ago
Or maybe stop using sqlite, and serialize your state to disk periodically, just like this document says. Modern filesystems are pretty good in maintaining file integrity if you use “atomic overwrite via rename” trick, and for extra assurance you can store an older version of config or five. This will give you a “pretty robust” solution which will work “most of the time”, and that will completely avoid fsyncs and prolong your SSD life. What’s not to like about it?

zbuf 6y ago

> You HAVE to commit the journal to disk in a stable fashion - that requires fsync() because without there no guarantee that the data is actually on disk. But does this indicate the issue; that userland only has fsync() API -- it's a case of full synchronous write, or not. Whereas the SQLite file could contain its own journal and data, and maintain the integrity using a write barrier. I still have no need for my browser history to be fully synchronous to disk. Yet that's what we're doing, and then... > Get a better storage solution for your home directories, ZFS with high lifespan Optane drives as a SLOG would dramatically increase the write endurance of standard NAND flash by batching writes while maintaining acceptable fsync performance ... spending on high grade tech in an attempt to solve the problem we just created. Fundamentally, we're asking these systems to do synchronous writes, when it's not what we actually want. When everything's a priority, nothing's a priority.

theamk 6y ago

Or maybe stop using sqlite, and serialize your state to disk periodically, just like this document says. Modern filesystems are pretty good in maintaining file integrity if you use “atomic overwrite via rename” trick, and for extra assurance you can store an older version of config or five. This will give you a “pretty robust” solution which will work “most of the time”, and that will completely avoid fsyncs and prolong your SSD life. What’s not to like about it?