3 ms·
> they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug. This is why whenever I need to pers
by Retr0id 2y ago
> they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug.
This is why whenever I need to persist any kind of state to disk, SQLite is the first tool I reach for. Filesystem APIs are scary, but SQLite is well-behaved.
Of course, it doesn't always make sense to do that, like the dropbox use case.
- ziddoap 2y ago>SQLite is the first tool I reach for. Hopefully in whichever particular mode is referenced!
- Retr0id 2y agoWAL mode, yes!
- nodamage 2y agoBefore becoming too overconfident in SQLite note that Rebello et al. (https://ramalagappan.github.io/pdfs/papers/cuttlefs.pdf https://ramalagappan.github.io/pdfs/papers/cuttlefs.pdf) tested SQLite (along with Redis, LMDB, LevelDB, and PostgreSQL) using a proxy file system to simulate fsync errors and found that none of them handled all failure conditions safely. In practice I believe I've seen SQLite databases corrupted due to what I suspect are two main causes: 1. The device powering off during the middle of a write, and 2. The device running out of space during the middle of a write.
- ablob 2y agoI believe it is impossible to prevent dataloss if the device powers off during a write. The point about corruption still stands and appears to be used correctly from what I skimmed in the paper. Nice reference.
- SoftTalker 2y agoOnly way I know of is if you have e.g. a RAID controller with a battery-backed write cache. Even that may not be 100% reliable but it's the closest I know of. Of course that's not a software solution at all.
- count 2y agoThat's uh, not running out of power in the middle of the write. That's having extra special backup power to finish the write. If your battery dies mid cache-write-out, you're still screwed.
- Dylan16807 2y agoI file that under hardware failure, not mundane power loss.
- wmf 2y agoIf the file system uses strict COW it should survive that situation.
- lmm 2y ago> I believe it is impossible to prevent dataloss if the device powers off during a write. Most devices write sectors atomically, and so you can build a system on top of that that does not lose committed data. (Of course if the device powers off during a write then you can lose the uncommitted data you were trying to write, but the point is you don't ever have corruption, you get either the data that was there before the write attempt or the data that is there after).
- justin66 2y agoI remembered Howard Chu commenting on that paper... https://lists.openldap.org/hyperkitty/list/openldap-devel@openldap.org/thread/YUUKXVYXA347IWW3UKRS6NJHBU4FEE6M/ https://lists.openldap.org/hyperkitty/list/openldap-devel@op... I'm pretty sure that's not where I originally saw his comments. I remember his criticisms being a little more pointed. Although I guess "This is a bunch of academic speculation, with a total absence of real world modeling to validate the failure scenarios they presented" is pretty pointed.
- eatonphil 2y agoDo you turn on SQLite checksumming or how do you feel comfortable that data on disk stays keeps integrity?