4 ms·

Regarding 2: I've had problems with this strategy. Even though I used the swap technique instead of overwriting, files easily get corrupted on common file syste

Regarding 2: I've had problems with this strategy. Even though I used the swap technique instead of overwriting, files easily get corrupted on common file systems, and if it's just the user force quitting the application during the operation. (I mostly use Ext4 for testing.) Whatever Sqlite does under the hood is way more resilient to such problems. I guess that's because the Sqlite developers have been testing ACID compliance extensively. In one of my Racket applications with high integrity demands I've resorted to writing the temp file, swapping it with the original, then opening it read-only and verifying all of its contents. That is very reliable but obviously slow. But in another application written in Go I've switched entirely to Sqlite even for simple settings files and had no problems since.

TekMol 4y ago
We would have to discuss the specific use case and how the data got corrupted. When interacting with a typical web application, the user cannot terminate it in the middle of a file write application.
Spakman 4y ago
Is the swap technique you mention writing to a tempfile and then performing an (atomic) rename to overwrite the existing file? I understand how that could cause data loss if the process exited before the rename but I don't understand how it might cause corruption. Are you able to expand on that? A related thing I discovered recently is the potential non-atomicity of writes using O_APPEND on Linux[1], although I couldn't get the attached test program to fail on my machine. I would love to find some kind of confirmation that this behaviour has been changed and that appends can be relied upon to be atomic. In the application I'm working on at the moment I've managed to get add more certainty around this stuff since I'm only ever appending to the file in 4096 byte chunks, so can perform a simple file size check to see if data was written completely or if the file needs to truncated to the nearest multiple of 4096. I'm fortunate that it prevents having to do something like open it read only to verify contents as you mentioned. 1. https://bugzilla.kernel.org/show_bug.cgi?id=55651 https://bugzilla.kernel.org/show_bug.cgi?id=55651
- jonathanstrange 4y ago
  Yes, I used a tempfile with atomic rename. I agree data corruption should not occur as long as fsync(2) is used before the swap. If I remember correctly an empty file was an error condition in this case, though, and that sometimes occurred. Problems can occur when the user shuts down the OS or hits that reset button. Besides, in Go it's easy to inadvertently write to a file concurrently and extra care has to be taken to avoid that. So you might need a locking mechanism, too, and then it's getting really complex. Of course, it should be perfectly possible to get this right in languages like Go or Racket. At least Go has low-level enough interfaces to the filesystem API. My point was merely that using Sqlite turned out to be less error-prone in my experience for end-user applications, especially if you use WAL and set some cautious Pragma options. It's really good at dealing with unusual filesystem conditions.
  - Spakman 4y ago
    Thanks for clarifying. I agree that reliably writing files is a sneakier problem than it first appears and that the likes of SQLite have already solved it. Despite that my preference these days is still for a DB, like almost all dependencies, to have to justify its existence in my projects rather than defaulting to it as many seem to favour. It's straightforward to add SQLite but I'd typically rather take a little extra pain around the filesystem upfront to not have to deal with all the extra complexity of an SQL database if I can avoid it. Naturally, this kind of simplicity is just one factor among many such as the size and shape of the data, expected access patterns and a host of others to weigh up during the trade-off decision.
    - gunnihinn 4y ago
      > SQLite does not compete with client/server databases. SQLite competes with fopen(). https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
deleted 4y ago
[deleted]
soulofmischief 4y ago
At least for my needs, NeDB[0] is the best of both worlds for prototyping and early-stage production releases. It's human-readable, on-disk, greppable, still supports indexing and a subset of Mongo features while remaining serverless and in-memory. [0] https://github.com/louischatriot/nedb https://github.com/louischatriot/nedb
jodrellblank 4y ago
See https://danluu.com/file-consistency/ https://danluu.com/file-consistency/ for a blog post on why files are hard.
- Spakman 4y ago
  Excellent link, thanks! One of the linked papers[1] in that article is highly recommended. 1. https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14...

TekMol 4y ago

We would have to discuss the specific use case and how the data got corrupted. When interacting with a typical web application, the user cannot terminate it in the middle of a file write application.

Spakman 4y ago

Is the swap technique you mention writing to a tempfile and then performing an (atomic) rename to overwrite the existing file? I understand how that could cause data loss if the process exited before the rename but I don't understand how it might cause corruption. Are you able to expand on that? A related thing I discovered recently is the potential non-atomicity of writes using O_APPEND on Linux[1], although I couldn't get the attached test program to fail on my machine. I would love to find some kind of confirmation that this behaviour has been changed and that appends can be relied upon to be atomic. In the application I'm working on at the moment I've managed to get add more certainty around this stuff since I'm only ever appending to the file in 4096 byte chunks, so can perform a simple file size check to see if data was written completely or if the file needs to truncated to the nearest multiple of 4096. I'm fortunate that it prevents having to do something like open it read only to verify contents as you mentioned. 1. https://bugzilla.kernel.org/show_bug.cgi?id=55651 https://bugzilla.kernel.org/show_bug.cgi?id=55651

jonathanstrange 4y ago
Yes, I used a tempfile with atomic rename. I agree data corruption should not occur as long as fsync(2) is used before the swap. If I remember correctly an empty file was an error condition in this case, though, and that sometimes occurred. Problems can occur when the user shuts down the OS or hits that reset button. Besides, in Go it's easy to inadvertently write to a file concurrently and extra care has to be taken to avoid that. So you might need a locking mechanism, too, and then it's getting really complex. Of course, it should be perfectly possible to get this right in languages like Go or Racket. At least Go has low-level enough interfaces to the filesystem API. My point was merely that using Sqlite turned out to be less error-prone in my experience for end-user applications, especially if you use WAL and set some cautious Pragma options. It's really good at dealing with unusual filesystem conditions.
- Spakman 4y ago
  Thanks for clarifying. I agree that reliably writing files is a sneakier problem than it first appears and that the likes of SQLite have already solved it. Despite that my preference these days is still for a DB, like almost all dependencies, to have to justify its existence in my projects rather than defaulting to it as many seem to favour. It's straightforward to add SQLite but I'd typically rather take a little extra pain around the filesystem upfront to not have to deal with all the extra complexity of an SQL database if I can avoid it. Naturally, this kind of simplicity is just one factor among many such as the size and shape of the data, expected access patterns and a host of others to weigh up during the trade-off decision.
  - gunnihinn 4y ago
    > SQLite does not compete with client/server databases. SQLite competes with fopen(). https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html

deleted 4y ago

[deleted]

soulofmischief 4y ago

At least for my needs, NeDB[0] is the best of both worlds for prototyping and early-stage production releases. It's human-readable, on-disk, greppable, still supports indexing and a subset of Mongo features while remaining serverless and in-memory. [0] https://github.com/louischatriot/nedb https://github.com/louischatriot/nedb

jodrellblank 4y ago

See https://danluu.com/file-consistency/ https://danluu.com/file-consistency/ for a blog post on why files are hard.

Spakman 4y ago
Excellent link, thanks! One of the linked papers[1] in that article is highly recommended. 1. https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14...