5 ms·

> Next, his problem was that the database would get corrupt sometimes when something bad happened in the middle of writing the file. I'm not sure i understand

> Next, his problem was that the database would get corrupt sometimes when something bad happened in the middle of writing the file. I'm not sure i understand how this can happen... unless you try to update JSON in-place (which is a very bad idea for any text-based format), what you do is encode/write the entire JSON from scratch. So either the file is written properly or it isn't written. Honestly from the entire message it doesn't sound like JSON was a bad idea but that your coworker didn't know what he was doing and if he was doing something else then he'd still be doing big mistakes.

oxymoron 6y ago
1. A cosmic ray storm turned all ASCII charactees into ECBDIC 2. Lightning struck the 12 V feed and upped the voltage to 10 MV, turning all 0 and 1’s into 6’s 3. Someone spilled a New England Pale Ale on the server 4. The process was assinated by the mysterious killer only known from his modus operandi of leaving OOM written in blood across the syslog 5. Birds nested within the server and fed all the SATA cables to their babies Seriously though, disk writes aren’t atomic.
- lalaland1125 6y ago
  File renames are atomic. This is a solved problem: 1. Write your updates to a copy of the file. 2. Do an atomic rename of that copy to the original.
  - Zarel 6y ago
    Renames aren't atomic on crash. https://danluu.com/file-consistency/ https://danluu.com/file-consistency/
    - jclulow 6y ago
      They are if you perform the correct sequence of fsync operations on both the file and the directories, and use a file system which is correctly implemented.
      - chrismorgan 6y ago
        Renames aren’t atomic on crash.
        badsectoracula 6y ago
        What crash? The article linked above never explains that part, it only assumes that it will happen. From the code it sounds as if the crash can happen in the OS itself (but then the entire kernel will crash). At that point things are completely outside your control and you might as well running on broken hardware.
    - mpweiher 6y ago
      Hmm...unlike the rest of the post, he just asserts this without support. His sources seem to disagree, certainly with that kind of blanket statement. For example: Our study takes a pessimistic view of file-system behavior; for example, we even consider the case where renames are not atomic on a system crash. So this is clearly considered an outlier/unusual. (e.g., a single 512-byte write or file rename operation are guaranteed to be atomic by many current file systems when running on a hard-disk drive) [https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14...] I remember reading quite a bit about the (performance reducing) lengths filesystems go to in order to ensure consistency of directory entries even in case of a crash, and for example how "soft updates" were introduced to accomplish the same consistency with less of a performance degradation. Looking at it from another angle, if you are running on top of a filesystem that cannot keep itself consistent, then you are SOL, there really isn't anything you can do to mitigate. Just like we can't guarantee that we will be able to persist data that's in memory to disk if the OS is free to kill us at any time. "Best effort" it is, which means getting the data to disk as quickly as possible and not corrupting what is there.
  - fanf2 6y ago
    1. Write your updates to a copy of the file. 1a. fsync() the file to ensure the contents are durable. 2. Do an atomic rename of that copy to the original. 2a. fsync() the directory to ensure the rename is durable.
    - fiddlerwoaroof 6y ago
      Or just use a database like SQLite or Postgres that has developers dedicated to solving this problem.
- badsectoracula 6y ago
  Disk writes aren't atomic but you know if they failed or not (unless your OS is lying to you but that is a problem with the OS, not your code).
ComputerGuru 6y ago
Here, have a read. Warning: if you haven’t done low-level development, you might walk away wondering if your entire life has been a lie. https://danluu.com/file-consistency/ https://danluu.com/file-consistency/
- SXX 6y ago
  Truly amazing read. Thanks for sharing.
- blitmap 6y ago
  I learned some things! Thank you. :)
- badsectoracula 6y ago
  I've read that at the past but this article has an issue: > if there's a crash during the write It never brings up how can there be a crash in the first place (also the entire article is too filesystem specific).
  - Groxx 6y ago
    Any crash. Power failure, hardware failure, kernel panic while working for another process, software bugs, etc. It's impossible to make crash-free systems. If your goal is to maximize the chance that your data remains valid, you have to plan on that.

oxymoron 6y ago

1. A cosmic ray storm turned all ASCII charactees into ECBDIC 2. Lightning struck the 12 V feed and upped the voltage to 10 MV, turning all 0 and 1’s into 6’s 3. Someone spilled a New England Pale Ale on the server 4. The process was assinated by the mysterious killer only known from his modus operandi of leaving OOM written in blood across the syslog 5. Birds nested within the server and fed all the SATA cables to their babies Seriously though, disk writes aren’t atomic.

lalaland1125 6y ago
File renames are atomic. This is a solved problem: 1. Write your updates to a copy of the file. 2. Do an atomic rename of that copy to the original.
- Zarel 6y ago
  Renames aren't atomic on crash. https://danluu.com/file-consistency/ https://danluu.com/file-consistency/
  - jclulow 6y ago
    They are if you perform the correct sequence of fsync operations on both the file and the directories, and use a file system which is correctly implemented.
    - chrismorgan 6y ago
      Renames aren’t atomic on crash.
      - badsectoracula 6y ago
        What crash? The article linked above never explains that part, it only assumes that it will happen. From the code it sounds as if the crash can happen in the OS itself (but then the entire kernel will crash). At that point things are completely outside your control and you might as well running on broken hardware.
  - mpweiher 6y ago
    Hmm...unlike the rest of the post, he just asserts this without support. His sources seem to disagree, certainly with that kind of blanket statement. For example: Our study takes a pessimistic view of file-system behavior; for example, we even consider the case where renames are not atomic on a system crash. So this is clearly considered an outlier/unusual. (e.g., a single 512-byte write or file rename operation are guaranteed to be atomic by many current file systems when running on a hard-disk drive) [https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-pillai.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14...] I remember reading quite a bit about the (performance reducing) lengths filesystems go to in order to ensure consistency of directory entries even in case of a crash, and for example how "soft updates" were introduced to accomplish the same consistency with less of a performance degradation. Looking at it from another angle, if you are running on top of a filesystem that cannot keep itself consistent, then you are SOL, there really isn't anything you can do to mitigate. Just like we can't guarantee that we will be able to persist data that's in memory to disk if the OS is free to kill us at any time. "Best effort" it is, which means getting the data to disk as quickly as possible and not corrupting what is there.
- fanf2 6y ago
  1. Write your updates to a copy of the file. 1a. fsync() the file to ensure the contents are durable. 2. Do an atomic rename of that copy to the original. 2a. fsync() the directory to ensure the rename is durable.
  - fiddlerwoaroof 6y ago
    Or just use a database like SQLite or Postgres that has developers dedicated to solving this problem.
badsectoracula 6y ago
Disk writes aren't atomic but you know if they failed or not (unless your OS is lying to you but that is a problem with the OS, not your code).

ComputerGuru 6y ago

Here, have a read. Warning: if you haven’t done low-level development, you might walk away wondering if your entire life has been a lie. https://danluu.com/file-consistency/ https://danluu.com/file-consistency/

SXX 6y ago
Truly amazing read. Thanks for sharing.
blitmap 6y ago
I learned some things! Thank you. :)
badsectoracula 6y ago
I've read that at the past but this article has an issue: > if there's a crash during the write It never brings up how can there be a crash in the first place (also the entire article is too filesystem specific).
- Groxx 6y ago
  Any crash. Power failure, hardware failure, kernel panic while working for another process, software bugs, etc. It's impossible to make crash-free systems. If your goal is to maximize the chance that your data remains valid, you have to plan on that.