4 ms·

Segue: What the heck is about Windows files and null characters? I've been almost exclusively dealing with POSIX file systems for the last 30 years, but I'm c

Segue: What the heck is about Windows files and null characters? I've been almost exclusively dealing with POSIX file systems for the last 30 years, but I'm currently shipping a cross-platform app and a lot of my Windows users are encountering corrupted files which exhibit a bunch of NULs in random places in the file. I've added ugly hacks to deal with them but it'd be nice to get down to root causes. Is there a handy list of things that are safe on POSIX but not on Windows so I can figure out what I'm doing wrong? I'm at the stage where I'm thinking "%$#@ this, I'm never going to write to the Windows file system again, I'm just going to create an SQLite DB and write to that instead". Will that fix my problems?

nradov 2y ago
The Windows NTFS is safe and reliable. It doesn't corrupt files. You have probably misunderstood the problem.
- bryanlarsen 2y ago
  However I'm using it it certainly isn't. It's certainly quite likely the problem is me, not Windows. Most of the files that are getting corrupted are being written to in an append-only fashion, which is generally one of the mechanisms for writing to files to avoid corruption, at least on POSIX.
- omoikane 2y ago
  I have observed a file filled NULs that was caused by a power loss in the middle of a write -- my UPS alerted me that utility power is gone, I tried to shutdown cleanly, but the battery died before a clean shutdown completed. This was NTFS on a HDD and not a SSD. I am not saying it happens often, but it does happen once in a while.
  - bryanlarsen 2y ago
    Yes, corruption does appear to be correlated with power cycles.
  - ale42 2y ago
    Had the same on journaled ext4 on Linux. Lots of NULL bytes in the middle of the syslog because of unclean shutdown.
  - dist-epoch 2y ago
    NTFS guarantees file-system metadata integrity, not file data integrity. Subtle but important difference. The file was corrupted, but the file-system remained consistent.
tatersolid 2y ago
Returning all zeros randomly is one of the failure modes of crappy consumer SSDs with buggy controllers. Especially those found in cheap laptops and on Amazon. If it’s a fully counterfeit drive it might even be maliciously reporting a size larger than it has flash chips to support. It will accept writes without error but return zeros for sectors that are “fake”. This can appear random due to wear-leveling in the controller.
alexisread 2y ago
I'd guess yes, and you get the SQL goodness to boot :) Sounds like you have an encoding issue somewhere, windows has it's own charset - Windows-1252, so I'd vet all your libs that touch the file (including eg. .Net libs etc). If one of them defaults to that encoding you may get it either mislabelling the file encoding, or adding in null after each append etc. SQLite is tested cross-platform so 100% the file will be cross-platform compatible.
dist-epoch 2y ago
A lot of your users encountering file corruption on Windows either means that somehow your users are much more likely to have broken hardware, or more realistically that you have a bug in your code/libraries.
- bryanlarsen 2y ago
  I think omoikane pinpointed the primary cause, hard power loss. I definitely do some writing when receiving the shutdown signal, and I bet some people initiate a normal shutdown but remove power before it finishes. I think trying to avoid writes upon receipt of a shutdown signal will help.
o11c 2y ago
Normally, there are 3 notable things that are unique about Windows files: * It's standard for "security" software to be installed and hook into the filesystem. Normally this only means giving up on several orders of magnitude of performance, but as this incident reminds us: security software is indistinguishable from malware. * The system has a very long history of relying on exclusive access to files. This is one of the major reasons for Windows needing to restart so many times, but also leads to a lot of programs being badly written, among other things making false assumptions regarding atomicity in the presence of system failure. * There's a difference between text files and binary files, and thus a whole pain of adding/removing \r before \n (which among other things means byte counts are wrong and checksums mismatching).

nradov 2y ago

The Windows NTFS is safe and reliable. It doesn't corrupt files. You have probably misunderstood the problem.

bryanlarsen 2y ago
However I'm using it it certainly isn't. It's certainly quite likely the problem is me, not Windows. Most of the files that are getting corrupted are being written to in an append-only fashion, which is generally one of the mechanisms for writing to files to avoid corruption, at least on POSIX.
omoikane 2y ago
I have observed a file filled NULs that was caused by a power loss in the middle of a write -- my UPS alerted me that utility power is gone, I tried to shutdown cleanly, but the battery died before a clean shutdown completed. This was NTFS on a HDD and not a SSD. I am not saying it happens often, but it does happen once in a while.
- bryanlarsen 2y ago
  Yes, corruption does appear to be correlated with power cycles.
- ale42 2y ago
  Had the same on journaled ext4 on Linux. Lots of NULL bytes in the middle of the syslog because of unclean shutdown.
- dist-epoch 2y ago
  NTFS guarantees file-system metadata integrity, not file data integrity. Subtle but important difference. The file was corrupted, but the file-system remained consistent.

tatersolid 2y ago

Returning all zeros randomly is one of the failure modes of crappy consumer SSDs with buggy controllers. Especially those found in cheap laptops and on Amazon. If it’s a fully counterfeit drive it might even be maliciously reporting a size larger than it has flash chips to support. It will accept writes without error but return zeros for sectors that are “fake”. This can appear random due to wear-leveling in the controller.

alexisread 2y ago

I'd guess yes, and you get the SQL goodness to boot :) Sounds like you have an encoding issue somewhere, windows has it's own charset - Windows-1252, so I'd vet all your libs that touch the file (including eg. .Net libs etc). If one of them defaults to that encoding you may get it either mislabelling the file encoding, or adding in null after each append etc. SQLite is tested cross-platform so 100% the file will be cross-platform compatible.

dist-epoch 2y ago

A lot of your users encountering file corruption on Windows either means that somehow your users are much more likely to have broken hardware, or more realistically that you have a bug in your code/libraries.

bryanlarsen 2y ago
I think omoikane pinpointed the primary cause, hard power loss. I definitely do some writing when receiving the shutdown signal, and I bet some people initiate a normal shutdown but remove power before it finishes. I think trying to avoid writes upon receipt of a shutdown signal will help.

o11c 2y ago

Normally, there are 3 notable things that are unique about Windows files: * It's standard for "security" software to be installed and hook into the filesystem. Normally this only means giving up on several orders of magnitude of performance, but as this incident reminds us: security software is indistinguishable from malware. * The system has a very long history of relying on exclusive access to files. This is one of the major reasons for Windows needing to restart so many times, but also leads to a lot of programs being badly written, among other things making false assumptions regarding atomicity in the presence of system failure. * There's a difference between text files and binary files, and thus a whole pain of adding/removing \r before \n (which among other things means byte counts are wrong and checksums mismatching).