5 ms·
Two issues: Your technique potentially changes the owner, group, and protection bits of the file, which may come as an unwelcome surprise to anyone trying to
by drfuchs 9y ago
Two issues:
Your technique potentially changes the owner, group, and protection bits of the file, which may come as an unwelcome surprise to anyone trying to use it as a drop-in replacement for directly over-writing the old file. (Worse, there seems to be no way to completely fix this in Unix-land as of the last time I looked back in the 90’s, when we had to pull a feature in FrameMaker over this).
Also, be aware that (many? some?) disk controllers routinely lie about whether their internal caches are actually flushed to physical media, and there’s no way for fsync to be 100% sure that the data has really made it all the way to the spinning platter of rust. Try testing your code by putting it in a loop and repeatedly pulling the power plug out of the wall, and see if you always end up with an uncorrupted file. (Yes, people do this sort of testing on highly fault-tolerant systems.)
- bringtheaction 9y ago> Worse, there seems to be no way to completely fix this in Unix-land as of the last time I looked back in the 90’s, when we had to pull a feature in FrameMaker over this. I had no idea FrameMaker ran on Unix. Apparently it started life on a Sun-2 workstation, and at one point according to Wikipedia, “FrameMaker ran on more than thirteen UNIX platforms, including NeXT Computer's NeXTSTEP and IBM's AIX operating systems”. https://en.wikipedia.org/wiki/Adobe_FrameMaker https://en.wikipedia.org/wiki/Adobe_FrameMaker It was interesting to read how the port to Windows affected FrameMaker, and to see how long Solaris remained a supported platform.
- matthewaveryusa 9y agoIs there anything you can do in software (just thinking Unix) if the hardware lies to you about fsync? Or at the very least detect such devices, or are you at the mercy of simply testing the disk?
- joetucek 9y agoYour OS or filesystem may maintain a blacklist of lying devices (IIRC XFS does so), although such probably isn't exported to the user. Regardless, those blacklists are built via testing, and probably aren't comprehensive. If you absolutely positively must be assured your write is persisted, I recommend fsync, sync, waiting five minutes, fsync and sync again, then top it off with a prayer.
- zaarn 9y agoYou're at the mercy of the disk. The best you can do is assume that anyone who uses your application in production will have enterprise hardware, those (usually) don't lie about fsync (consumer hdd's are somewhat likely to do this) I've also not yet run across a SSD that lies about fsync. The disappointing thing is that most filesystems also blindly assume this. Ext4 is somewhat robust by design but I've heard of atleast one time where a (very) cheap consumer harddisk trashed any filesystem on crash, including ZFS to the point of becoming unmountable.
- michaelmior 9y agoDid you mean lie about fsync?
- zaarn 9y agohmm, yes, thanks. My spelling can go bad when I type a bit fast.
- pjc50 9y ago> Try testing your code by putting it in a loop and repeatedly pulling the power plug out of the wall, and see if you always end up with an uncorrupted file. (Yes, people do this sort of testing on highly fault-tolerant systems I've had "fun" with this on some Windows CE systems, building my own little log-structured database to address the problem. All writes are appends, one disk block at a time, flushing afterwards. Worked well when using TFAT ("transactional" FAT) as the underlying file system, but when tried on a normal PC we discovered a new exciting failure mode: the file ended up with a random chunk of recently-deleted data at the end of it.
- jeffreyrogers 9y agoThanks, I'll look into what can be done about the owner, group, and protection bits. Obviously there's not much that can be done about the disk controllers though.