6 ms·

POSIX v. reality: A position on O_PONIES (2009)

98 points by nolist_policy 3y ago

deleted 3y ago
[deleted]
082349872349872 3y ago
Would that more PM's had had ponies as children: then they might understand that some shiny must-have features are the sort where (a) one occasionally has to call in expensive consultants, just to return them to the (b) steady state where they merely require constant feeding and cleaning-up-after.
- h2odragon 3y ago
  There's a narrow window of "has ponies" and "has to care for the ponies" that learns those lessons.
nolist_policy 3y ago
By the way, Valerie has more great articles on lwn.net: https://lwn.net/Archives/GuestIndex/#Aurora_Henson_Valerie https://lwn.net/Archives/GuestIndex/#Aurora_Henson_Valerie
dale_glass 3y ago
I believe the solution is actually pretty simple, though maybe not easily implementable: Provide new API calls with precisely defined semantics. Rather have this rigamarole with fsync and rename, provide an actual syscall with the actual effect the userspace developers are looking for. Eg, an atomic_replace() syscall that ensures that either a file is replaced with a new, fully written to disk version, or nothing happens. The main problem I see is that this of course would be Linux specific, so of course somebody would build a library to either invoke the syscall or do the fsync/rename mess underneath, and this would of course run into the same exact problem on those systems.
- IshKebab 3y ago
  I don't think there's really any issue with it being Linux specific. There are plenty of Linux specific APIs that people happily use already. Seems to me the bigger issue is the deification of POSIX and UNIX. There's a stupidly large contingent of people that think that they are flawless and must be followed unthinkingly.
  - anonacct37 3y ago
    I've noticed that this week. After spending a little time just reading on how to correctly write posix compliant code that was also guaranteed to do what I want, it's hard to come to any other conclusion than "posix is a last ditch attempt to slap a bandaid on Unix fragmentation by attempting to retcon some lowest common denominator behavior from a couple popular unixish operating systems and calling that a spec". It's not what I would actually call designed. The closest analogy I can think of for non-c programmers is that posix is like if we decided that people should only make websites using javascript that was mutually interpretable by IE6 and Netscape navigator and we occasionally made updates every decade or two. There's really nothing particularly noble or good or correct or elegant about all the API calls with implementation defined semantics. At best it's a necessary evil for compatibility. You can admire the cleverness required to get a single simple c codebase that works correctly on multiple operating systems and future operating systems that conform to posix in creative ways, but only in the way I admire those old school zines that are simultaneously a PDF and a jpeg and a shell script. Clever and ingenious but not actually good engineering design.
    - dale_glass 3y ago
      There's that, and there's a lot of UNIX actually sucks these days and was made for different times. Like signals, for instance. Okay idea when users are writing C from scratch to implement simple things. Horrible pain in modern times, with threads and libraries not playing well with signals.
    - vacuity 3y ago
      If there's a bug in a Unix derivative, does the issue lie with the implementation or the POSIX spec? Answer: It lies with the people wanting a fix! Moral of the story: backwards compatibility is a menace.
kazinator 3y ago
fsync doesn't have to "perform", because it doesn't have to do anything. It just has to wait for some certain events to occur that will eventually occur anyway. Now some people might not like how long it takes; maybe it's waiting for some unnecessary events that are not relevant to this file.
- Joker_vD 3y ago
  Surely fsync() should at least initiate syncing the data to the disk? Otherwise, everything FS-related could theoretically still fit in the memory cache forever, and so fsync() will never return.
- nh2 3y ago
  No. If you don't call fsync(), there's no guarantee that your data will _ever_ be written to disk. Some file systems with some settings may write data to hardware after some delay or periodically, but others may not. fsync() "performs" e.g. an ATA FLUSH command on the underlying hardware.
throwaway09223 3y ago
This article is just plain wrong. The author doesn't seem to understand how filesystems work - specifically the difference between writing data and renaming directory entries. LWN should either issue a correction or take it down. First: The author is confusing atomicity techniques with durability techniques. rename() isn't at all relevant to durability. Never has been. Second: Nothing about using rename() is relevant to what happens when pages are lost on crash. Using rename() or link() has no bearing on dirty writes. This whole section should just be deleted. Third: When pages are lost, they're not back-filled with random unallocated space (and if they were, see the second point above). Filesystems are designed to ensure these regions are zeroed out before the area can be used - either proactively or in fsck. Finally, this quote: "Which brings us to the present day fsync/rename/O_PONIES controversy, in which many file systems developers argue that applications should explicitly call fsync() before renaming a file if they want the file's data to be on disk before the rename takes effect" There is no such controversy because rename() is utterly unrelated to the durability of data on a disk. This is a pure hallucination. What a bizarre article.
- throwaway09223 3y ago
  I read a bit more and figured out why the article was written. In 2009, ext4 (and ONLY ext4) adopted this behavior: https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/commit/?id=dbc85aa9f11d8c13c15527d43a3def8d7beffdc8 https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.g... I don't think any other unix filesystems do this. It's not posix, and it's not behavior that any apps should be depending on if they care about their data. I would suggest updating the LWN article to clarify that this is a piece specifically about ext4 -- not POSIX, or even Linux in general.
  - nh2 3y ago
    What you say is correct, and ext4's current default behaviour can be quite confusing: * ext4 turns rename() into an invisible fsync(), if the target file existed. * ext4 turns close() into an invisible fsync(), if the opened file existed. This leads to surprising results: If you unzip something the first time, it's nice and fast. If you unzip the same file again, force-overwriting existing files, suddenly it's 10x slower. You now spend hours finding out why, and start cargo-culting vodoo that doing `rm` before makes it inexplicably fast again. Invisible fsync()s that the application programmer cannot turn off are quite bad. My speculation is that Ted Ts'o agrees with that in principle, but that he got tired of having to explain people that if they don't fsync(), there are no guarantees; so he added this hack that makes 95% of the cases go away with surprising special-case behaviour.
tlb 3y ago
On modern systems, crashes happen maybe once a year, while outright disk failures or undetected memory errors corrupting disk blocks happen maybe every 100 years. So if the chance of corruption from a crash is 1%, the overall risks are comparable. Given that, I’d probably choose performance over guaranteed crash recovery for most systems.

98 points by nolist_policy 3y ago

deleted 3y ago

[deleted]

Would that more PM's had had ponies as children: then they might understand that some shiny must-have features are the sort where (a) one occasionally has to call in expensive consultants, just to return them to the (b) steady state where they merely require constant feeding and cleaning-up-after.

h2odragon 3y ago
There's a narrow window of "has ponies" and "has to care for the ponies" that learns those lessons.

nolist_policy 3y ago

By the way, Valerie has more great articles on lwn.net: https://lwn.net/Archives/GuestIndex/#Aurora_Henson_Valerie https://lwn.net/Archives/GuestIndex/#Aurora_Henson_Valerie

dale_glass 3y ago

I believe the solution is actually pretty simple, though maybe not easily implementable: Provide new API calls with precisely defined semantics. Rather have this rigamarole with fsync and rename, provide an actual syscall with the actual effect the userspace developers are looking for. Eg, an atomic_replace() syscall that ensures that either a file is replaced with a new, fully written to disk version, or nothing happens. The main problem I see is that this of course would be Linux specific, so of course somebody would build a library to either invoke the syscall or do the fsync/rename mess underneath, and this would of course run into the same exact problem on those systems.

IshKebab 3y ago
I don't think there's really any issue with it being Linux specific. There are plenty of Linux specific APIs that people happily use already. Seems to me the bigger issue is the deification of POSIX and UNIX. There's a stupidly large contingent of people that think that they are flawless and must be followed unthinkingly.
- anonacct37 3y ago
  I've noticed that this week. After spending a little time just reading on how to correctly write posix compliant code that was also guaranteed to do what I want, it's hard to come to any other conclusion than "posix is a last ditch attempt to slap a bandaid on Unix fragmentation by attempting to retcon some lowest common denominator behavior from a couple popular unixish operating systems and calling that a spec". It's not what I would actually call designed. The closest analogy I can think of for non-c programmers is that posix is like if we decided that people should only make websites using javascript that was mutually interpretable by IE6 and Netscape navigator and we occasionally made updates every decade or two. There's really nothing particularly noble or good or correct or elegant about all the API calls with implementation defined semantics. At best it's a necessary evil for compatibility. You can admire the cleverness required to get a single simple c codebase that works correctly on multiple operating systems and future operating systems that conform to posix in creative ways, but only in the way I admire those old school zines that are simultaneously a PDF and a jpeg and a shell script. Clever and ingenious but not actually good engineering design.
  - dale_glass 3y ago
    There's that, and there's a lot of UNIX actually sucks these days and was made for different times. Like signals, for instance. Okay idea when users are writing C from scratch to implement simple things. Horrible pain in modern times, with threads and libraries not playing well with signals.
  - vacuity 3y ago
    If there's a bug in a Unix derivative, does the issue lie with the implementation or the POSIX spec? Answer: It lies with the people wanting a fix! Moral of the story: backwards compatibility is a menace.

kazinator 3y ago

fsync doesn't have to "perform", because it doesn't have to do anything. It just has to wait for some certain events to occur that will eventually occur anyway. Now some people might not like how long it takes; maybe it's waiting for some unnecessary events that are not relevant to this file.

Joker_vD 3y ago
Surely fsync() should at least initiate syncing the data to the disk? Otherwise, everything FS-related could theoretically still fit in the memory cache forever, and so fsync() will never return.
nh2 3y ago
No. If you don't call fsync(), there's no guarantee that your data will _ever_ be written to disk. Some file systems with some settings may write data to hardware after some delay or periodically, but others may not. fsync() "performs" e.g. an ATA FLUSH command on the underlying hardware.

throwaway09223 3y ago

This article is just plain wrong. The author doesn't seem to understand how filesystems work - specifically the difference between writing data and renaming directory entries. LWN should either issue a correction or take it down. First: The author is confusing atomicity techniques with durability techniques. rename() isn't at all relevant to durability. Never has been. Second: Nothing about using rename() is relevant to what happens when pages are lost on crash. Using rename() or link() has no bearing on dirty writes. This whole section should just be deleted. Third: When pages are lost, they're not back-filled with random unallocated space (and if they were, see the second point above). Filesystems are designed to ensure these regions are zeroed out before the area can be used - either proactively or in fsck. Finally, this quote: "Which brings us to the present day fsync/rename/O_PONIES controversy, in which many file systems developers argue that applications should explicitly call fsync() before renaming a file if they want the file's data to be on disk before the rename takes effect" There is no such controversy because rename() is utterly unrelated to the durability of data on a disk. This is a pure hallucination. What a bizarre article.

throwaway09223 3y ago
I read a bit more and figured out why the article was written. In 2009, ext4 (and ONLY ext4) adopted this behavior: https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/commit/?id=dbc85aa9f11d8c13c15527d43a3def8d7beffdc8 https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.g... I don't think any other unix filesystems do this. It's not posix, and it's not behavior that any apps should be depending on if they care about their data. I would suggest updating the LWN article to clarify that this is a piece specifically about ext4 -- not POSIX, or even Linux in general.
- nh2 3y ago
  What you say is correct, and ext4's current default behaviour can be quite confusing: * ext4 turns rename() into an invisible fsync(), if the target file existed. * ext4 turns close() into an invisible fsync(), if the opened file existed. This leads to surprising results: If you unzip something the first time, it's nice and fast. If you unzip the same file again, force-overwriting existing files, suddenly it's 10x slower. You now spend hours finding out why, and start cargo-culting vodoo that doing `rm` before makes it inexplicably fast again. Invisible fsync()s that the application programmer cannot turn off are quite bad. My speculation is that Ted Ts'o agrees with that in principle, but that he got tired of having to explain people that if they don't fsync(), there are no guarantees; so he added this hack that makes 95% of the cases go away with surprising special-case behaviour.

tlb 3y ago

On modern systems, crashes happen maybe once a year, while outright disk failures or undetected memory errors corrupting disk blocks happen maybe every 100 years. So if the chance of corruption from a crash is 1%, the overall risks are comparable. Given that, I’d probably choose performance over guaranteed crash recovery for most systems.