5 ms·
fsync() guarantees that writes have hit the disk. But is there a guarantee about what's written before an fsync()? Can it be anywhere between "nothing" and "eve
by pradn 3y ago
fsync() guarantees that writes have hit the disk. But is there a guarantee about what's written before an fsync()? Can it be anywhere between "nothing" and "everything"? I suppose this must be a loose guarantee if the "write-back" parameter can be tweaked at will.
- nolist_policy 3y ago> Can it be anywhere between "nothing" and "everything"? Yes, and that won't change because the hardware with it's own buffers behaves the same way.
- toast0 3y ago> Can it be anywhere between "nothing" and "everything"? Yes, and there's not (generally) any ordering constraint, either. The last thing you wrote may be persisted, and not the first, etc.
- loeg 3y agoThere's some user input to this via posix_fadvise POSIX_FADV_DONTNEED but it doesn't guarantee anything.
- vlovich123 3y agoIf you're generating an immutable file, a common technique is to play fsync + rename tricks although I think a more modern technique would be: 1. open directory (dir_fd) 2. create an unnamed O_TMPFILE file (file_fd) 3. write to file_fd 4. fdatasync(file_fd) 5. linkat(dir_fd, "", dir_fd, "file name", AT_EMPTY_PATH) 5. fsync(dir_fd) This should guarantee that either "file name" will have the old contents or the new contents and no transient version is observed. The "old" mechanism is similar in that you write out to a temporary sibling and rename (these days you'd use RENAME_EXCHANGE w/ renameat2 to guarantee the atomicity or get an error) with the differentiating difference being that the temporary data could be observed on the filesystem / left around if you have a machine reboot.
- vlovich123 3y agoCorrection to typo: 5. linkat(file_fd, "", dir_fd, "file name", AT_EMPTY_PATH)
- pradn 3y agoThank you - it's baffling to me that something as basic as saving some data to a file has become so convoluted!
- vlovich123 3y agoIt was always this convoluted because the POSIX APIs suck and the Linux kernel still refuses to provide an API to write large amounts of data transactionally so you have to know the magic incantation for doing it. And note this is when you have all the data up-front. Doing an append is significantly expensive or requires specific filesystem support for reflinks.
- fao_ 3y agoSSDs and HDD on-board firmware will actually cache-locally and then lie about if it's been written to the disk or not. Pretty much every level of the stack caches and lies about it to the point that if you do a deep dive into "has information been written to disk or not" the answer comes up as "this is impossible to verify".