2 ms·

Have you tried proposing this to be included in POSIX? Or tried implementing this in the VFS layer in Linux? I imagine all of these could be xattrs.

by sargun 10y ago

Animats 10y ago
The usual objections are: - Unit file updates are not needed because of (complex non-portable workaround for atomic file replacement). - It's not backwards compatible with NFS. - Lock files are good enough. - If I want to seek while adding to a log file and rewrite in the middle, I should be allowed to do so. Database people like the idea of a callback when data is safely on disk. They work hard to get the effect of that, and struggle with "flush" calls either flushing too much, or return when the data reaches the drive controller, not the disk surface. Doing this right needs file system and driver support, though.
- daurnimator 10y ago
  > - Unit file updates are not needed because of (complex non-portable workaround for atomic file replacement). I agree with all your points except this. What is wrong with creating the file somewhere else and using rename() to atomically overwrite?
  - FooBarWidget 10y ago
    Because that is not portable and/or has performance problems. I'm not kidding you. I know the conventional wisdom is that a rename is enough. It is not. Depending on the filesystem, you may have to fsync() the directory entry too, or fsync the parent directory entry too. Or on other filesystems, don't do these because doing these may cause problems. And do all this in the right order, because if you get it slightly wrong then it doesn't do what you want in the event of a crash. The problem is maddening. There is an article out there that explains all this in much more detail. I think it is this one but I don't quite remember whether it is the same as the one I read: http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/ http://blog.httrack.com/blog/2013/11/15/everything-you-alway... The performance problem comes from the fact that you need two fscyncs.
tedunangst 10y ago
Most of this is pretty easy to achieve. "Unit" files are just a temp file and rename away. Append only files have existed in BSD since forever (and honestly, they're more pita than anything).

Have you tried proposing this to be included in POSIX? Or tried implementing this in the VFS layer in Linux? I imagine all of these could be xattrs.

by sargun 10y ago

Have you tried proposing this to be included in POSIX? Or tried implementing this in the VFS layer in Linux? I imagine all of these could be xattrs.

Animats 10y ago

The usual objections are: - Unit file updates are not needed because of (complex non-portable workaround for atomic file replacement). - It's not backwards compatible with NFS. - Lock files are good enough. - If I want to seek while adding to a log file and rewrite in the middle, I should be allowed to do so. Database people like the idea of a callback when data is safely on disk. They work hard to get the effect of that, and struggle with "flush" calls either flushing too much, or return when the data reaches the drive controller, not the disk surface. Doing this right needs file system and driver support, though.

daurnimator 10y ago
> - Unit file updates are not needed because of (complex non-portable workaround for atomic file replacement). I agree with all your points except this. What is wrong with creating the file somewhere else and using rename() to atomically overwrite?
- FooBarWidget 10y ago
  Because that is not portable and/or has performance problems. I'm not kidding you. I know the conventional wisdom is that a rename is enough. It is not. Depending on the filesystem, you may have to fsync() the directory entry too, or fsync the parent directory entry too. Or on other filesystems, don't do these because doing these may cause problems. And do all this in the right order, because if you get it slightly wrong then it doesn't do what you want in the event of a crash. The problem is maddening. There is an article out there that explains all this in much more detail. I think it is this one but I don't quite remember whether it is the same as the one I read: http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/ http://blog.httrack.com/blog/2013/11/15/everything-you-alway... The performance problem comes from the fact that you need two fscyncs.

tedunangst 10y ago

Most of this is pretty easy to achieve. "Unit" files are just a temp file and rename away. Append only files have existed in BSD since forever (and honestly, they're more pita than anything).