9 ms·

POSIX is also so old and essential that it's hard to imagine an alternative.

by __loam 2y ago

jcranmer 2y ago
Not really, there's been lots of APIs that have improved on the POSIX model. The kind of model I prefer is something based on atomicity. Most applications can get by with file-level atomicity--make whole file read/writes atomic with a copy-on-write model, and you can eliminate whole classes of filesystem bugs pretty quickly. (Note that something like writeFileAtomic is already a common primitive in many high-level filesystem APIs, and it's something that's already easily buildable with regular POSIX APIs). For cases like logging, you can extend the model slightly with atomic appends, where the only kind of write allowed is to atomically append a chunk of data to the file (so readers can only possibly either see no new data or the entire chunk of data at once). I'm less knowledgeable about the way DBs interact with the filesystem, but there the solution is probably ditching the concept of the file stream entirely and just treating files as a sparse map of offsets to blocks, which can be atomically updated. (My understanding is that DBs basically do this already, except that "atomically updated" is difficult with the current APIs).
- Joker_vD 2y ago
  > Most applications can get by with file-level atomicity--make whole file read/writes atomic with a copy-on-write model, and you can eliminate whole classes of filesystem bugs pretty quickly. int fd = open(".config", O_RDWR | O_CREAT | O_SYNC_ON_CLOSE, 0o666); // effects of calls to write(2)/etc. are invisible through any other file description // until the close(2) is called on all descriptors to this file description. close(fd); So now you can watch for e.g. either IN_MODIFY or IN_CLOSE_WRITE (and you don't need to balance it with IN_OPEN), it doesn't matter, you'll never see partial updates... would be nice!
  - BobbyTables2 2y ago
    Surely this can’t always be true? What happens when a lot of data is written and exceeds the dirty threshold?
    - Joker_vD 2y ago
      It gets written on the disk but into different inodes, I imagine.
- kragen 2y ago
  It's not hard to design a less bug-prone API that would enable you to do everything the POSIX file API permits and admits equally-high-performance implementations. But making that new API a replacement for the POSIX API would require rewriting essentially all of the software that somebody cares about to use your new, better API instead of the POSIX API. This is probably only feasible in practice for small embedded systems with a fairly small universe of software.
  - josephg 2y ago
    You could do a phased transition, where both the legacy posix api and the new api are available. This has already happened with a lot of the old C standard library. Old, unsafe functions like strcpy were gradually replaced by safer alternatives like strncpy. Database developers don’t want the complexity or poor performance of posix. It’s wild to me that we still don’t have any alternative to fsync in Linux that can act as a barrier without also flushing caches at the same time.
    - namibj 2y ago
      NVMe has no barrier that doesn't flush the pipeline/ringbuffer of IO requests submitted to it :(
      - kragen 2y ago
        The kernel could implement a non-flushing barrier, even if the underlying device doesn't. You could even do it without any barrier support at all from the underlying device, as long as it reliably tells you when each request has completed; you just don't send it any requests from after the barrier until all the requests before the barrier have completed.
        ryao 2y ago
        That would not work as you describe it. The device will return completion upon the writes reaching its cache. You need a flush to ensure that the data reaches stable storage. You could probably abuse Force Unit Access to make it work by marking all IOs as Force Unit Access, but a number of buggy devices do not implement FUA properly, which defeats the purpose of using it. That would be why Microsoft disabled the NTFS feature that uses FUA on commodity hardware: https://learn.microsoft.com/en-us/windows/win32/fileio/deploying-transactional-ntfs https://learn.microsoft.com/en-us/windows/win32/fileio/deplo... What you seem to want is FreeBSD’s UFS2 Softupdates that uses force unit access to avoid the need for flushes for metadata updates. It has the downside that it is unreliable on hardware that does not implement FUA properly. Also, UFS2 softupdates does not actually implement do anything to protect data when fsync(2) is called if this mailing list email is accurate: https://lists.freebsd.org/pipermail/freebsd-fs/2011-November/013086.html https://lists.freebsd.org/pipermail/freebsd-fs/2011-November... As pjd said: > Synchronous writes (or BIO_FLUSH) are needed to handle O_SYNC/fsync(2) properly, which UFS currently doesn't care about. That said, avoiding flushes for a fsync(2) would require doing FUA on all IOs. Presumably, this is not done because it would make all requests take longer all the time, raising queue depths and causing things to have to wait for queue limits more often, killing performance. Raising the OS queue depth to compensate would not work since SATA has a maximum queue depth of 32, although it might work for NVMe where the maximum queue depth is 65536, if keeping track of an increased number of inflight IOs does not cause additional issues at the storage devices (such as IOs that never complete as long as the device is kept busy because the device will keep reordering them to the end of the queue). Using FUA only on metadata as is done in UFS2 soft updates improves performance by eliminating the need for journalling in all cases but the case of space usage, which still needs journalling (or fsck after power loss if you choose to forgo it).
- timewizard 2y ago
  > make whole file read/writes atomic with a copy-on-write model, I have many files that are several GB. Are you sure this is a good idea? What if my application only requires best effort? > eliminate whole classes of filesystem bugs pretty quickly. Block level deduplication is notoriously difficult. > where the only kind of write allowed is to atomically append a chunk of data to the file Which sounds good until you think about the complications involved in block oriented storage medium. You're stuck with RMW whether you think you're strictly appending or not.
  - josephg 2y ago
    It doesn’t have to be one or the other. Developers could decide by passing flags to open. But even then, doing atomic writes of multi gigabyte files doesn’t sound that hard to implement efficiently. Just write to disk first and update the metadata atomically at the end. Or whenever you choose to as a programmer. The downside is that, when overwriting, you’ll need enough free space to store both the old and new versions of your data. But I think that’s usually a good trade off. It would allow all sorts of useful programs to be written easily - like an atomic mode for apt, where packages either get installed or not installed. But they can’t be half installed.
    - ryao 2y ago
      Packages consist of multiple files. An atomic file write would not allow packages to be either installed or not installed by APT.
      - huntaub 2y ago
        This is kind of an interesting thought that more mirrors how Docker uses OverlayFS to track changes to the entire file system. No need for new file APIs.
        ryao 2y ago
        It can also use ZFS to do this.
- ryao 2y ago
  Writes in the POSIX API can be atomic depending on the underlying filesystem. For example, small writes on ZFS through the POSIX API are atomic since they either happen in their entirety or they do not (during power failure), although if the writes are big enough (spanning many records), they are split into separate transactions and partial writes are then possible: https://github.com/openzfs/zfs/blob/34205715e1544d343f9a64149b17c371282306ae/module/zfs/zfs_vnops.c#L755 https://github.com/openzfs/zfs/blob/34205715e1544d343f9a6414... Writes on ZFS cease to be atomic around approximately 32MB in size if I read the code correctly.
MisterTea 2y ago
I use Plan 9 regularly and while its Unix heritage is there, it most certainly isn't Unix and completely does away with POSIX.
emmelaich 2y ago
Some of the problems transcend POSIX. Someone I know maintains a non-relational db on IBM mainframes. When diving into a data issue, he was gob-smacked to find out that sync'd writes did not necessarily make it to the disk. They were cached in the drive memory and (I think) the disk controller memory. If all failed, data was lost.
- mangamadaiyan 2y ago
  This is precisely why well-designed enterprise-grade storage systems disable the drive cache and rely upon some variant of striping to achieve good I/O performance.
- hackit2 2y ago
  Just wait till he has to deal with raid controllers.

POSIX is also so old and essential that it's hard to imagine an alternative.

by __loam 2y ago

POSIX is also so old and essential that it's hard to imagine an alternative.

Not really, there's been lots of APIs that have improved on the POSIX model. The kind of model I prefer is something based on atomicity. Most applications can get by with file-level atomicity--make whole file read/writes atomic with a copy-on-write model, and you can eliminate whole classes of filesystem bugs pretty quickly. (Note that something like writeFileAtomic is already a common primitive in many high-level filesystem APIs, and it's something that's already easily buildable with regular POSIX APIs). For cases like logging, you can extend the model slightly with atomic appends, where the only kind of write allowed is to atomically append a chunk of data to the file (so readers can only possibly either see no new data or the entire chunk of data at once). I'm less knowledgeable about the way DBs interact with the filesystem, but there the solution is probably ditching the concept of the file stream entirely and just treating files as a sparse map of offsets to blocks, which can be atomically updated. (My understanding is that DBs basically do this already, except that "atomically updated" is difficult with the current APIs).

Joker_vD 2y ago
> Most applications can get by with file-level atomicity--make whole file read/writes atomic with a copy-on-write model, and you can eliminate whole classes of filesystem bugs pretty quickly. int fd = open(".config", O_RDWR | O_CREAT | O_SYNC_ON_CLOSE, 0o666); // effects of calls to write(2)/etc. are invisible through any other file description // until the close(2) is called on all descriptors to this file description. close(fd); So now you can watch for e.g. either IN_MODIFY or IN_CLOSE_WRITE (and you don't need to balance it with IN_OPEN), it doesn't matter, you'll never see partial updates... would be nice!
- BobbyTables2 2y ago
  Surely this can’t always be true? What happens when a lot of data is written and exceeds the dirty threshold?
  - Joker_vD 2y ago
    It gets written on the disk but into different inodes, I imagine.
kragen 2y ago
It's not hard to design a less bug-prone API that would enable you to do everything the POSIX file API permits and admits equally-high-performance implementations. But making that new API a replacement for the POSIX API would require rewriting essentially all of the software that somebody cares about to use your new, better API instead of the POSIX API. This is probably only feasible in practice for small embedded systems with a fairly small universe of software.
- josephg 2y ago
  You could do a phased transition, where both the legacy posix api and the new api are available. This has already happened with a lot of the old C standard library. Old, unsafe functions like strcpy were gradually replaced by safer alternatives like strncpy. Database developers don’t want the complexity or poor performance of posix. It’s wild to me that we still don’t have any alternative to fsync in Linux that can act as a barrier without also flushing caches at the same time.
  - namibj 2y ago
    NVMe has no barrier that doesn't flush the pipeline/ringbuffer of IO requests submitted to it :(
    - kragen 2y ago
      The kernel could implement a non-flushing barrier, even if the underlying device doesn't. You could even do it without any barrier support at all from the underlying device, as long as it reliably tells you when each request has completed; you just don't send it any requests from after the barrier until all the requests before the barrier have completed.
      - ryao 2y ago
        That would not work as you describe it. The device will return completion upon the writes reaching its cache. You need a flush to ensure that the data reaches stable storage. You could probably abuse Force Unit Access to make it work by marking all IOs as Force Unit Access, but a number of buggy devices do not implement FUA properly, which defeats the purpose of using it. That would be why Microsoft disabled the NTFS feature that uses FUA on commodity hardware: https://learn.microsoft.com/en-us/windows/win32/fileio/deploying-transactional-ntfs https://learn.microsoft.com/en-us/windows/win32/fileio/deplo... What you seem to want is FreeBSD’s UFS2 Softupdates that uses force unit access to avoid the need for flushes for metadata updates. It has the downside that it is unreliable on hardware that does not implement FUA properly. Also, UFS2 softupdates does not actually implement do anything to protect data when fsync(2) is called if this mailing list email is accurate: https://lists.freebsd.org/pipermail/freebsd-fs/2011-November/013086.html https://lists.freebsd.org/pipermail/freebsd-fs/2011-November... As pjd said: > Synchronous writes (or BIO_FLUSH) are needed to handle O_SYNC/fsync(2) properly, which UFS currently doesn't care about. That said, avoiding flushes for a fsync(2) would require doing FUA on all IOs. Presumably, this is not done because it would make all requests take longer all the time, raising queue depths and causing things to have to wait for queue limits more often, killing performance. Raising the OS queue depth to compensate would not work since SATA has a maximum queue depth of 32, although it might work for NVMe where the maximum queue depth is 65536, if keeping track of an increased number of inflight IOs does not cause additional issues at the storage devices (such as IOs that never complete as long as the device is kept busy because the device will keep reordering them to the end of the queue). Using FUA only on metadata as is done in UFS2 soft updates improves performance by eliminating the need for journalling in all cases but the case of space usage, which still needs journalling (or fsck after power loss if you choose to forgo it).
timewizard 2y ago
> make whole file read/writes atomic with a copy-on-write model, I have many files that are several GB. Are you sure this is a good idea? What if my application only requires best effort? > eliminate whole classes of filesystem bugs pretty quickly. Block level deduplication is notoriously difficult. > where the only kind of write allowed is to atomically append a chunk of data to the file Which sounds good until you think about the complications involved in block oriented storage medium. You're stuck with RMW whether you think you're strictly appending or not.
- josephg 2y ago
  It doesn’t have to be one or the other. Developers could decide by passing flags to open. But even then, doing atomic writes of multi gigabyte files doesn’t sound that hard to implement efficiently. Just write to disk first and update the metadata atomically at the end. Or whenever you choose to as a programmer. The downside is that, when overwriting, you’ll need enough free space to store both the old and new versions of your data. But I think that’s usually a good trade off. It would allow all sorts of useful programs to be written easily - like an atomic mode for apt, where packages either get installed or not installed. But they can’t be half installed.
  - ryao 2y ago
    Packages consist of multiple files. An atomic file write would not allow packages to be either installed or not installed by APT.
    - huntaub 2y ago
      This is kind of an interesting thought that more mirrors how Docker uses OverlayFS to track changes to the entire file system. No need for new file APIs.
      - ryao 2y ago
        It can also use ZFS to do this.
ryao 2y ago
Writes in the POSIX API can be atomic depending on the underlying filesystem. For example, small writes on ZFS through the POSIX API are atomic since they either happen in their entirety or they do not (during power failure), although if the writes are big enough (spanning many records), they are split into separate transactions and partial writes are then possible: https://github.com/openzfs/zfs/blob/34205715e1544d343f9a64149b17c371282306ae/module/zfs/zfs_vnops.c#L755 https://github.com/openzfs/zfs/blob/34205715e1544d343f9a6414... Writes on ZFS cease to be atomic around approximately 32MB in size if I read the code correctly.

MisterTea 2y ago

I use Plan 9 regularly and while its Unix heritage is there, it most certainly isn't Unix and completely does away with POSIX.

emmelaich 2y ago

Some of the problems transcend POSIX. Someone I know maintains a non-relational db on IBM mainframes. When diving into a data issue, he was gob-smacked to find out that sync'd writes did not necessarily make it to the disk. They were cached in the drive memory and (I think) the disk controller memory. If all failed, data was lost.

mangamadaiyan 2y ago
This is precisely why well-designed enterprise-grade storage systems disable the drive cache and rely upon some variant of striping to achieve good I/O performance.
hackit2 2y ago
Just wait till he has to deal with raid controllers.