6 ms·

I believe the solution is actually pretty simple, though maybe not easily implementable: Provide new API calls with precisely defined semantics. Rather have t

I believe the solution is actually pretty simple, though maybe not easily implementable: Provide new API calls with precisely defined semantics. Rather have this rigamarole with fsync and rename, provide an actual syscall with the actual effect the userspace developers are looking for. Eg, an atomic_replace() syscall that ensures that either a file is replaced with a new, fully written to disk version, or nothing happens. The main problem I see is that this of course would be Linux specific, so of course somebody would build a library to either invoke the syscall or do the fsync/rename mess underneath, and this would of course run into the same exact problem on those systems.

IshKebab 3y ago
I don't think there's really any issue with it being Linux specific. There are plenty of Linux specific APIs that people happily use already. Seems to me the bigger issue is the deification of POSIX and UNIX. There's a stupidly large contingent of people that think that they are flawless and must be followed unthinkingly.
- anonacct37 3y ago
  I've noticed that this week. After spending a little time just reading on how to correctly write posix compliant code that was also guaranteed to do what I want, it's hard to come to any other conclusion than "posix is a last ditch attempt to slap a bandaid on Unix fragmentation by attempting to retcon some lowest common denominator behavior from a couple popular unixish operating systems and calling that a spec". It's not what I would actually call designed. The closest analogy I can think of for non-c programmers is that posix is like if we decided that people should only make websites using javascript that was mutually interpretable by IE6 and Netscape navigator and we occasionally made updates every decade or two. There's really nothing particularly noble or good or correct or elegant about all the API calls with implementation defined semantics. At best it's a necessary evil for compatibility. You can admire the cleverness required to get a single simple c codebase that works correctly on multiple operating systems and future operating systems that conform to posix in creative ways, but only in the way I admire those old school zines that are simultaneously a PDF and a jpeg and a shell script. Clever and ingenious but not actually good engineering design.
  - dale_glass 3y ago
    There's that, and there's a lot of UNIX actually sucks these days and was made for different times. Like signals, for instance. Okay idea when users are writing C from scratch to implement simple things. Horrible pain in modern times, with threads and libraries not playing well with signals.
  - vacuity 3y ago
    If there's a bug in a Unix derivative, does the issue lie with the implementation or the POSIX spec? Answer: It lies with the people wanting a fix! Moral of the story: backwards compatibility is a menace.
Gibbon1 3y ago
I think that was the problem with how Linux implemented pselect() circa 10 years ago. I think pselect fixes the race condition you get when you set a timer then call select(). Linux's implementation was a function that set a timer and then called select(). ... slam head on keyboard. So I think that fear is valid that they'll implement the API without the guarantees. Myself I'm annoyed with the reckless push always to remove guarantees in return for 'performance'.
- leoc 3y ago
  I'm starting to come round to the belief that the only robust remedy is to release the chaos monkey: have the kernel delete all the state of the filesystem driver at randomly-chosen intervals, a few hours apart on average, and force the driver to recover itself.
  - bcrl 3y ago
    That is pretty close to what those of us building robust storage systems in the real world have to do. At a previous employer, we had test suites that exercised all kinds of corner cases by triggering failover and system reboots in the middle of heavy persistent messaging workloads. Filesystems have other horrors that you learn about during testing. I had one test case where it would take ext4 80 seconds to write out an 8MB file after a fresh mount of an 8TB filesystem. Free space was fragmented in just the right way that we hit the single threaded reading of block groups and bitmaps in the kernel and it took forever to get that data off the disk array.
    - Gibbon1 3y ago
      I've done the same with embedded drivers, randomly flip bits in the drivers state and see what happens. Does it recover, throw fault, hang, or explode like a bomb. I got that from a hardware designer talking about robust state machines. Any random illegal state should sequence back to a known good state.
      - zbentley 3y ago
tedunangst 3y ago
The problem is every filesystem would implement a different version of fsync_that_works or rename_for_safety and then applications will have to call all of them, and then people will start building a compat framework that tries to assemble the optimal sequence of calls for each filesystem except it won't work in every case and so a new fsync_but_no_cheating function will be proposed.
- appplication 3y ago
  Does this suggest that the idea of an overarching fsync/rename is perhaps being proposed at the wrong level of abstraction? If it can’t be reliably implemented at every subsequent level, what’s the point in having it at this level?
  - tedunangst 3y ago
    Different filesystems have different on disk structures, with varying degrees of complexity and cost in getting those things on disk. Everything can reliably get everything written eventually, but if you want to fully utilize your disk bandwidth, you only want to flush the data that can get there fast, which is going to vary by filesystem. And different applications have different durability requirements.
    - layer8 3y ago
      This is still no argument to not define syscalls for the semantics that applications typically care about. If it is slow on a particular file system, then so be it, it’s still the semantics the application requires.
      - tedunangst 3y ago
        So call fsync.
- dale_glass 3y ago
  The trick is not allowing such a thing. Don't have something vague like "fsync_that_works" that could be implemented arbitrarily. Have a function that is documented to implement a highly specific contract, like "writes all pending blocks of this file to disk, and ensures they'll be there if there's a crash after the function returns".
Animats 3y ago
I've proposed this informally before, along these lines: - Unit files. You create, you write, you close, and then others can read the new version. Until the original writer closes and gets a good close status, nobody else can read it. If the program aborts or the system crashes before closing cleanly, the file reverts. All readers see a fully written file. This is the default. It's what most programs need. Replacing a file by creating a new one on top of it is both permitted and an atomic operation. UCLA-LOCUS and some IBM systems that followed worked that way. - Temporary files. Disappear on system crash. - Log files. Append-only. All readers are guaranteed to see an end of file position that corresponds to the end of a previous write. Usually the most recent write, but buffering may make log file reading run a little behind. - Managed files. Read, write, share. Write operations return two completions, probably via some async mechanism. The first completion means "buffer contents taken". The second completion means "committed to storage that will survive a crash". Database systems would use this, but few other programs would bother. This tells the database what it really needs to know - when is the data safe? That tells the database when it can commit a transaction. The database can do other things, including more I/O, while waiting for commitment. "fsync" is really a clunky way of getting that second completion.
- the8472 3y ago
  I think most of those things are implementable in userspace with the right combination of O_TMPFILE, rename, fsync, fsync(dir), sync_file_range and io_uring. So all that's needed is encapsulating it in a library that provides those different abstractions. Well ok, we could also use an additional flag for linkat() that allows atomically replacing the target to close that tiny window where a temporary file might get left behind after a crash.
  - vlovich123 3y ago
    Actually, a good chunk of that is available via memfd + linkat. - Unit files: memfd, fdatasync, linkat, fsync(dir) - Temporary files. memfd, preferably within /tmp to avoid any disk I/O unless you have a particularly large file to buffer - Log files. This is trickier. Probably requires running on a fs that supports reflinks and creating a new memfd that’s a true clone of the original file, appending, & then replacing it. The one piece that is incompatible with this is that other file descriptors are stale and you have to reopen to see new data but that’s at odds with APIs available today - either readers can see partial writes or you have to have a new file. There have been proposals to allow buffering writes but delaying publishing the metadata acking the data but that hasn’t gone anywhere unfortunately. - Managed files. I’m not sure that completions are the right contract here because technically the kernel could defer that completion indefinitely whereas databases typical need some kind of latency bounds of “OK - I really want this to commit to disk nowish”.
Denvercoder9 3y ago
I don't see how new APIs will help here. The problem is that the semantics that application developers want is expensive (performance-wise), so they use a similar API that doesn't guarantee those semantics but provides them in practice 99% of the time. You could create a new set of API's, but it's really hard to implement any API in a way that doesn't give extra, unspecified semantics in practice, so what prevents application developers from making the same choice?
- Filligree 3y ago
  They aren't that expensive. ZFS provides them by default, in that no operations are ever reordered in a user-visible fashion even without fsync. That's a stronger guarantee than we're asking for here.
throwaway09223 3y ago
But there is no rigamarole. The article is just super confused. Atomicity is orthogonal to durability. rename() is atomic and always has been atomic. It is atomic without guaranteeing durability. In fact, to guarantee durability after rename you must: 1) rename(a, b) 2) open(the parent dir) 3) fsync(the parent dir) This is to guarantee durability of the metadata written by rename() in the dirent. All of this is totally separate from guaranteeing the durability of the data in the file, which requires its own separate fsync() on the file itself. Atomicity and durability are never the same. This article is gibberish.
- aidenn0 3y ago
  The phrase "atomic across crashes" has a meaning that is distinct from durability. It is also different from atomic across a live system.
  - throwaway09223 3y ago
    The phrase "atomic across crashes" is meaningless. Atomicity deals with the running system state at the moment a system call executes. It has nothing to do with crashes (which are not atomic in and of themselves), or what bits actually end up on disk.
    - aidenn0 3y ago
      Are you being willfully obtuse, or do you honestly believe that the word "atomicity" only applies to such a narrow context?
      - throwaway09223 3y ago
        The latter. What do you think it means?
        rcxdude 3y ago
        Atomic simply means indivisible: you don't observe half of an atomic operation. Of course it strongly depends on where you are observing the operation from, and that does not necessarily just mean 'a running system', hence OP specifying that they wish for the operation to be atomic even when observed from a system before and after a crash.

IshKebab 3y ago

I don't think there's really any issue with it being Linux specific. There are plenty of Linux specific APIs that people happily use already. Seems to me the bigger issue is the deification of POSIX and UNIX. There's a stupidly large contingent of people that think that they are flawless and must be followed unthinkingly.

anonacct37 3y ago
I've noticed that this week. After spending a little time just reading on how to correctly write posix compliant code that was also guaranteed to do what I want, it's hard to come to any other conclusion than "posix is a last ditch attempt to slap a bandaid on Unix fragmentation by attempting to retcon some lowest common denominator behavior from a couple popular unixish operating systems and calling that a spec". It's not what I would actually call designed. The closest analogy I can think of for non-c programmers is that posix is like if we decided that people should only make websites using javascript that was mutually interpretable by IE6 and Netscape navigator and we occasionally made updates every decade or two. There's really nothing particularly noble or good or correct or elegant about all the API calls with implementation defined semantics. At best it's a necessary evil for compatibility. You can admire the cleverness required to get a single simple c codebase that works correctly on multiple operating systems and future operating systems that conform to posix in creative ways, but only in the way I admire those old school zines that are simultaneously a PDF and a jpeg and a shell script. Clever and ingenious but not actually good engineering design.
- dale_glass 3y ago
  There's that, and there's a lot of UNIX actually sucks these days and was made for different times. Like signals, for instance. Okay idea when users are writing C from scratch to implement simple things. Horrible pain in modern times, with threads and libraries not playing well with signals.
- vacuity 3y ago
  If there's a bug in a Unix derivative, does the issue lie with the implementation or the POSIX spec? Answer: It lies with the people wanting a fix! Moral of the story: backwards compatibility is a menace.

Gibbon1 3y ago

I think that was the problem with how Linux implemented pselect() circa 10 years ago. I think pselect fixes the race condition you get when you set a timer then call select(). Linux's implementation was a function that set a timer and then called select(). ... slam head on keyboard. So I think that fear is valid that they'll implement the API without the guarantees. Myself I'm annoyed with the reckless push always to remove guarantees in return for 'performance'.

leoc 3y ago
I'm starting to come round to the belief that the only robust remedy is to release the chaos monkey: have the kernel delete all the state of the filesystem driver at randomly-chosen intervals, a few hours apart on average, and force the driver to recover itself.
- bcrl 3y ago
  That is pretty close to what those of us building robust storage systems in the real world have to do. At a previous employer, we had test suites that exercised all kinds of corner cases by triggering failover and system reboots in the middle of heavy persistent messaging workloads. Filesystems have other horrors that you learn about during testing. I had one test case where it would take ext4 80 seconds to write out an 8MB file after a fresh mount of an 8TB filesystem. Free space was fragmented in just the right way that we hit the single threaded reading of block groups and bitmaps in the kernel and it took forever to get that data off the disk array.
  - Gibbon1 3y ago
    I've done the same with embedded drivers, randomly flip bits in the drivers state and see what happens. Does it recover, throw fault, hang, or explode like a bomb. I got that from a hardware designer talking about robust state machines. Any random illegal state should sequence back to a known good state.
    - zbentley 3y ago

tedunangst 3y ago

The problem is every filesystem would implement a different version of fsync_that_works or rename_for_safety and then applications will have to call all of them, and then people will start building a compat framework that tries to assemble the optimal sequence of calls for each filesystem except it won't work in every case and so a new fsync_but_no_cheating function will be proposed.

appplication 3y ago
Does this suggest that the idea of an overarching fsync/rename is perhaps being proposed at the wrong level of abstraction? If it can’t be reliably implemented at every subsequent level, what’s the point in having it at this level?
- tedunangst 3y ago
  Different filesystems have different on disk structures, with varying degrees of complexity and cost in getting those things on disk. Everything can reliably get everything written eventually, but if you want to fully utilize your disk bandwidth, you only want to flush the data that can get there fast, which is going to vary by filesystem. And different applications have different durability requirements.
  - layer8 3y ago
    This is still no argument to not define syscalls for the semantics that applications typically care about. If it is slow on a particular file system, then so be it, it’s still the semantics the application requires.
    - tedunangst 3y ago
      So call fsync.
dale_glass 3y ago
The trick is not allowing such a thing. Don't have something vague like "fsync_that_works" that could be implemented arbitrarily. Have a function that is documented to implement a highly specific contract, like "writes all pending blocks of this file to disk, and ensures they'll be there if there's a crash after the function returns".

Animats 3y ago

I've proposed this informally before, along these lines: - Unit files. You create, you write, you close, and then others can read the new version. Until the original writer closes and gets a good close status, nobody else can read it. If the program aborts or the system crashes before closing cleanly, the file reverts. All readers see a fully written file. This is the default. It's what most programs need. Replacing a file by creating a new one on top of it is both permitted and an atomic operation. UCLA-LOCUS and some IBM systems that followed worked that way. - Temporary files. Disappear on system crash. - Log files. Append-only. All readers are guaranteed to see an end of file position that corresponds to the end of a previous write. Usually the most recent write, but buffering may make log file reading run a little behind. - Managed files. Read, write, share. Write operations return two completions, probably via some async mechanism. The first completion means "buffer contents taken". The second completion means "committed to storage that will survive a crash". Database systems would use this, but few other programs would bother. This tells the database what it really needs to know - when is the data safe? That tells the database when it can commit a transaction. The database can do other things, including more I/O, while waiting for commitment. "fsync" is really a clunky way of getting that second completion.

the8472 3y ago
I think most of those things are implementable in userspace with the right combination of O_TMPFILE, rename, fsync, fsync(dir), sync_file_range and io_uring. So all that's needed is encapsulating it in a library that provides those different abstractions. Well ok, we could also use an additional flag for linkat() that allows atomically replacing the target to close that tiny window where a temporary file might get left behind after a crash.
- vlovich123 3y ago
  Actually, a good chunk of that is available via memfd + linkat. - Unit files: memfd, fdatasync, linkat, fsync(dir) - Temporary files. memfd, preferably within /tmp to avoid any disk I/O unless you have a particularly large file to buffer - Log files. This is trickier. Probably requires running on a fs that supports reflinks and creating a new memfd that’s a true clone of the original file, appending, & then replacing it. The one piece that is incompatible with this is that other file descriptors are stale and you have to reopen to see new data but that’s at odds with APIs available today - either readers can see partial writes or you have to have a new file. There have been proposals to allow buffering writes but delaying publishing the metadata acking the data but that hasn’t gone anywhere unfortunately. - Managed files. I’m not sure that completions are the right contract here because technically the kernel could defer that completion indefinitely whereas databases typical need some kind of latency bounds of “OK - I really want this to commit to disk nowish”.

Denvercoder9 3y ago

I don't see how new APIs will help here. The problem is that the semantics that application developers want is expensive (performance-wise), so they use a similar API that doesn't guarantee those semantics but provides them in practice 99% of the time. You could create a new set of API's, but it's really hard to implement any API in a way that doesn't give extra, unspecified semantics in practice, so what prevents application developers from making the same choice?

Filligree 3y ago
They aren't that expensive. ZFS provides them by default, in that no operations are ever reordered in a user-visible fashion even without fsync. That's a stronger guarantee than we're asking for here.

throwaway09223 3y ago

But there is no rigamarole. The article is just super confused. Atomicity is orthogonal to durability. rename() is atomic and always has been atomic. It is atomic without guaranteeing durability. In fact, to guarantee durability after rename you must: 1) rename(a, b) 2) open(the parent dir) 3) fsync(the parent dir) This is to guarantee durability of the metadata written by rename() in the dirent. All of this is totally separate from guaranteeing the durability of the data in the file, which requires its own separate fsync() on the file itself. Atomicity and durability are never the same. This article is gibberish.

aidenn0 3y ago
The phrase "atomic across crashes" has a meaning that is distinct from durability. It is also different from atomic across a live system.
- throwaway09223 3y ago
  The phrase "atomic across crashes" is meaningless. Atomicity deals with the running system state at the moment a system call executes. It has nothing to do with crashes (which are not atomic in and of themselves), or what bits actually end up on disk.
  - aidenn0 3y ago
    Are you being willfully obtuse, or do you honestly believe that the word "atomicity" only applies to such a narrow context?
    - throwaway09223 3y ago
      The latter. What do you think it means?
      - rcxdude 3y ago
        Atomic simply means indivisible: you don't observe half of an atomic operation. Of course it strongly depends on where you are observing the operation from, and that does not necessarily just mean 'a running system', hence OP specifying that they wish for the operation to be atomic even when observed from a system before and after a crash.