3 ms·

Are you being willfully obtuse, or do you honestly believe that the word "atomicity" only applies to such a narrow context?

throwaway09223 3y ago
The latter. What do you think it means?
- rcxdude 3y ago
  Atomic simply means indivisible: you don't observe half of an atomic operation. Of course it strongly depends on where you are observing the operation from, and that does not necessarily just mean 'a running system', hence OP specifying that they wish for the operation to be atomic even when observed from a system before and after a crash.
  - throwaway09223 3y ago
    Yes, but as I said above that doesn't make sense. The atomicity we're discussing here is with respect to time, and sequencing (not say size, like an atom). We say that rename() is atomic because you will either see file A or file B. There are no other possible states. With a crash, writes might not be committed. You might get an earlier filesystem state.
    - aidenn0 3y ago
      > With a crash, writes might not be committed. You might get an earlier filesystem state. And the whole point of TFA is that this is perfectly fine as long as the metadata writes aren't committed before the data writes. That is: Precondition: File at path A exits with data X 1. Create file at path B 2. Write data Y to file B. 3. Close file B 4. Rename B to A 5. Crash --- As long as the contents in the file at path A are either X or Y, then we have achieve atomicity in the sense used in TFA. This is what we mean by rename acting "atomically across crashes" Note that we only care about the file at path A; all of these are fine: - File at path B exists and has data Y - File at path B exists and has garbage data - File at path B doesn't exist - File at path B exists and is of size 0
      - rcxdude 3y ago
        Specifically (to hopefully clarify your point), the operation we wish to be atomic is 'replacing the mapping path A->data X with path A->data Y'. This is hoped to be done by placing data Y in path B (which does not need to be atomic )and then using the atomicity of the rename to move around the path. This then puts an ordering constraint that any observer (whether after a crash and looking at the written filesystem, or during the normal operation of the system) sees data Y going into path B before the rename B to A, and this is what POSIX does not guarantee.
        throwaway09223 3y ago
        Yeah, I get it. The issue is that filesystems don't work like you've assumed. I outlined why this idea cannot work here: https://news.ycombinator.com/item?id=38407472 https://news.ycombinator.com/item?id=38407472 Again, manipulating links has nothing to do with file data.
        rcxdude 3y ago
        It blatently can work, it's just not guaranteed by the standard (at least without an fsync). I don't think anyone is talking about situations where B is being actively written to while the rename happens.
        throwaway09223 3y ago
        This is like saying sync() works. You can sync to durable storage at any point, yes. But you cannot do it in a semantically useful fashion. "I don't think anyone is talking about situations where B is being actively written to while the rename happens." Well, you specifically are ignoring this. I'm not. Filesystems are free to generate extra syncs whenever they like. You could even have a filesystem sync every single write operation to durable storage - why not?
      - throwaway09223 3y ago
        This is fundamentally not how unix filesystems work. No file data is written in the above scenario with rename(). Rename changes links - not files. Let me rewrite this for you using more correct language: Given a link to a file at path A exists. The file contains data X. There may be other links at paths C, D and E. 1. Acquire a filesystem-global lock on link ops (link/unlink et al, but not read/write). 2. Create a link to X at path B 3. Remove the link at path A 4. Release the filesystem-global lock. Note: The data in X is not relevant. The data in X may be undergoing active modification while the above 4 steps are performed. The data in X may be memory mapped and may change many times during the "atomic" rename() syscall. The atomic properties of rename() are only vis a vis link/unlink semantics within a single filesystem. Other files can't be created or deleted while the rename() is in progress. File data however is under no such restriction -- and in fact it cannot be because the file may be memory-mapped and modified without any syscall interactions. There is no system call sequence point to gate reads and writes. What you are describing is fundamentally incompatible with how the system actually works. What you describe cannot be made to work, ever. It is architecturally invalid at a fundamental level.

throwaway09223 3y ago

The latter. What do you think it means?

rcxdude 3y ago
Atomic simply means indivisible: you don't observe half of an atomic operation. Of course it strongly depends on where you are observing the operation from, and that does not necessarily just mean 'a running system', hence OP specifying that they wish for the operation to be atomic even when observed from a system before and after a crash.
- throwaway09223 3y ago
  Yes, but as I said above that doesn't make sense. The atomicity we're discussing here is with respect to time, and sequencing (not say size, like an atom). We say that rename() is atomic because you will either see file A or file B. There are no other possible states. With a crash, writes might not be committed. You might get an earlier filesystem state.
  - aidenn0 3y ago
    > With a crash, writes might not be committed. You might get an earlier filesystem state. And the whole point of TFA is that this is perfectly fine as long as the metadata writes aren't committed before the data writes. That is: Precondition: File at path A exits with data X 1. Create file at path B 2. Write data Y to file B. 3. Close file B 4. Rename B to A 5. Crash --- As long as the contents in the file at path A are either X or Y, then we have achieve atomicity in the sense used in TFA. This is what we mean by rename acting "atomically across crashes" Note that we only care about the file at path A; all of these are fine: - File at path B exists and has data Y - File at path B exists and has garbage data - File at path B doesn't exist - File at path B exists and is of size 0
    - rcxdude 3y ago
      Specifically (to hopefully clarify your point), the operation we wish to be atomic is 'replacing the mapping path A->data X with path A->data Y'. This is hoped to be done by placing data Y in path B (which does not need to be atomic )and then using the atomicity of the rename to move around the path. This then puts an ordering constraint that any observer (whether after a crash and looking at the written filesystem, or during the normal operation of the system) sees data Y going into path B before the rename B to A, and this is what POSIX does not guarantee.
      - throwaway09223 3y ago
        Yeah, I get it. The issue is that filesystems don't work like you've assumed. I outlined why this idea cannot work here: https://news.ycombinator.com/item?id=38407472 https://news.ycombinator.com/item?id=38407472 Again, manipulating links has nothing to do with file data.
        rcxdude 3y ago
        It blatently can work, it's just not guaranteed by the standard (at least without an fsync). I don't think anyone is talking about situations where B is being actively written to while the rename happens.
        throwaway09223 3y ago
        This is like saying sync() works. You can sync to durable storage at any point, yes. But you cannot do it in a semantically useful fashion. "I don't think anyone is talking about situations where B is being actively written to while the rename happens." Well, you specifically are ignoring this. I'm not. Filesystems are free to generate extra syncs whenever they like. You could even have a filesystem sync every single write operation to durable storage - why not?
    - throwaway09223 3y ago
      This is fundamentally not how unix filesystems work. No file data is written in the above scenario with rename(). Rename changes links - not files. Let me rewrite this for you using more correct language: Given a link to a file at path A exists. The file contains data X. There may be other links at paths C, D and E. 1. Acquire a filesystem-global lock on link ops (link/unlink et al, but not read/write). 2. Create a link to X at path B 3. Remove the link at path A 4. Release the filesystem-global lock. Note: The data in X is not relevant. The data in X may be undergoing active modification while the above 4 steps are performed. The data in X may be memory mapped and may change many times during the "atomic" rename() syscall. The atomic properties of rename() are only vis a vis link/unlink semantics within a single filesystem. Other files can't be created or deleted while the rename() is in progress. File data however is under no such restriction -- and in fact it cannot be because the file may be memory-mapped and modified without any syscall interactions. There is no system call sequence point to gate reads and writes. What you are describing is fundamentally incompatible with how the system actually works. What you describe cannot be made to work, ever. It is architecturally invalid at a fundamental level.