4 ms·

Also, there seems to be an effort brewing in the kernel to push userspace away from depending on inode #s due to difficulty in guaranting uniqueness and stabili

Also, there seems to be an effort brewing in the kernel to push userspace away from depending on inode #s due to difficulty in guaranting uniqueness and stability across reboots. https://youtu.be/TNWK1zbTMOU https://youtu.be/TNWK1zbTMOU

AndrewDavis 1y ago
They definitely aren't unique even without reboots. Postfix uses the inode number as a queue id. At $dayjob we've seen reuse surprisingly quickly, even within a few hours. Which is a little annoying when we're log spelunking and we get two sets of results because of the repeating id! (there is now a long queue id option which adds a time component)
- amiga386 1y ago
  ...but it's unique while the file exists, right? The combination of st_dev and st_ino from stat() should be unique on a machine, while the device remains mounted and the file continues to exist. If the file is deleted, a different file might get the inode, and if a device is unmounted, another device might get the device id.
  - the_mitsuhiko 1y ago
    > The combination of st_dev and st_ino from stat() should be unique on a machine It should, but it seems no longer to be the case. I believe there was an attempt to get a sysctl flag in to force the kernel to return the same inode for all files to see what breaks.
  - AndrewDavis 1y ago
    Yes! It's reusable, but not duplicated.
  - londons_explore 1y ago
    > ...but it's unique while the file exists, right? I don't think all filesystems guarantee this. Especially network filesystems.
    - the_mitsuhiko 1y ago
      It's effectively impossible to guarantee this when you have a file system that unifies and re-exports. Network file systems being an obvious one, but overlayfs is in a similar position. Even if inodes still work nowadays they will eventually run into issues a few years down the line.
      - account42 1y ago
        Then unifying file systems is not something that POSIX support and a POSIX system shouldn't do it unless it can somehow map inodes with POSIX semantics. E.g. for a network mount spanning multiple remote filesystems you could also have multiple st_dev locally.
    - amiga386 1y ago
      That's a problem for programs that do recursive fs descent (e.g. find, tar) because they use st_dev and st_ino alone for remembering what directories they've been in. They can't just use the absolute path, because symbolic links allow for loops. find: * https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/find/sharefile.c#n51 https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/fi... * https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/find/pred.c#n900 https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/fi... tar: * https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/create.c#n1413 https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/crea... * https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/names.c#n905 https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/name... * https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/incremen.c#n524 https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/incr... In particular, I'm intrigued by the comment in the last link: /* With NFS, the same file can have two different devices if an NFS directory is mounted in multiple locations, which is relatively common when automounting. To avoid spurious incremental redumping of directories, consider all NFS devices as equal, relying on the i-node to establish differences. */ So GNU tar expects an inode to be unique across _all_ NFS mounts...
      - the_mitsuhiko 1y ago
- koverstreet 1y ago
  The combination of st_ino and the inode generation is guaranteed to be unique (excepting across subvolumes, because snapshots screw everything up). Filesystems maintain a generation number that's incremented when an inode number is being used, for NFS. Unfortunately, it doesn't even seem to be exposed in statx (!). There's change_cookie, but that's different. If anyone wants to submit a patch for this, I'll be happy to review it.
  - formerly_proven 1y ago
    We tried to handle hard-links correctly ~ten years ago in a backup tool and even back then it was "well obviously you have to use (dev, ino, gen), not just (dev, ino), that's moronic!" and we're like... "so where do we get the generation number?". There actually was an ext4 ioctl to do so, but at least back then it was only implemented by ext4. IME it's a pretty consistent pattern with unixy/linux filesystems that what you can do with the normal APIs is wrong/incorrect/full of bugs and race conditions, while what you're "obviously supposed to do" instead is some absurd fs-specific concoction or just straight up isn't exposed by the kernel.
    - koverstreet 1y ago
      Tell me about it. Getting proper APIs is just an enormous hassle, we're bad about bikeshedding things to death. Although it has gotten somewhat better.
quotemstr 1y ago
The problem isn't relying on inode numers; it's inode numbers being too short. Make them GUIDs and the problems of uniqueness disappear. As for stability: that's just a matter of filesystem durability in general.
- the_mitsuhiko 1y ago
  > The problem isn't relying on inode numers; it's inode numbers being too short. It's a bit of both. inodes are conflating two things in a way. They are used by the file system to identify a record but they are _also_ exposed in APIs that are really cross file system (and it comes to a head in case of network file systems or overlayfs). What's a more realistic path is to make inodes just an FS thing, let it do it's thing, and then create a set of APIs that is not relying on inodes as much. Linux for instance is trying to move towards file handles as being that API layer.
- bastawhiz 1y ago
  You could make it bigger, but then your inode table gets pretty big. If an inode number is 32 bits today, then UUIDs would take up four times the space. I'd also guess that the cost of hashing the UUIDs is significant enough that you'd see a user-visible performance hit. And really, it's not even super necessary. 64-bit inode numbers already exist in modern file systems. You don't need UUIDs to have unique IDs forever: you'll never run out of 64-bit integers. But the problem wasn't really ever that you'd run out, the problem is in the way they're handled.
  - quotemstr 1y ago
    > You could make it bigger, but then your inode table gets pretty big. You could do it like Java's Object.identityHashCode() and allocate durable IDs only on demand. > If an inode number is 32 bits today, then UUIDs would take up four times the space. We probably waste more space on filesystems that lack tail-packing. > I'd also guess that the cost of hashing the UUIDs is significant enough that you'd see a user-visible performance hit. We're hashing filenames for H-tree indexing anyway, aren't we? > you'll never run out of 64-bit integers Yeah, but with 128-bit ones you'll additionally never collide.

AndrewDavis 1y ago

They definitely aren't unique even without reboots. Postfix uses the inode number as a queue id. At $dayjob we've seen reuse surprisingly quickly, even within a few hours. Which is a little annoying when we're log spelunking and we get two sets of results because of the repeating id! (there is now a long queue id option which adds a time component)

amiga386 1y ago
...but it's unique while the file exists, right? The combination of st_dev and st_ino from stat() should be unique on a machine, while the device remains mounted and the file continues to exist. If the file is deleted, a different file might get the inode, and if a device is unmounted, another device might get the device id.
- the_mitsuhiko 1y ago
  > The combination of st_dev and st_ino from stat() should be unique on a machine It should, but it seems no longer to be the case. I believe there was an attempt to get a sysctl flag in to force the kernel to return the same inode for all files to see what breaks.
- AndrewDavis 1y ago
  Yes! It's reusable, but not duplicated.
- londons_explore 1y ago
  > ...but it's unique while the file exists, right? I don't think all filesystems guarantee this. Especially network filesystems.
  - the_mitsuhiko 1y ago
    It's effectively impossible to guarantee this when you have a file system that unifies and re-exports. Network file systems being an obvious one, but overlayfs is in a similar position. Even if inodes still work nowadays they will eventually run into issues a few years down the line.
    - account42 1y ago
      Then unifying file systems is not something that POSIX support and a POSIX system shouldn't do it unless it can somehow map inodes with POSIX semantics. E.g. for a network mount spanning multiple remote filesystems you could also have multiple st_dev locally.
  - amiga386 1y ago
    That's a problem for programs that do recursive fs descent (e.g. find, tar) because they use st_dev and st_ino alone for remembering what directories they've been in. They can't just use the absolute path, because symbolic links allow for loops. find: * https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/find/sharefile.c#n51 https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/fi... * https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/find/pred.c#n900 https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/fi... tar: * https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/create.c#n1413 https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/crea... * https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/names.c#n905 https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/name... * https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/incremen.c#n524 https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/incr... In particular, I'm intrigued by the comment in the last link: /* With NFS, the same file can have two different devices if an NFS directory is mounted in multiple locations, which is relatively common when automounting. To avoid spurious incremental redumping of directories, consider all NFS devices as equal, relying on the i-node to establish differences. */ So GNU tar expects an inode to be unique across _all_ NFS mounts...
    - the_mitsuhiko 1y ago
koverstreet 1y ago
The combination of st_ino and the inode generation is guaranteed to be unique (excepting across subvolumes, because snapshots screw everything up). Filesystems maintain a generation number that's incremented when an inode number is being used, for NFS. Unfortunately, it doesn't even seem to be exposed in statx (!). There's change_cookie, but that's different. If anyone wants to submit a patch for this, I'll be happy to review it.
- formerly_proven 1y ago
  We tried to handle hard-links correctly ~ten years ago in a backup tool and even back then it was "well obviously you have to use (dev, ino, gen), not just (dev, ino), that's moronic!" and we're like... "so where do we get the generation number?". There actually was an ext4 ioctl to do so, but at least back then it was only implemented by ext4. IME it's a pretty consistent pattern with unixy/linux filesystems that what you can do with the normal APIs is wrong/incorrect/full of bugs and race conditions, while what you're "obviously supposed to do" instead is some absurd fs-specific concoction or just straight up isn't exposed by the kernel.
  - koverstreet 1y ago
    Tell me about it. Getting proper APIs is just an enormous hassle, we're bad about bikeshedding things to death. Although it has gotten somewhat better.

quotemstr 1y ago

The problem isn't relying on inode numers; it's inode numbers being too short. Make them GUIDs and the problems of uniqueness disappear. As for stability: that's just a matter of filesystem durability in general.

the_mitsuhiko 1y ago
> The problem isn't relying on inode numers; it's inode numbers being too short. It's a bit of both. inodes are conflating two things in a way. They are used by the file system to identify a record but they are _also_ exposed in APIs that are really cross file system (and it comes to a head in case of network file systems or overlayfs). What's a more realistic path is to make inodes just an FS thing, let it do it's thing, and then create a set of APIs that is not relying on inodes as much. Linux for instance is trying to move towards file handles as being that API layer.
bastawhiz 1y ago
You could make it bigger, but then your inode table gets pretty big. If an inode number is 32 bits today, then UUIDs would take up four times the space. I'd also guess that the cost of hashing the UUIDs is significant enough that you'd see a user-visible performance hit. And really, it's not even super necessary. 64-bit inode numbers already exist in modern file systems. You don't need UUIDs to have unique IDs forever: you'll never run out of 64-bit integers. But the problem wasn't really ever that you'd run out, the problem is in the way they're handled.
- quotemstr 1y ago
  > You could make it bigger, but then your inode table gets pretty big. You could do it like Java's Object.identityHashCode() and allocate durable IDs only on demand. > If an inode number is 32 bits today, then UUIDs would take up four times the space. We probably waste more space on filesystems that lack tail-packing. > I'd also guess that the cost of hashing the UUIDs is significant enough that you'd see a user-visible performance hit. We're hashing filenames for H-tree indexing anyway, aren't we? > you'll never run out of 64-bit integers Yeah, but with 128-bit ones you'll additionally never collide.