6 ms·

As far as I know it is a fundamental design difference between btrfs and ZFS. Btrfs has extent back references. Given a location on disk btrfs has an efficient

As far as I know it is a fundamental design difference between btrfs and ZFS. Btrfs has extent back references. Given a location on disk btrfs has an efficient data structure that gets from the disk location to the items using that disk location. E.g. if a sector on disk gets corrupted it has a command that lets you list all the the files using that sector. With ZFS you'd have to iterate through all files, list block locations and check if one of them is affected. Of course such an additional index comes with a performance cost for managing that index (any kind of write or delete operation plus more metadata space used). On the top of my head this is used for rebalance, quota groups, device removal, send/receive (finding reflinks) and file system shrinking. For a similar trade off compare hard links on Windows vs. ext4 and co. on Linux. On Linux there is no command to list all hard links to a file while Windows (NTFS) has one (FindFirstFileName etc.). This is why creating hard links on Windows is much slower and there is a limit of 1024 hard links to a file. To find all hard links to a file on Linux one would need to iterate over all files and check if they have the same inode number, however.

craftkiller 7y ago
Couldn't such an index simply be generated at the start of a grow/shrink/reshape operation and then thrown away after?
djsumdog 7y ago
I thought most *nix filesystems were like how you describe ZFS (and ext2/3/4) where all the metadata is at the beginning of the disk. At least with the ext family, doesn't the metadata reference the first 10 blocks of a file (where the last two blocks are pointers to double linked lists and the last a reference to a triple linked list). ... It's been a long time since I took that filesystems class, but I recall it's something like that?
- Dylan16807 7y ago
  That's about right. But with ext2/3/4 there are no snapshots and when you have a reference to a block you know that's the only reference, so it's not a problem there. https://en.wikipedia.org/wiki/Inode_pointer_structure https://en.wikipedia.org/wiki/Inode_pointer_structure has a summary https://docs.microsoft.com/en-us/archive/blogs/askcore/the-four-stages-of-ntfs-file-growth https://docs.microsoft.com/en-us/archive/blogs/askcore/the-f... talks about the NTFS equivalent
mehrdadn 7y ago
> This is why creating hard links on Windows is much slower Are you actually sure that's why? Usually the Windows bottleneck is in the I/O subsystem overhead rather than the file system itself. Note that FindFirstFileName requires opening the file first, which is the operation that incurs a lot of overhead irrespective of what you're doing after. I read that on Linux this is avoided through the use of a cache (hash table?), which Windows doesn't have. Once the file is opened, the I/O performance on Windows and Linux is comparable.
- tobias3 7y ago
  Ok, to be correct it is one of the factors. (one extra random write per hard link is a lot). Add another extra write to the Windows change journal and you probably have 3x the random writes per hard link compared to Linux (discouting active shadow copies which can additionally double that to 6x).
paulmd 7y ago
> With ZFS you'd have to iterate through all files, list block locations and check if one of them is affected. I mean... you just described a scrub. I'd gladly sit through a scrub if it meant that I could grow a raidz vdev. I'm suspecting there's some other hurdle.
- Dylan16807 7y ago
  That's what makes it impractical to use for anything other than very big and rare events. But on top of that you need a way to plan the changes and make them completely safe to do, in a way that doesn't collide with any other changes made to this presumably-live filesystem. And you have to store tons of data from the scrub somewhere to use during this process.
  - garmaine 7y ago
    > That's what makes it impractical to use for anything other than very big and rare events. Which is totally fine. Growing an array is a big and rare event. I'd be totally fine if I have to take the array offline or read-only for the duration. I just want the option to do so at all.
- rainbowzootsuit 7y ago
  Raidz expansion is currently a project under development in openzfs. https://m.youtube.com/watch?v=ZF8V7Tc9G28 https://m.youtube.com/watch?v=ZF8V7Tc9G28 https://youtu.be/Njt82e_3qVo https://youtu.be/Njt82e_3qVo
  - justinclift 7y ago
    Also, the work-in-progress (alpha quality) preview code: https://github.com/zfsonlinux/zfs/pull/8853 https://github.com/zfsonlinux/zfs/pull/8853
  - paulmd 7y ago
    I'm aware, but I really really don't like the solution of all future reads/writes requiring O(N) reads to resolve the data locations.

craftkiller 7y ago

Couldn't such an index simply be generated at the start of a grow/shrink/reshape operation and then thrown away after?

djsumdog 7y ago

I thought most *nix filesystems were like how you describe ZFS (and ext2/3/4) where all the metadata is at the beginning of the disk. At least with the ext family, doesn't the metadata reference the first 10 blocks of a file (where the last two blocks are pointers to double linked lists and the last a reference to a triple linked list). ... It's been a long time since I took that filesystems class, but I recall it's something like that?

Dylan16807 7y ago
That's about right. But with ext2/3/4 there are no snapshots and when you have a reference to a block you know that's the only reference, so it's not a problem there. https://en.wikipedia.org/wiki/Inode_pointer_structure https://en.wikipedia.org/wiki/Inode_pointer_structure has a summary https://docs.microsoft.com/en-us/archive/blogs/askcore/the-four-stages-of-ntfs-file-growth https://docs.microsoft.com/en-us/archive/blogs/askcore/the-f... talks about the NTFS equivalent

mehrdadn 7y ago

> This is why creating hard links on Windows is much slower Are you actually sure that's why? Usually the Windows bottleneck is in the I/O subsystem overhead rather than the file system itself. Note that FindFirstFileName requires opening the file first, which is the operation that incurs a lot of overhead irrespective of what you're doing after. I read that on Linux this is avoided through the use of a cache (hash table?), which Windows doesn't have. Once the file is opened, the I/O performance on Windows and Linux is comparable.

tobias3 7y ago
Ok, to be correct it is one of the factors. (one extra random write per hard link is a lot). Add another extra write to the Windows change journal and you probably have 3x the random writes per hard link compared to Linux (discouting active shadow copies which can additionally double that to 6x).

paulmd 7y ago

> With ZFS you'd have to iterate through all files, list block locations and check if one of them is affected. I mean... you just described a scrub. I'd gladly sit through a scrub if it meant that I could grow a raidz vdev. I'm suspecting there's some other hurdle.

Dylan16807 7y ago
That's what makes it impractical to use for anything other than very big and rare events. But on top of that you need a way to plan the changes and make them completely safe to do, in a way that doesn't collide with any other changes made to this presumably-live filesystem. And you have to store tons of data from the scrub somewhere to use during this process.
- garmaine 7y ago
  > That's what makes it impractical to use for anything other than very big and rare events. Which is totally fine. Growing an array is a big and rare event. I'd be totally fine if I have to take the array offline or read-only for the duration. I just want the option to do so at all.
rainbowzootsuit 7y ago
Raidz expansion is currently a project under development in openzfs. https://m.youtube.com/watch?v=ZF8V7Tc9G28 https://m.youtube.com/watch?v=ZF8V7Tc9G28 https://youtu.be/Njt82e_3qVo https://youtu.be/Njt82e_3qVo
- justinclift 7y ago
  Also, the work-in-progress (alpha quality) preview code: https://github.com/zfsonlinux/zfs/pull/8853 https://github.com/zfsonlinux/zfs/pull/8853
- paulmd 7y ago
  I'm aware, but I really really don't like the solution of all future reads/writes requiring O(N) reads to resolve the data locations.