8 ms·

Btrfs on Zoned Block Devices

84 points by fomine3 6y ago

Vecr 6y ago
Is this supported on top of DM-crypt's zoned support, or is that planed?
- tremon 6y ago
  Nitpick: the device mapper zoned-device support isn't built on top of the device mapper encrypted-device support; both dm-crypt and dm-zoned are independent device mapper targets. This is native support for btrfs-on-SMR, without the dm-zoned layer in between. DM-zoned is meant for filesystems who are zone-unaware, and it works by batching writes and redirecting blocks into appropriate areas. Having the filesystem allocator be aware of the underlying device's zones allows for more efficient/performant use of the zones.
KirillPanov 6y ago
How about fixing RAID5 first please? At least for non-metadata blocks ("-draid5 -mraid1" or "-draid6 -mraid1c3")? https://phoronix.com/scan.php?page=news_item&px=Btrfs-Warning-RAID5-RAID6 https://phoronix.com/scan.php?page=news_item&px=Btrfs-Warnin... It's kind of insane that two companies now sell a fixed btrfs-RAID5 as a proprietary commercial product (yes, in spite of the GPL). They do it by using the mdadm RAID5 code, which works, and add proprietary hooks to allow btrfs to use its checksums to figure out which spindle is corrupt in a parity mismatch situation. This (a) closes the write hole without a journal doubling the I/O load and (b) detects silent corruption, neither of which mdadm can do. Or just upstream this patch set: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg93181.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg... ...so we can do this ourselves with LVM (split each spindle into a small metadata device and a large device, stitch the large data devices together with mdadm-raid5/6, then add that and all the small devices to a btrfs filesystem with the small devices marked "metadata only" and -dsingle -mraid1c3). I really feel like the btrfs guys are stuck in "shiny new thing" mode here. Last time I checked host-managed SMR devices were only available in engineering sample quantities. Even if that's changed they surely are still very rare, a tiny minority of worldwide storage device sales. The fact that btrfs-RAID5 scrubs take something like O(num_spindles^2) seek latencies is crazy... I have an 8-spindle array that scrubs in 12 hours with "-draid0" but takes 10 days with "-draid5".
- scaladev 6y ago
  I am sure that Western Digital (whom the author of this patchset works for) will be very interested in your opinion. He implements whatever makes business sense for WD, and not what some random forum commenter wants. I am not at all surprised that a storage vendor would want upstream Linux support for its upcoming devices before they're actually being sold in stores.
- yokaze 6y ago
  I think, it's simply not high importance for the commercial backers of the project. It's not a hobby those people get payed for it. Raid5/6 seems to me more popular in home use.
  - kalleboo 6y ago
    Whats sad is that there is clearly a commercial value to the feature, but that commercial value is captured by proprietary implementations of it (e.g. Synology's btrfs-on-mdraid)
    - KirillPanov 6y ago
      Exactly this. The fact that Synology's makes money selling their proprietary btrfs-RAID5 is incontrovertible proof of the commercial relevance.
- pantalaimon 6y ago
  Looks like there is someone working on it https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg95863.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...
habitue 6y ago
Can someone knowledgeable explain why Btrfs isn't like better than ZFS at this point in terms of stability and feature set? Is it just designed poorly, or neglected, or what? It seems like it's been many years and it's still having a ton of trouble.
- Woung1938 6y ago
  Btrfs is actually in use by some big companies like Facebook but the initial issues seem to linger in people's memory and thus everyone and their cat avoids btrfs like fire. It reminds me of systemd for some reason. For the record I'm using btrfs on Arch (so recent kernel) for years with no issues (including LUKS encrypted root filesystem and RAID1 arrays for backups).
  - yjftsjthsd-h 6y ago
    (N=1) I have a single-disk laptop running opensuse (tumbleweed) on btrfs. It's the only machine I've ever owned to corrupt its root filesystem beyond repair, and it's done so twice IIRC (definitely 2, maybe 3), within the last few years. It's not just the initial issues.
    - scaladev 6y ago
      The other way to interpret this would be "it was the only filesystem to detect corruption on my malfunctioning hardware", because that's what usually happens in the last few years. Or have you been using ZFS on the same hardware?
      - NavinF 6y ago
        He said "corrupt its root filesystem beyond repair", not "detect checksum errors" Btw raid5/6 is still broken on btrfs which makes it a hard sell for any system with more than 2 disks. cf. raidz on ZFS
        scaladev 6y ago
scottlamb 6y ago
Are these host-managed SMR HDDs available for individual purchase, or are they enterprise-only? Last time I looked it seemed like the latter. I'd be curious to know if anyone has found a way to purchase them individually, and if they're significantly cheaper or higher-capacity than the closest equivalent PMR drive. I've played with device-managed SMR drives and decided I hate them. Too much I/O happening behind my back, so a low workload they can sustain before hitting a horrible performance cliff. Also, in my very limited sample, poor reliability even if you stay within the rated workload. (2/2 drives that were in continuous use started accumulating bad sectors within a couple years.) In theory, host-managed SMR drives with the right software might have much less write multiplication. The NVR software I'm working on more or less uses them as a big ring buffer (or several big ring buffers, one per stream) which seems ideal. But I suspect getting host-managed SMR to work optimally isn't worth the effort unless you have datacenters full of drives.
- PixelOfDeath 6y ago
  I wonder if a single bad bit could permanently destroy a whole ~16MiB sector? So SMR drives would need significantly more reserves to compensate for that.
  - petertodd 6y ago
    Shingled drives can still be read sector-by-sector. The shingling just changes how the data is written; it doesn't change how the data is read. So no, a single bad bit would only affect a single sector. And in fact, a single bad bit wouldn't do any harm, as all drives use error correcting codes that can handle a lot more than a single bit going bad. You'd actually lose a whole sector at time (512 bytes or 4096 bytes, depending on the drive).
  - magicalhippo 6y ago
    The difference between a regular and a SMR drive is that the SMR drives have a narrower read head compared to the write head. This means that when writing data it overwrites neighboring tracks, and hence writing is partially overlapped like shingles. Apart from that they're more or less the same as normal drives as far as I know, firmware aside. So a single bad bit shouldn't affect more than in a non-SMR drive.
mjevans 6y ago
F2FS - I'd currently rather use this, if I had to use a Zoned device (Assuming it's under 16TB in size). https://manpages.ubuntu.com/manpages/bionic/man8/mkfs.f2fs.8.html https://manpages.ubuntu.com/manpages/bionic/man8/mkfs.f2fs.8... -m -z #-of-sections-per-zone https://zonedstorage.io/linux/fs/ https://zonedstorage.io/linux/fs/ The f2fs section says "Zoned block device support was added to f2fs with kernel 4.10. Since f2fs uses a metadata block on-disk format with fixed block location, only zoned block devices which include conventional zones can be supported. Zoned devices composed entirely of sequential zones cannot be used with f2fs as a standalone device and require a multi-device setup to place metadata blocks on a randomly writable storage."
- scaladev 6y ago
  Has f2fs reliability changed much since this publication? https://www.usenix.org/conference/atc19/presentation/jaffer https://www.usenix.org/conference/atc19/presentation/jaffer I'd be afraid to use it for anything other than throwaway test data.
p1mrx 6y ago
Would it be possible to define a "zone" that GRUB_SAVEDEFAULT can overwrite? I reverted from btrfs to ext4 on my main desktop a few years ago, because grub couldn't remember the last selected menu item (error: sparse file not allowed), and I didn't feel like creating a dedicated /boot partition.
- _kdave 6y ago
  Booting from zoned device on btrfs is not supported, the 0th zone contains the super block and the boot loader data are before any superblock so updating them would require reset and complete rewrite of the live data in the zone. The 'sparse file file not found' is caused by grub, it would try to overwrite file blocks directly, but on btrfs it would cause checksum mismatch. This has been solved by storing the env block outside of the filesystem at 256K and synced back and forth once the system is booted. 256K is ok as btrfs does not use the first 1M on any device for bootloaders.

84 points by fomine3 6y ago

Vecr 6y ago

Is this supported on top of DM-crypt's zoned support, or is that planed?

tremon 6y ago
Nitpick: the device mapper zoned-device support isn't built on top of the device mapper encrypted-device support; both dm-crypt and dm-zoned are independent device mapper targets. This is native support for btrfs-on-SMR, without the dm-zoned layer in between. DM-zoned is meant for filesystems who are zone-unaware, and it works by batching writes and redirecting blocks into appropriate areas. Having the filesystem allocator be aware of the underlying device's zones allows for more efficient/performant use of the zones.

KirillPanov 6y ago

How about fixing RAID5 first please? At least for non-metadata blocks ("-draid5 -mraid1" or "-draid6 -mraid1c3")? https://phoronix.com/scan.php?page=news_item&px=Btrfs-Warning-RAID5-RAID6 https://phoronix.com/scan.php?page=news_item&px=Btrfs-Warnin... It's kind of insane that two companies now sell a fixed btrfs-RAID5 as a proprietary commercial product (yes, in spite of the GPL). They do it by using the mdadm RAID5 code, which works, and add proprietary hooks to allow btrfs to use its checksums to figure out which spindle is corrupt in a parity mismatch situation. This (a) closes the write hole without a journal doubling the I/O load and (b) detects silent corruption, neither of which mdadm can do. Or just upstream this patch set: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg93181.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg... ...so we can do this ourselves with LVM (split each spindle into a small metadata device and a large device, stitch the large data devices together with mdadm-raid5/6, then add that and all the small devices to a btrfs filesystem with the small devices marked "metadata only" and -dsingle -mraid1c3). I really feel like the btrfs guys are stuck in "shiny new thing" mode here. Last time I checked host-managed SMR devices were only available in engineering sample quantities. Even if that's changed they surely are still very rare, a tiny minority of worldwide storage device sales. The fact that btrfs-RAID5 scrubs take something like O(num_spindles^2) seek latencies is crazy... I have an 8-spindle array that scrubs in 12 hours with "-draid0" but takes 10 days with "-draid5".

scaladev 6y ago
I am sure that Western Digital (whom the author of this patchset works for) will be very interested in your opinion. He implements whatever makes business sense for WD, and not what some random forum commenter wants. I am not at all surprised that a storage vendor would want upstream Linux support for its upcoming devices before they're actually being sold in stores.
yokaze 6y ago
I think, it's simply not high importance for the commercial backers of the project. It's not a hobby those people get payed for it. Raid5/6 seems to me more popular in home use.
- kalleboo 6y ago
  Whats sad is that there is clearly a commercial value to the feature, but that commercial value is captured by proprietary implementations of it (e.g. Synology's btrfs-on-mdraid)
  - KirillPanov 6y ago
    Exactly this. The fact that Synology's makes money selling their proprietary btrfs-RAID5 is incontrovertible proof of the commercial relevance.
pantalaimon 6y ago
Looks like there is someone working on it https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg95863.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...

habitue 6y ago

Can someone knowledgeable explain why Btrfs isn't like better than ZFS at this point in terms of stability and feature set? Is it just designed poorly, or neglected, or what? It seems like it's been many years and it's still having a ton of trouble.

Woung1938 6y ago
Btrfs is actually in use by some big companies like Facebook but the initial issues seem to linger in people's memory and thus everyone and their cat avoids btrfs like fire. It reminds me of systemd for some reason. For the record I'm using btrfs on Arch (so recent kernel) for years with no issues (including LUKS encrypted root filesystem and RAID1 arrays for backups).
- yjftsjthsd-h 6y ago
  (N=1) I have a single-disk laptop running opensuse (tumbleweed) on btrfs. It's the only machine I've ever owned to corrupt its root filesystem beyond repair, and it's done so twice IIRC (definitely 2, maybe 3), within the last few years. It's not just the initial issues.
  - scaladev 6y ago
    The other way to interpret this would be "it was the only filesystem to detect corruption on my malfunctioning hardware", because that's what usually happens in the last few years. Or have you been using ZFS on the same hardware?
    - NavinF 6y ago
      He said "corrupt its root filesystem beyond repair", not "detect checksum errors" Btw raid5/6 is still broken on btrfs which makes it a hard sell for any system with more than 2 disks. cf. raidz on ZFS
      - scaladev 6y ago

scottlamb 6y ago

Are these host-managed SMR HDDs available for individual purchase, or are they enterprise-only? Last time I looked it seemed like the latter. I'd be curious to know if anyone has found a way to purchase them individually, and if they're significantly cheaper or higher-capacity than the closest equivalent PMR drive. I've played with device-managed SMR drives and decided I hate them. Too much I/O happening behind my back, so a low workload they can sustain before hitting a horrible performance cliff. Also, in my very limited sample, poor reliability even if you stay within the rated workload. (2/2 drives that were in continuous use started accumulating bad sectors within a couple years.) In theory, host-managed SMR drives with the right software might have much less write multiplication. The NVR software I'm working on more or less uses them as a big ring buffer (or several big ring buffers, one per stream) which seems ideal. But I suspect getting host-managed SMR to work optimally isn't worth the effort unless you have datacenters full of drives.

PixelOfDeath 6y ago
I wonder if a single bad bit could permanently destroy a whole ~16MiB sector? So SMR drives would need significantly more reserves to compensate for that.
- petertodd 6y ago
  Shingled drives can still be read sector-by-sector. The shingling just changes how the data is written; it doesn't change how the data is read. So no, a single bad bit would only affect a single sector. And in fact, a single bad bit wouldn't do any harm, as all drives use error correcting codes that can handle a lot more than a single bit going bad. You'd actually lose a whole sector at time (512 bytes or 4096 bytes, depending on the drive).
- magicalhippo 6y ago
  The difference between a regular and a SMR drive is that the SMR drives have a narrower read head compared to the write head. This means that when writing data it overwrites neighboring tracks, and hence writing is partially overlapped like shingles. Apart from that they're more or less the same as normal drives as far as I know, firmware aside. So a single bad bit shouldn't affect more than in a non-SMR drive.

mjevans 6y ago

F2FS - I'd currently rather use this, if I had to use a Zoned device (Assuming it's under 16TB in size). https://manpages.ubuntu.com/manpages/bionic/man8/mkfs.f2fs.8.html https://manpages.ubuntu.com/manpages/bionic/man8/mkfs.f2fs.8... -m -z #-of-sections-per-zone https://zonedstorage.io/linux/fs/ https://zonedstorage.io/linux/fs/ The f2fs section says "Zoned block device support was added to f2fs with kernel 4.10. Since f2fs uses a metadata block on-disk format with fixed block location, only zoned block devices which include conventional zones can be supported. Zoned devices composed entirely of sequential zones cannot be used with f2fs as a standalone device and require a multi-device setup to place metadata blocks on a randomly writable storage."

scaladev 6y ago
Has f2fs reliability changed much since this publication? https://www.usenix.org/conference/atc19/presentation/jaffer https://www.usenix.org/conference/atc19/presentation/jaffer I'd be afraid to use it for anything other than throwaway test data.

p1mrx 6y ago

Would it be possible to define a "zone" that GRUB_SAVEDEFAULT can overwrite? I reverted from btrfs to ext4 on my main desktop a few years ago, because grub couldn't remember the last selected menu item (error: sparse file not allowed), and I didn't feel like creating a dedicated /boot partition.

_kdave 6y ago
Booting from zoned device on btrfs is not supported, the 0th zone contains the super block and the boot loader data are before any superblock so updating them would require reset and complete rewrite of the live data in the zone. The 'sparse file file not found' is caused by grub, it would try to overwrite file blocks directly, but on btrfs it would cause checksum mismatch. This has been solved by storing the env block outside of the filesystem at 256K and synced back and forth once the system is booted. 256K is ok as btrfs does not use the first 1M on any device for bootloaders.