2 ms·

> ZFS has a lot of genuine potential to work well with SMR drives, but that potential depends on glue code that nobody has written yet. ...and I'm not sure tha

> ZFS has a lot of genuine potential to work well with SMR drives, but that potential depends on glue code that nobody has written yet. ...and I'm not sure that will change. The thing is, the "drive-managed SMR" models I've managed to get my hands on don't have a zoned storage API. I'm not sure how a filesystem (or application that directly goes to the block device layer) is supposed to work with them intelligently. There are supposed to be host-managed and hybrid host+drive-managed SMR drives out there, but AFAICT they're only sold to enterprises/hyperscalers (maybe only the latter even). And those are using custom proprietary software (e.g. Google's D servers). ...so who would write/use this glue code? unless the manufacturers decide to contract filesystem developers as a lead-up to wider sale of these things, or something to that effect.

yencabulator 2y ago
> ...and I'm not sure that will change. The thing is, the "drive-managed SMR" models I've managed to get my hands on don't have a zoned storage API. I'm not sure how a filesystem (or application that directly goes to the block device layer) is supposed to work with them intelligently. Mostly by making large sequential writes. F2FS is a lot faster on a consumer drive-managed SMR drive than the others (at least until you need to fsck / upgrade kernel version / do any of the actions that utterly suck with f2fs).
- scottlamb 2y ago
  Maybe, but I'm not sure I trust that to be enough. The drive's proprietary SMR management firmware might still end up doing something really wasteful under normal usage patterns, in a way that's hard to diagnose and might not be necessary if the application were in control. Let's imagine an NVR that simply uses the drive as a ring buffer for media data. [1] And it does it via an application using the whole thing at the block device layer, no filesystem. And all the metadata is stored separately on an SSD. This should be the absolute best case for SMR, but even then I'm not confident it will work well. Eventually the drive fills and it starts rewriting. On each write, now it has to guarantee the bytes just beyond your write stay the same. My understanding is it has some minimum write size, or always some fixed-length overwrite, or something to that effect. Using large sequential writes surely helps, yes, and the device presumably has a generous amount of RAM-based write cache, but even so I'm suspicious it'll end up regularly reading and rewriting a bunch of data that will probably be overwritten anyway pretty soon, and even swapping to the CMR area of the drive constantly. If the application were in control, it'd probably ensure there's some "don't care" buffer between the write zone and the read zone to avoid this. But I don't think there's any way to tell many of these drive-managed SMR HDDs to do that. [1] You might want a ring per camera stream so you use less read bandwidth on playback of a single stream, and you might want the ability to redact bits of recording in the middle, but let's ignore that.
  - Dylan16807 2y ago
    > now it has to guarantee the bytes just beyond your write stay the same [...] "don't care" buffer between the write zone and the read zone to avoid this. If the firmware is competent the drive will have TRIM support. Also it could start overwriting a zone even when there's old live data in it, as long as it buffers a few megabytes to stay at least a track away from the old live data. And even if it does have to read back the whole zone, it could keep that in RAM to keep the performance impact from being too bad. (Ideally with some kind of power loss protection.)
    - scottlamb 2y ago
      > If the firmware is competent the drive will have TRIM support. Sure, if false then false is a true statement. It's not competent, though? iirc my ST8000DM004 did not support TRIM. I think it's typical of the genre of drives on the market. I have no faith these drives will not degenerate to the stupidest possible access pattern; their firmware is opaque and bad and their interface is limited.
      - Dylan16807 2y ago
        I know some drives do it though. I found a comment saying "Sadly it’s 2024 and WD seem to be the only manufacturer actually implementing TRIM." And I'd expect it to be more common on the higher end drives where someone might intentionally choose SMR to get more space, as opposed to the <=8TB market where SMR is a cost saving they don't want anyone to notice.
        scottlamb 2y ago
        > And I'd expect it to be more common on the higher end drives where someone might intentionally choose SMR to get more space, as opposed to the <=8TB market where SMR is a cost saving they don't want anyone to notice. Yeah, "don't want anyone to notice" is probably a big factor, and in fairness the drive I mentioned was of the surprise SMR scandal era. (8TB was decently large then though I think.) I guess fundamentally the hidden complexity of the firmware managing the storage also applies to SSD, but I think they've got more wiggle room there due to the inherent better performance of the media, and for whatever reason the average firmware quality appears better IMHO, with some notable exceptions: https://news.ycombinator.com/item?id=32031243 https://news.ycombinator.com/item?id=32031243

yencabulator 2y ago

> ...and I'm not sure that will change. The thing is, the "drive-managed SMR" models I've managed to get my hands on don't have a zoned storage API. I'm not sure how a filesystem (or application that directly goes to the block device layer) is supposed to work with them intelligently. Mostly by making large sequential writes. F2FS is a lot faster on a consumer drive-managed SMR drive than the others (at least until you need to fsck / upgrade kernel version / do any of the actions that utterly suck with f2fs).

scottlamb 2y ago
Maybe, but I'm not sure I trust that to be enough. The drive's proprietary SMR management firmware might still end up doing something really wasteful under normal usage patterns, in a way that's hard to diagnose and might not be necessary if the application were in control. Let's imagine an NVR that simply uses the drive as a ring buffer for media data. [1] And it does it via an application using the whole thing at the block device layer, no filesystem. And all the metadata is stored separately on an SSD. This should be the absolute best case for SMR, but even then I'm not confident it will work well. Eventually the drive fills and it starts rewriting. On each write, now it has to guarantee the bytes just beyond your write stay the same. My understanding is it has some minimum write size, or always some fixed-length overwrite, or something to that effect. Using large sequential writes surely helps, yes, and the device presumably has a generous amount of RAM-based write cache, but even so I'm suspicious it'll end up regularly reading and rewriting a bunch of data that will probably be overwritten anyway pretty soon, and even swapping to the CMR area of the drive constantly. If the application were in control, it'd probably ensure there's some "don't care" buffer between the write zone and the read zone to avoid this. But I don't think there's any way to tell many of these drive-managed SMR HDDs to do that. [1] You might want a ring per camera stream so you use less read bandwidth on playback of a single stream, and you might want the ability to redact bits of recording in the middle, but let's ignore that.
- Dylan16807 2y ago
  > now it has to guarantee the bytes just beyond your write stay the same [...] "don't care" buffer between the write zone and the read zone to avoid this. If the firmware is competent the drive will have TRIM support. Also it could start overwriting a zone even when there's old live data in it, as long as it buffers a few megabytes to stay at least a track away from the old live data. And even if it does have to read back the whole zone, it could keep that in RAM to keep the performance impact from being too bad. (Ideally with some kind of power loss protection.)
  - scottlamb 2y ago
    > If the firmware is competent the drive will have TRIM support. Sure, if false then false is a true statement. It's not competent, though? iirc my ST8000DM004 did not support TRIM. I think it's typical of the genre of drives on the market. I have no faith these drives will not degenerate to the stupidest possible access pattern; their firmware is opaque and bad and their interface is limited.
    - Dylan16807 2y ago
      I know some drives do it though. I found a comment saying "Sadly it’s 2024 and WD seem to be the only manufacturer actually implementing TRIM." And I'd expect it to be more common on the higher end drives where someone might intentionally choose SMR to get more space, as opposed to the <=8TB market where SMR is a cost saving they don't want anyone to notice.
      - scottlamb 2y ago
        > And I'd expect it to be more common on the higher end drives where someone might intentionally choose SMR to get more space, as opposed to the <=8TB market where SMR is a cost saving they don't want anyone to notice. Yeah, "don't want anyone to notice" is probably a big factor, and in fairness the drive I mentioned was of the surprise SMR scandal era. (8TB was decently large then though I think.) I guess fundamentally the hidden complexity of the firmware managing the storage also applies to SSD, but I think they've got more wiggle room there due to the inherent better performance of the media, and for whatever reason the average firmware quality appears better IMHO, with some notable exceptions: https://news.ycombinator.com/item?id=32031243 https://news.ycombinator.com/item?id=32031243