5 ms·

It's weird to me that there's so many promising reliefs that seem near at hand, but which have simply never been delivered. Zoned Storage has existed for a whil

It's weird to me that there's so many promising reliefs that seem near at hand, but which have simply never been delivered. Zoned Storage has existed for a while, for Shingled Magnetic Recording & tape, and there was much hubbub in 2018-2020 about NVMe getting Zoned Namespaces, which happened. https://zonedstorage.io/ https://zonedstorage.io/ And there's now a bunch of fs implementations. f2fs and btfs both have some support. But there's still no products actually available to buy. We could be getting so much better at NVMe, so systematically making things better. But we've kind of been stalled out for a while, after a bunch of ceremony figuring out what we wanted to do.

bcrl 3y ago
SMR and zoned storage offer compete and total utter crap performance in the real world. They have all the glass jaws of early flash firmware (garbage collection that took seconds to complete during which time no additional I/Os completed) while running on a medium that has many orders of magnitude higher latency than flash. Take 2008 era USB flash drives and that's about what you'll get out of an SMR HDD when using a write workload that isn't 100% large sequential writes. SMR might have become more relevant if it had provided a useful increase in density, but that never materialized as was originally promised. Retail prices for SMR drives were never much better than CMR. The largest generally available HDDs are available in CMR flavours because that's what's needed in real world servers. There's no 10x density improvement (or even 2x). Meanwhile flash is marching relentlessly down the cost reduction path afford by Moore's law + 3D layer stacking. Really fast 1TB NVMe SSDs are under $100 now, and it doesn't look like flash cost reduction is going to slow down any time soon.
- rektide 3y ago
  I wasn't really talking about SMR though, just some protocols they had that are now being re-used. I'm talking about Zoned Namespace SSDs (ZNS). Let's go back in time almost a decade. A bunch of smart people had figured out that the cumbersome Flash Translation Layer (FTL) on SSDs made it really hard to get expectable & consistent performance. They were building possible specs to try to directly access each flash block, to be able to fill them up as they saw fit & clear them out as they saw fit. They wanted direct access, with far less work juggling complex data-mappings done on the SSD itself. Two examples of Open Channel SSD works: https://openchannelssd.readthedocs.io/en/latest/ https://openchannelssd.readthedocs.io/en/latest/ http://lightnvm.io/ http://lightnvm.io/ Note the huge banner on the second: "Zoned Namespace (ZNS) SSDs have replaced the work on OCSSDs, and is now a standardized interface." Everyone realized that the protocols we had for Zoned Storage were basically adequate to get us what we wanted. We'd just make a lot of 128kB or whatever sized zones on the SSD, and let people manage them themselves. Currently this means opening blocks & then appending to them. There's been outstanding hope, in the future, that we might go further and allow more random write patterns, but still with the same contract of no over-writes (just clears). That work seemed like it was ready to go in 2020 & 2021. SSDs were supposedly sampling/becoming available. https://blog.westerndigital.com/zns-ssd-ultrastar-dc-zn540-sampling/ https://blog.westerndigital.com/zns-ssd-ultrastar-dc-zn540-s... https://semiconductor.samsung.com/newsroom/news/samsung-introduces-its-first-zns-ssd-with-maximized-user-capacity-and-enhanced-lifespan/ https://semiconductor.samsung.com/newsroom/news/samsung-intr... And that was reaffirmed again a year latter. https://www.tomshardware.com/news/samsung-and-western-digital-team-up-for-zns-ssds https://www.tomshardware.com/news/samsung-and-western-digita... But here we are in 2023 & there's still no Zoned Namespace SSDs (ZNS) one can purchase.
  - wtallis 3y ago
    You can purchase them, just not from retailers. These drive break compatibility in annoying ways (eg. cannot boot a system off one), so it's totally reasonable that drive vendors would not be offering them through channels where unsuspecting customers could so easily buy something they cannot use.
    - Dylan16807 3y ago
      They didn't need to completely remove compatibility modes. And I'm not asking to get one off a shelf but not only can I not buy one on amazon or newegg I can't even find a price when I look for a specific model! I think it's fair to say that to a first approximation I, as a person, can't get one. Edit: I found one site with a price on a WD drive, but they only have refurbished drives and the price is 25% off of $2200 for 1TB, so I'm going to ignore that site.
      - wtallis 3y ago
        > I can't even find a price when I look for a specific model! That's true of most enterprise/server components. At most you'll find a laughably inflated list price that hardly anyone actually pays.
      - rektide 3y ago
        I really wouldn't mind buying special purpose hardware. In fact, I think I prefer it. The whole story here is that modern SSD controllers tend to embed a bunch of very fast very expensive data-processing cores to quickly do a ton of fancy mapping. We don't hear it quite as clearly these days, but for example for a while higher end Samsung SSD controllers were announced as penta-core ARM Cortex-R[1] systems, and I think those Cortex-Rs are what most folks do. (It was groundbreaking news that WD released an open source "swerv" RISC-V core which they'd experimented with using instead, https://blog.westerndigital.com/risc-v-swerv-core-open-source/ https://blog.westerndigital.com/risc-v-swerv-core-open-sourc... .) Ideally, the fantasy is, we can build much cheaper & faster controllers that do far less. Zoned Namespace drives should ideally have enterprise grade, fast, dual-port access, but in many ways, I feel like they should ideally be far cheaper than DRAM-less drives. They should leave even more up to the host. They should be incredibly dumb & simple drives. They should be lower power, by far. They should eskew DRAM. Drop all the legacy baggage & expose what you are, un-intermediated, so we can be fast & use it well. Which you, the drive, with all purpose tasks, could never do. It's not worth it, to me, to make a drive that still keeps the ancillary old baggage of convention. If you want to make a good ZNS drive, make a good ZNS drive. [1] https://en.wikipedia.org/wiki/ARM_Cortex-R https://en.wikipedia.org/wiki/ARM_Cortex-R
- tremon 3y ago
  SMR and zoned storage offer compete and total utter crap performance in the real world Yes, that's because we're in the drive-managed transition phase of zoned storage (not clear when -if ever- we'll leave that phase). The first zoned disks needed to present themselves as a dumb unit to the OS, because none of the operating systems had any support for it. Now that at least some OS'es have native support for zoned storage, we might finally see host-managed zoned storage on the market, but as you say -- the technology has not delivered on the projected storage benefits, so the value-add of CMR disks is very much an open question. And there's also the consumer confusion angle: how to market a device that can physically connect to their computer, but might logically not work at all? So I'd suspect to see the first host-managed drives in enterprise datacenters (think Amazon Glacier) -- but FAFAIK they're nowhere to be found.
  - tremon 3y ago
    the value-add of CMR disks Argh. SMR, that is.
b33j0r 3y ago
The thing that still bothers me is that the read-write cycles and lifetime are sort of unpredictable to me personally. For an analogy, I only buy LED lightbulbs, love that tech. When they first became widely-available, my house had a few incandescent bulbs on the porch that I probably just left on for 5 years. The supposedly superior led bulbs in that adoption period often died surprisingly fast, compared to the claims. To bring it back to storage. I use as much NVMe as I can, but I’m still a bit uncertain on how much a risk I’m taking with my filesystem choice and access patterns. My answer: buy more SSDs when they go on sale! I don’t have intuition about lifetime even now. I know, HDDs have equivalent classes of worse problems. I just learned to expect them to reliably fail, like a few days after a build ;)
- macjohnmcc 3y ago
  The recent issues with Samsung's SSD's don't help with those feelings I'm sure. I have a WD-Black 1TB SSD in my system that I built in 2018 and I used it heavily all day long and I have not suffered any issues from the SSD. I also used a MacBook Pro from early 2013 and only recently sold it still going strong with it's SSD (proprietary). I read recently that some HDD were starting to have reliability problems after a few years.
- the_pwner224 3y ago
  $ sudo smartctl -A /dev/nvme0n1 Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 626,936 [320 GB] Data Units Written: 22,262,139 [11.3 TB] This is on a few months old PC which is why percentage used is still 0%. The number does slowly go up over time especially if you do heavy writing. As with LED lightbulbs, the consumer industry is focused on costcutting to deliver shitty products at low prices. Any high quality SSD is more than adequate for consumer use. Main thing to look at is how many layers the flash has. SLC is pretty much not available in consumer products. Triple layer is very adequate. All the cheap products have four layer, which you don't want unless you know you're not going to be writing often. Firmware bugs or bad controllers are still a potential issue. I have a 1.5 year old drive which is at something like 65% percentage used, it's a big drive with a high TBW rating and only a moderate amount of data written to it, but the excessive wear is known issue with that model. It'll get replaced under warranty but that's only kicking the can down the road since the replacement will have the same issue.
kijiki 3y ago
Google, Facebook, Microsoft, Amazon and so forth buy most of the spinning rust, and have for years. Each of them have their own extremely proprietary storage layer that knows how to manage Shingled Magnetic Recording. So when SMR got released into the open market, it was exclusively drive managed, and mainly on the low-end, as a cost reduction strategy. With all the performance downsides you'd expect. It is a real shame that host managed SMR isn't available outside the mega-scale corps. It would be nice to have access to an intelligent storage stack mixing flash and SMR drives without having to go work for one of the big N companies.

bcrl 3y ago

SMR and zoned storage offer compete and total utter crap performance in the real world. They have all the glass jaws of early flash firmware (garbage collection that took seconds to complete during which time no additional I/Os completed) while running on a medium that has many orders of magnitude higher latency than flash. Take 2008 era USB flash drives and that's about what you'll get out of an SMR HDD when using a write workload that isn't 100% large sequential writes. SMR might have become more relevant if it had provided a useful increase in density, but that never materialized as was originally promised. Retail prices for SMR drives were never much better than CMR. The largest generally available HDDs are available in CMR flavours because that's what's needed in real world servers. There's no 10x density improvement (or even 2x). Meanwhile flash is marching relentlessly down the cost reduction path afford by Moore's law + 3D layer stacking. Really fast 1TB NVMe SSDs are under $100 now, and it doesn't look like flash cost reduction is going to slow down any time soon.

rektide 3y ago
I wasn't really talking about SMR though, just some protocols they had that are now being re-used. I'm talking about Zoned Namespace SSDs (ZNS). Let's go back in time almost a decade. A bunch of smart people had figured out that the cumbersome Flash Translation Layer (FTL) on SSDs made it really hard to get expectable & consistent performance. They were building possible specs to try to directly access each flash block, to be able to fill them up as they saw fit & clear them out as they saw fit. They wanted direct access, with far less work juggling complex data-mappings done on the SSD itself. Two examples of Open Channel SSD works: https://openchannelssd.readthedocs.io/en/latest/ https://openchannelssd.readthedocs.io/en/latest/ http://lightnvm.io/ http://lightnvm.io/ Note the huge banner on the second: "Zoned Namespace (ZNS) SSDs have replaced the work on OCSSDs, and is now a standardized interface." Everyone realized that the protocols we had for Zoned Storage were basically adequate to get us what we wanted. We'd just make a lot of 128kB or whatever sized zones on the SSD, and let people manage them themselves. Currently this means opening blocks & then appending to them. There's been outstanding hope, in the future, that we might go further and allow more random write patterns, but still with the same contract of no over-writes (just clears). That work seemed like it was ready to go in 2020 & 2021. SSDs were supposedly sampling/becoming available. https://blog.westerndigital.com/zns-ssd-ultrastar-dc-zn540-sampling/ https://blog.westerndigital.com/zns-ssd-ultrastar-dc-zn540-s... https://semiconductor.samsung.com/newsroom/news/samsung-introduces-its-first-zns-ssd-with-maximized-user-capacity-and-enhanced-lifespan/ https://semiconductor.samsung.com/newsroom/news/samsung-intr... And that was reaffirmed again a year latter. https://www.tomshardware.com/news/samsung-and-western-digital-team-up-for-zns-ssds https://www.tomshardware.com/news/samsung-and-western-digita... But here we are in 2023 & there's still no Zoned Namespace SSDs (ZNS) one can purchase.
- wtallis 3y ago
  You can purchase them, just not from retailers. These drive break compatibility in annoying ways (eg. cannot boot a system off one), so it's totally reasonable that drive vendors would not be offering them through channels where unsuspecting customers could so easily buy something they cannot use.
  - Dylan16807 3y ago
    They didn't need to completely remove compatibility modes. And I'm not asking to get one off a shelf but not only can I not buy one on amazon or newegg I can't even find a price when I look for a specific model! I think it's fair to say that to a first approximation I, as a person, can't get one. Edit: I found one site with a price on a WD drive, but they only have refurbished drives and the price is 25% off of $2200 for 1TB, so I'm going to ignore that site.
    - wtallis 3y ago
      > I can't even find a price when I look for a specific model! That's true of most enterprise/server components. At most you'll find a laughably inflated list price that hardly anyone actually pays.
    - rektide 3y ago
      I really wouldn't mind buying special purpose hardware. In fact, I think I prefer it. The whole story here is that modern SSD controllers tend to embed a bunch of very fast very expensive data-processing cores to quickly do a ton of fancy mapping. We don't hear it quite as clearly these days, but for example for a while higher end Samsung SSD controllers were announced as penta-core ARM Cortex-R[1] systems, and I think those Cortex-Rs are what most folks do. (It was groundbreaking news that WD released an open source "swerv" RISC-V core which they'd experimented with using instead, https://blog.westerndigital.com/risc-v-swerv-core-open-source/ https://blog.westerndigital.com/risc-v-swerv-core-open-sourc... .) Ideally, the fantasy is, we can build much cheaper & faster controllers that do far less. Zoned Namespace drives should ideally have enterprise grade, fast, dual-port access, but in many ways, I feel like they should ideally be far cheaper than DRAM-less drives. They should leave even more up to the host. They should be incredibly dumb & simple drives. They should be lower power, by far. They should eskew DRAM. Drop all the legacy baggage & expose what you are, un-intermediated, so we can be fast & use it well. Which you, the drive, with all purpose tasks, could never do. It's not worth it, to me, to make a drive that still keeps the ancillary old baggage of convention. If you want to make a good ZNS drive, make a good ZNS drive. [1] https://en.wikipedia.org/wiki/ARM_Cortex-R https://en.wikipedia.org/wiki/ARM_Cortex-R
tremon 3y ago
SMR and zoned storage offer compete and total utter crap performance in the real world Yes, that's because we're in the drive-managed transition phase of zoned storage (not clear when -if ever- we'll leave that phase). The first zoned disks needed to present themselves as a dumb unit to the OS, because none of the operating systems had any support for it. Now that at least some OS'es have native support for zoned storage, we might finally see host-managed zoned storage on the market, but as you say -- the technology has not delivered on the projected storage benefits, so the value-add of CMR disks is very much an open question. And there's also the consumer confusion angle: how to market a device that can physically connect to their computer, but might logically not work at all? So I'd suspect to see the first host-managed drives in enterprise datacenters (think Amazon Glacier) -- but FAFAIK they're nowhere to be found.
- tremon 3y ago
  the value-add of CMR disks Argh. SMR, that is.

b33j0r 3y ago

The thing that still bothers me is that the read-write cycles and lifetime are sort of unpredictable to me personally. For an analogy, I only buy LED lightbulbs, love that tech. When they first became widely-available, my house had a few incandescent bulbs on the porch that I probably just left on for 5 years. The supposedly superior led bulbs in that adoption period often died surprisingly fast, compared to the claims. To bring it back to storage. I use as much NVMe as I can, but I’m still a bit uncertain on how much a risk I’m taking with my filesystem choice and access patterns. My answer: buy more SSDs when they go on sale! I don’t have intuition about lifetime even now. I know, HDDs have equivalent classes of worse problems. I just learned to expect them to reliably fail, like a few days after a build ;)

macjohnmcc 3y ago
The recent issues with Samsung's SSD's don't help with those feelings I'm sure. I have a WD-Black 1TB SSD in my system that I built in 2018 and I used it heavily all day long and I have not suffered any issues from the SSD. I also used a MacBook Pro from early 2013 and only recently sold it still going strong with it's SSD (proprietary). I read recently that some HDD were starting to have reliability problems after a few years.
the_pwner224 3y ago
$ sudo smartctl -A /dev/nvme0n1 Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 626,936 [320 GB] Data Units Written: 22,262,139 [11.3 TB] This is on a few months old PC which is why percentage used is still 0%. The number does slowly go up over time especially if you do heavy writing. As with LED lightbulbs, the consumer industry is focused on costcutting to deliver shitty products at low prices. Any high quality SSD is more than adequate for consumer use. Main thing to look at is how many layers the flash has. SLC is pretty much not available in consumer products. Triple layer is very adequate. All the cheap products have four layer, which you don't want unless you know you're not going to be writing often. Firmware bugs or bad controllers are still a potential issue. I have a 1.5 year old drive which is at something like 65% percentage used, it's a big drive with a high TBW rating and only a moderate amount of data written to it, but the excessive wear is known issue with that model. It'll get replaced under warranty but that's only kicking the can down the road since the replacement will have the same issue.

kijiki 3y ago

Google, Facebook, Microsoft, Amazon and so forth buy most of the spinning rust, and have for years. Each of them have their own extremely proprietary storage layer that knows how to manage Shingled Magnetic Recording. So when SMR got released into the open market, it was exclusively drive managed, and mainly on the low-end, as a cost reduction strategy. With all the performance downsides you'd expect. It is a real shame that host managed SMR isn't available outside the mega-scale corps. It would be nice to have access to an intelligent storage stack mixing flash and SMR drives without having to go work for one of the big N companies.