4 ms·

Absolutely zero mention of retention for a storage device is disturbing. The endurance figures seem to suggest anywhere between 6.6k and 11k cycles, which is b

Absolutely zero mention of retention for a storage device is disturbing. The endurance figures seem to suggest anywhere between 6.6k and 11k cycles, which is both a wide range and unusually high for TLC flash - this is the normally expected range for decent MLC and 5 years of retention, so I suspect they're massaging the retention downwards to get those numbers. Related: https://news.ycombinator.com/item?id=43702193 https://news.ycombinator.com/item?id=43702193

jauntywundrkind 1y ago
To ground these endurance figures a little more concretely, consumer drives will advertise 600TBW:TB, or 600 cycles! Starting at 10x that is a very solid starting place! On the other hand, an enterprise drive like Kioxia CM7 will offer either 1 or 3 drive writes per day (for regular and write-intensive drive models, respectively), across the 5 year warranty. That's ~1800 cycles or just shy of 5500 cycles.
- userbinator 1y ago
  Enterprise drives have always been rated for lower retention and hence higher cycles than consumer drives, since those are inversely correlated.
  - daymanstep 1y ago
    Do you have a source for that claim? Everything I've read suggests that data retention is mostly about the write temperature vs storage temperature, and that enterprise drives have about equivalent retention to consumer drives. By retention I'm assuming you're referring to the amount of time it takes for data loss to occur on SSDs in cold unpowered storage.
    - userbinator 1y ago
      From the horse's mouth: https://americas.kioxia.com/content/dam/kioxia/en-us/business/memory/mlc-nand/asset/KIOXIA_NAND_Flash_Data_Retention_Technical_Brief.pdf https://americas.kioxia.com/content/dam/kioxia/en-us/busines... (See graph on page 2.) https://www.macronix.com/Lists/ApplicationNote/Attachments/1920/AN0339V1-Endurance%20and%20Retention%20of%20NAND%20Flash.pdf https://www.macronix.com/Lists/ApplicationNote/Attachments/1... (See graph on page 3.) Then look up the retention specs for enterprise drives and compare to consumer ones, and the conclusion is obvious.
      - daymanstep 1y ago
        Correct me if I'm mistaken but it looks to me like your graphs are talking about the number of P/E cycles accumulated, rather than the number of P/E cycles that a drive is rated for? What this seems to suggest is that as a drive gets more "worn out", its data retention gets worse. But I don't see how that can be taken to imply that enterprise drives have worse data retention than consumer drives. Nothing that I've seen suggests this.
        zozbot234 1y ago
        > What this seems to suggest is that as a drive gets more "worn out", its data retention gets worse. This is what informal tests (i.e. via scrubbing/resilvering the drive after leaving it powered off for a long time) have found. Retention/data remanence is remarkably good for a drive that has been written over just once, and quite bad (i.e. you start seeing bit errors) for one that's almost worn out. This is actually very good news for the EEPROM-like use case where rewrites are quite rare. (Note that "almost worn out" in this case can mean going far beyond the formal total-data-written rating of the drive. We're talking the range where the hardware itself is about to croak.)
        userbinator 1y ago
        But I don't see how that can be taken to imply that enterprise drives have worse data retention than consumer drives Look up the specs. Commonly quoted as 3 months for enterprise ratings but the whole picture is in these tables: https://www.legitreviews.com/wp-content/uploads/2015/05/ssd-temps-endurance.jpg https://www.legitreviews.com/wp-content/uploads/2015/05/ssd-... Page 22 of this "says the quiet part out loud": https://www.snia.org/sites/default/files/AnilVasudeva_Are_SSDs_Ready_Enterprise_Storage_Systemsv4.pdf https://www.snia.org/sites/default/files/AnilVasudeva_Are_SS... ("Lower Data Retention allows for higher endurance")
        daymanstep 1y ago
        I'm not sure I fully understand what that page is saying. It is my understanding that JEDEC standard tests the data retention of the "worst case" scenario where a drive is fully worn out, i.e the drive has reached its maximum rated P/E cycles. If I'm understanding correctly, that page you linked is saying that enterprise drives have firmware that essentially allows more P/E cycles, which then means that at the end of those cycles, the drive will be more "worn out" and thus will have a worse data retention. But in a real world usage scenario where we subject a consumer SSD and an enterprise SSD to the same number of P/E cycles, would they have different data retention? I thought the JEDEC data was only for end-of-life scenarios.
- justincormack 1y ago
  Generally they just chop off some space to allow for more to be used when it wears out, which is why enterprise is 3.84TB not 4TB. If you partition the drive smaller and never use the rest you can achieve similar endurance.
drtgh 1y ago
Very disturbing, the article talks about the number of bits to be read per cell like if it were only a matter of speed. In addition to your comment (related), when 3D-NAND cells are read, interference by the traces (charge-trap disturbs) requires the neighbour cells to be refreshed with writes, if the controller wants to conserve data integrity. This did not happen with 2D-NAND in the past. Reading data from one cell in 3D-NAND involves writing cells; reading data in 3D-NAND consumes disk endurance. (Not to mention, temperature/number of layers/endurance)
- zozbot234 1y ago
  The read disturb effect happens with 2D-NAND, it's just rare.
- pkaye 1y ago
  I used to work in SSD controller firmware. This kind of issue existed more than 10 years ago. To achieve higher capacity with each generation of NAND you are trading off everything else (endurance, retention, read/program times, etc) little by little. 20 years ago it was so rare to see any kind of errors with SLC NAND that error detection and handling was fairly simple.
  - userbinator 1y ago
    I have some small USB drives from that era with SLC rated for 10 years of retention after 100k cycles, meaning a 64MB drive has a total endurance of 6.4TB. True binary capacity too, with no wear leveling either, as only the spares on each page were needed for a simple ECC. Too bad planned obsolescence got in the way, or we would've ended up today with bigger SLC drives that are fast and simple and just as reliable.
    - Dylan16807 1y ago
      There's no planned obsolescence for my use case. I've never done enough writes to wear out an SSD. While I might buy an SLC flash drive if it was properly available, for internal drives TLC has barely any downsides. I very rarely get limited to 500MB/s, I need to turn it on sometimes, and I get 2.8x the space. Sounds great.
devttyeu 1y ago
I don't think most people grasp how abstractly high even 1 DWPD is compared to enterprise HDDs. On the enterprise side you'll often read that a hard drive is rated for maybe 550TB/year, translating to 0.05~0.1 DRWPD [1] (yes, combined read AND write) and you have to be fine with that. (..yeah admittedly the workloads for each are quite different, you can realistically achieve >1 DWPD on an nvme with e.g. a large LSM database). What makes NVMe endurance ratings even better (though not for warranty purposes) is when your workload has sequential writes you can expect much higher effective endurance as most DWPD metrics are calculated for random 4k write, which is just about the worst case for flash with multi-megabyte erase blocks. It's my understanding that it's also in large part why there is some push for Zoned (hm-smr like) NVMe, where you can declare much higher DWPD. * [1] https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/data-center-drives/ultrastar-dc-hc600-series/data-sheet-ultrastar-dc-hc670.pdf https://documents.westerndigital.com/content/dam/doc-library...
- zozbot234 1y ago
  I assume that flash translation layers use LSM-like patterns underneath to cope with small random writes. The best case for flash with a good translation layer is data that can be erased/discarded in bulk, since this minimizes write amplification. This is close enough to the "large sequential writes" case but not necessarily equivalent.

jauntywundrkind 1y ago

To ground these endurance figures a little more concretely, consumer drives will advertise 600TBW:TB, or 600 cycles! Starting at 10x that is a very solid starting place! On the other hand, an enterprise drive like Kioxia CM7 will offer either 1 or 3 drive writes per day (for regular and write-intensive drive models, respectively), across the 5 year warranty. That's ~1800 cycles or just shy of 5500 cycles.

userbinator 1y ago
Enterprise drives have always been rated for lower retention and hence higher cycles than consumer drives, since those are inversely correlated.
- daymanstep 1y ago
  Do you have a source for that claim? Everything I've read suggests that data retention is mostly about the write temperature vs storage temperature, and that enterprise drives have about equivalent retention to consumer drives. By retention I'm assuming you're referring to the amount of time it takes for data loss to occur on SSDs in cold unpowered storage.
  - userbinator 1y ago
    From the horse's mouth: https://americas.kioxia.com/content/dam/kioxia/en-us/business/memory/mlc-nand/asset/KIOXIA_NAND_Flash_Data_Retention_Technical_Brief.pdf https://americas.kioxia.com/content/dam/kioxia/en-us/busines... (See graph on page 2.) https://www.macronix.com/Lists/ApplicationNote/Attachments/1920/AN0339V1-Endurance%20and%20Retention%20of%20NAND%20Flash.pdf https://www.macronix.com/Lists/ApplicationNote/Attachments/1... (See graph on page 3.) Then look up the retention specs for enterprise drives and compare to consumer ones, and the conclusion is obvious.
    - daymanstep 1y ago
      Correct me if I'm mistaken but it looks to me like your graphs are talking about the number of P/E cycles accumulated, rather than the number of P/E cycles that a drive is rated for? What this seems to suggest is that as a drive gets more "worn out", its data retention gets worse. But I don't see how that can be taken to imply that enterprise drives have worse data retention than consumer drives. Nothing that I've seen suggests this.
      - zozbot234 1y ago
        > What this seems to suggest is that as a drive gets more "worn out", its data retention gets worse. This is what informal tests (i.e. via scrubbing/resilvering the drive after leaving it powered off for a long time) have found. Retention/data remanence is remarkably good for a drive that has been written over just once, and quite bad (i.e. you start seeing bit errors) for one that's almost worn out. This is actually very good news for the EEPROM-like use case where rewrites are quite rare. (Note that "almost worn out" in this case can mean going far beyond the formal total-data-written rating of the drive. We're talking the range where the hardware itself is about to croak.)
      - userbinator 1y ago
        But I don't see how that can be taken to imply that enterprise drives have worse data retention than consumer drives Look up the specs. Commonly quoted as 3 months for enterprise ratings but the whole picture is in these tables: https://www.legitreviews.com/wp-content/uploads/2015/05/ssd-temps-endurance.jpg https://www.legitreviews.com/wp-content/uploads/2015/05/ssd-... Page 22 of this "says the quiet part out loud": https://www.snia.org/sites/default/files/AnilVasudeva_Are_SSDs_Ready_Enterprise_Storage_Systemsv4.pdf https://www.snia.org/sites/default/files/AnilVasudeva_Are_SS... ("Lower Data Retention allows for higher endurance")
        daymanstep 1y ago
        I'm not sure I fully understand what that page is saying. It is my understanding that JEDEC standard tests the data retention of the "worst case" scenario where a drive is fully worn out, i.e the drive has reached its maximum rated P/E cycles. If I'm understanding correctly, that page you linked is saying that enterprise drives have firmware that essentially allows more P/E cycles, which then means that at the end of those cycles, the drive will be more "worn out" and thus will have a worse data retention. But in a real world usage scenario where we subject a consumer SSD and an enterprise SSD to the same number of P/E cycles, would they have different data retention? I thought the JEDEC data was only for end-of-life scenarios.
justincormack 1y ago
Generally they just chop off some space to allow for more to be used when it wears out, which is why enterprise is 3.84TB not 4TB. If you partition the drive smaller and never use the rest you can achieve similar endurance.

drtgh 1y ago

Very disturbing, the article talks about the number of bits to be read per cell like if it were only a matter of speed. In addition to your comment (related), when 3D-NAND cells are read, interference by the traces (charge-trap disturbs) requires the neighbour cells to be refreshed with writes, if the controller wants to conserve data integrity. This did not happen with 2D-NAND in the past. Reading data from one cell in 3D-NAND involves writing cells; reading data in 3D-NAND consumes disk endurance. (Not to mention, temperature/number of layers/endurance)

zozbot234 1y ago
The read disturb effect happens with 2D-NAND, it's just rare.
pkaye 1y ago
I used to work in SSD controller firmware. This kind of issue existed more than 10 years ago. To achieve higher capacity with each generation of NAND you are trading off everything else (endurance, retention, read/program times, etc) little by little. 20 years ago it was so rare to see any kind of errors with SLC NAND that error detection and handling was fairly simple.
- userbinator 1y ago
  I have some small USB drives from that era with SLC rated for 10 years of retention after 100k cycles, meaning a 64MB drive has a total endurance of 6.4TB. True binary capacity too, with no wear leveling either, as only the spares on each page were needed for a simple ECC. Too bad planned obsolescence got in the way, or we would've ended up today with bigger SLC drives that are fast and simple and just as reliable.
  - Dylan16807 1y ago
    There's no planned obsolescence for my use case. I've never done enough writes to wear out an SSD. While I might buy an SLC flash drive if it was properly available, for internal drives TLC has barely any downsides. I very rarely get limited to 500MB/s, I need to turn it on sometimes, and I get 2.8x the space. Sounds great.

devttyeu 1y ago

I don't think most people grasp how abstractly high even 1 DWPD is compared to enterprise HDDs. On the enterprise side you'll often read that a hard drive is rated for maybe 550TB/year, translating to 0.05~0.1 DRWPD [1] (yes, combined read AND write) and you have to be fine with that. (..yeah admittedly the workloads for each are quite different, you can realistically achieve >1 DWPD on an nvme with e.g. a large LSM database). What makes NVMe endurance ratings even better (though not for warranty purposes) is when your workload has sequential writes you can expect much higher effective endurance as most DWPD metrics are calculated for random 4k write, which is just about the worst case for flash with multi-megabyte erase blocks. It's my understanding that it's also in large part why there is some push for Zoned (hm-smr like) NVMe, where you can declare much higher DWPD. * [1] https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/data-center-drives/ultrastar-dc-hc600-series/data-sheet-ultrastar-dc-hc670.pdf https://documents.westerndigital.com/content/dam/doc-library...

zozbot234 1y ago
I assume that flash translation layers use LSM-like patterns underneath to cope with small random writes. The best case for flash with a good translation layer is data that can be erased/discarded in bulk, since this minimizes write amplification. This is close enough to the "large sequential writes" case but not necessarily equivalent.