17 ms·

Update on Samsung SSD Reliability

487 points by Akharin 4y ago

gjsman-1000 4y ago
There is always the possibility that their S.M.A.R.T. implementation is borked...
- TillE 4y ago
  The article does say they've seen "abnormally high failure rates in the field", so it's not just that.
- jpk 4y ago
  If that's all it was, then it's likely a firmware update would not only prevent the issue, but also reverse it if the storage is actually healthy. That doesn't seem to be the case here, though.
  - kmeisthax 4y ago
    Keep in mind that one of the things SSD firmware does is deliberately write-lock the drive if the media is too worn to erase. So buggy firmware overestimating media wear is also likely to cause a failed drive.
Dalewyn 4y ago
I wasn't aware the 980 Pro 2TB was also affected; I have four of those in a new machine I put together last year. Time to install some bloatware and see about updating their firmwares, I guess...
jeffbee 4y ago
Would like to know more. Were failures in the field from wear-out or sudden death? Are the health indicators losing 1% per week consistent with the datasheet TBW, or worse?
ssdpain 4y ago
I installed 2 x 980 Pro 2Tb in a laptop in Nov 2022. Running a daily Robocopy bat script to backup a folder in C: to D: would freeze a couple of times a week and lock the D: drive. After reboot, a drive check would find no errors and everything would work as normal. I've used the same script for years with no issues. Since the firmware update last week Robocopy has not frozen the drive at all this week.
- issafram 4y ago
  Could you provide that batch script please? Like in a GitHub Gist or something similar.
  - ssdpain 4y ago
    Sure... @echo off pause robocopy "C:\Users\o\Desktop\2023" "D:\2023" /e /mir /np /v /tee /r:0 /w:0 /log+:"C:\Users\o\Desktop\log_robocopy.txt" pause @echo on
    - skissane 4y ago
      You don’t need to turn echo back on at the end of your batch file. That line is pointless.
      - thrdbndndn 4y ago
        And don't need /e since /mir=/e /purge
        ssdpain 4y ago
        Thanks. This script has adapted over time and I need to lookup most of the switches these days to remind me what they do.
  - deleted 4y ago
    [deleted]
- 4y ago
pifm_guy 4y ago
I really want to see ssd manufacturers offer a decent warranty... This drive costs $100, and will last 10 years or until 100TB has been written to it, as long as you keep it within the specified temperature/humidity/power conditions. If it fails to do that, we will return $1000 to you.
- walterbell 4y ago
  In theory, a 3rd party insurance equivalent to AppleCare could be constructed for some technology products, but this is hampered by short product lifecycles, lack of BOM transparency (e.g components changed within a single product generation) and ability of firmware updates to change product behavior and invalidate previously collected data on reliability. Open-source SSD firmware would provide more transparency on performance and reliability.
  - CharlesW 4y ago
    > Open-source SSD firmware would provide more transparency on performance and reliability. This seems fantastic. Are you saying you could review the firmware source and know that the 980 Pro would lose ~1% of its endurance per week?
- TacticalCoder 4y ago
  Back when HDD would fail really a lot warranty was working. I'd happily fill an online form, Web 1.0 style, and then send my Seagate (I'm in Europe, was sending them to the Netherlands IIRC) disks and a few weeks later I'd receive a new drive. I probably still have a few screenshots of these forms somewhere.
- mrtksn 4y ago
  This sounds like an SLA agreement, its very unlikely you'll get that for 100 bucks. Even if this manufacturer somehow perfected their process and have zero defects, they are still acquiring a 10 years liability for 100 dollars of revenue.
  - rlpb 4y ago
Aardwolf 4y ago
I have this exact model, 980 Pro 2TB. It says to update firmware, but how can you do that from Linux? The instructions are all about some Windows program. Thanks! EDIT: I'm quite happy with the warning from this article, fixed a potential future problem!
- TacticalCoder 4y ago
  From TFA it's not just the 980 Pro 2 TB but also all the newer 990, so it's problematic.
  - Aardwolf 4y ago
    The article switches from the 980 issue to the 990 issue in a bit of an unclear way, but I think they're independent problems and the firmware update should fix the 980 one?
- pentamassiv 4y ago
  A few months ago I already updated my Samsung SSD by following this procedure: https://askubuntu.com/a/1386451 https://askubuntu.com/a/1386451. Theoretically they provide an image to boot from to do the update, but the image seems very outdated and did not recognize my keyboard so it was unusable.
  - Aardwolf 4y ago
    I now found and followed this: https://blog.quindorian.org/2021/05/firmware-update-samsung-ssd-in-linux.html/ https://blog.quindorian.org/2021/05/firmware-update-samsung-... And it seems to have worked. After extracting this updater tool and running it, smartctl kept showing the old firmware version (3B2QGXA7), but after reboot it now shows the new version (5B2QGXA7). I took the risk of running this while the OS (Archlinux) was running with the disk mounted (this is the OS install disk), and at first sight this didn't cause issues. But still do it at your own risk!!
    - FullyFunctional 4y ago
adrenvi 4y ago
Samsung 870 EVO drives were also known to fail early including my 2TB model. https://www.techpowerup.com/forums/threads/samsung-870-evo-beware-certain-batches-prone-to-failure.291504/ https://www.techpowerup.com/forums/threads/samsung-870-evo-b...
- acabal 4y ago
  Yes, this bit me just last month. Around October 2022 I purchased 3 Samsung 870 EVO 2TBs for use in a RAID array. By January 2023, all three of them failed within a week of each other! Fortunately they failed one by one, so I was barely was able to recover my RAID array by pulling out one drive at a time, powering the computer off, and waiting for the RMA replacement to arrive. But imagine my shock to see one drive fail... only to replace it with an RMA... and then days later, seeing the next drive fail... and the next!
  - bombcar 4y ago
    A perfect example why RAID ain’t backup
    - acabal 4y ago
      Indeed! Fortunately the RAID was also backed-up offsite. But, the entire process was shocking in several ways. At least Samsung was fairly speedy with the RMAs and it was basically no-questions-asked... because I imagine they're getting tons of these mailed back to them.
      - zargon 4y ago
        Did they require the failed drives to be returned? Did the secure erase function work on a failed drive? I guess this is a good reason for full disk encryption even on machines where I'm satisfied with their physical security.
        acabal 4y ago
smiley1437 4y ago
I've been trying to find a decent endurance NVME in the m.2 form factor for write-heavy applications and it appears that true 2-bit MLC has all but disappeared, replaced by 3-bit TLC and higher (with commensurate loss of endurance) The high endurance SSDs appear to be only available in u.2\u.3\hhhl and god-help-me EDSFF form factors Any suggestions? Micron's 7450 isn't readily available
- mnadkvlb 4y ago
  i recommend samsung pm9a3 versions. not as popular, but are enterprise products and also the endurance is like 3 times 980 pro i believe (please check it, not 100% sure). Been using in my threadrupper workstation with a lot of vms which are put to sleep every day with around .25tb written and read each time the vms are started. keep in mind these are 22110 form factor
  - jeffbee 4y ago
    These are not available with retail support so if you manage to acquire one (which may have been pulled from service with an unknown level of previous wear) you will get ZERO support from Samsung no matter what goes wrong.
    - mnadkvlb 4y ago
      well i bought from digitec here in switzerland and they take care of warranty. other than that i don't expect any support from any ssd vendor. I rely on the physical store i buy from where i live. that's the reason i only buy either physically or from amazon germnay (their support had been rock solid in the last 10 years i had been using them).
  - csdvrx 4y ago
    SR-IOV support? Any firmware issue? (https://forums.servethehome.com/index.php?threads/pm9a3-firmware-errormod-related-to-fw-version.36822/ https://forums.servethehome.com/index.php?threads/pm9a3-firm...)
- adgjlsfhk1 4y ago
flyinglizard 4y ago
Had issues with a 2TB 980 Pro. Things have stabilized with recent updates.
xoa 4y ago
Ars had a piece covering this as well, and I do wonder if there is something going on somewhere else in the Samsung stack, not just the NVMe 900 series line. Pure anecdote, but two years ago I did a NAS for a client using 24x 2TB Samsung 870 Evo drives (they'd gotten some incentive deal for it). While it was all one type vs mixed, there was the "luxury" of time because at that point getting the system they wanted together had a significant lead time. So I did ensure that the drives were purchased over the course of around 7 months, from multiple different reputable sellers (B&H, CDW, Provantage etc) in separate batches. System was solid, an Epyc 2 based SuperMicro server, running TrueNAS. And then last year with around 5500-7500 quite light hours of runtime (primarily reads, ~0.08 DWPD, well under official rating of 0.3 DWPD) drives started failing. These were definitely real failures, first indication came from regular automated ZFS scrubs and reporting increasing checksum errors and ATA errors. It was for so many drives and I'd always considered Samsung SSDs relatively reliable (even for consumer ones) that at first I thought it was a SATA controller failure, and our rep agreed and warranties back the server. They were great, gold plated support contracts pay off once in a while, and motherboard replacement and thorough testing later back in service. More drive problems. SMART short tests said everything was healthy, first longs did too. But then drives exceeded error limits and started getting faulted, and at last SMART long tests started failing. Digging in showed worrisome stats. So began swapping out and warrantying drives (cheers to the stress test to TrueNAS, in the end zero downtime or need to restore from backups). In the end, THIRTEEN (13) out of 24 failed. Brutal >50% dead drive rate. I talked to some others around and they'd seen <1 year rates also at 30-60%. Big :\. Rep also indicated they were hearing more about Samsung failures. Anyway, gave me a talking point going forward to really, really press management on "it's worth paying for drives from 3-4x brands and maybe splurging for higher rated vs consumer too", but also does made me wonder if there is something going on, or was (pandemic related?), at Samsung's storage division. It's definitely pure anecdote but still, I spread those drive purchases out reasonably hard, and they had radically different serial numbers. Same with other folks I know at other businesses using various Samsung drives, everyone has been going to real effort following decent practices to prevent buying drives all from a single lot. Even 10% failure rate for consumer drives I could have seen, but 54%? And not a bathtub curve all frontloaded in the first month or two but after 7-11 months? That feels high? Samsung did replace them all no questions asked, they paid for shipping too. I don't have any global insight into how this all looks and it could be just plain bad luck for all of us in the region, but still.
rekoil 4y ago
I've had 2x Samsung 980 Pro 2TB fail in as many years. Last time I buy Samsung.
- sushid 4y ago
  My Samsung 980s (1TB so not affected per article) are still going strong but Samsung has been dropping the ball across the board for me in recent years. I've had a bad experience with their Samsung Frame last year, their fridges are a nightmare to fix, and I'm now hearing bad things about their washing machine and their SSDs. I know these departments are not related but it's not a good look.
walrus01 4y ago
I wish more independent review organizations would conduct destructive "write lifespan until ultimate failure" real world tests on SSDs. With a mixture of real world large contiguous files and small random writes. Real ultimate write lifespan on 3-level-cell and QLC consumer grade SSDs varies wildly for things of the same capacity and similar price. Such as this series of tests from 7 years ago: https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/ https://techreport.com/review/27909/the-ssd-endurance-experi... It looks like the bar charts and other data in that URL are now broken, which is sad, because I recall reading it when it was first published and it shows some amazing differences between the drives that died first, and the ones that died last. another similar: https://www.guru3d.com/news-story/endurance-test-of-samsung-850-pro-comes-to-an-end-after-9100tb-of-writes.html https://www.guru3d.com/news-story/endurance-test-of-samsung-...
- KennyBlanken 4y ago
  What's even more important than durability is what the drive does when it runs out of write cycles. They should just become read-only, but it seems that in the vast majority, the controller just shuts off and bricks the drive.
- rom-antics 4y ago
  I'd like to see this too! The official line is that endurance should not matter for most people. For example the Samsung 990 Pro 4TB is rated for 2400TB TBW - which, if the drive has a service of 5 years, is 1.3TB of data written per day. The average user will need < 1% of that. Where that falls down though of course is cases like this. The point of a review is to show when real-world performance doesn't match the marketing. Tech reviewers seem to be blindly trusting the marketing on this one. They're really dropping the ball.
- Havoc 4y ago
  >destructive "write lifespan until ultimate failure" real world tests on SSDs >from 7 years ago It's from 7 years back for good reason. They stopped doing those tests when it became impractical as endurance increased. The drives are now good enough that you can't wear them out fast enough to make sense in a review setting ...unless fundamentally broken like these
mgdlbp 4y ago
Apparently a few months ago it became known on the Chinese internet that the 980 Pro, 970 Evo Plus with new controller, and OEM versions are prone to getting unreadable sectors, where SMART 'Media and Data Integrity Errors' increases on every read attempt. https://www.reddit.com/r/buildapc/comments/x82mwe/samsung_ssd_smart_0e_issue/ https://www.reddit.com/r/buildapc/comments/x82mwe/samsung_ss... https://www.reddit.com/r/DataHoarder/comments/x8arle/psa_samsung_980_pro_users_in_china_are_observing/ https://www.reddit.com/r/DataHoarder/comments/x8arle/psa_sam... How I came across this: Ran into this last week(!) on a 6-month old drive -- but I'm not in China....hmm. Not just one bad batch? Interestingly, it's non deterministic - the data is backed up but trying ddrescue, it occasionally succeeds at reading a few kilobytes from the 5 MB of several runs of 512-16384 bytes that can't be read or written. Curious to see what happens with a firmware update and secure erase.
- voidfunc 4y ago
  Hmm I'm going to need to check my Samsung ssd from oct 2021 that failed the first week of Jan 2023. I had started noticing some quirks in spring 2022 but it wasn't a super important drive so I ignored it.
  - Dnguyen 4y ago
    I have similar issue. It started failing mid last year. Then it got more and more frequent toward the end of the year. Last month I got tired of reinstalling OS for the 4th time and got a new system.
- jamhan 4y ago
  My anecdata: tl;dr: All 3 of my Samsung M.2 NVMe SSDs have failed in less than 3 years. 100% failure rate. My first SSD was a 1TB Samsung 970 EVO. It failed after 2 years and 8 months. It was replaced under warranty with a 1TB 970 EVO Plus. That replacement has now also failed after 1 year and 9 months. I bought a 2nd 1TB 970 EVO Plus in May 2019. It has now also failed (2 years and 7 months). Both are expected to be replaced under warranty. The 2 970 EVO Plus SSDs clearly had hardware errors (that were not accurately reflected in SMART data) that caused everything from system hangs, game crashes to file corruption on OTHER drives. I couldn't believe it at first but after 5 days of testing and trial and error, I had it confirmed. As soon as I removed those SSDs, my PC was completely stable again. In the meantime, I have bought a Kingston KC3000 1TB drive as I no longer trust Samsung M.2 NVMe SSDs. On the other hand, I have a Samsung EVO 850 SATA drive which has been rock-solid.
geocrasher 4y ago
Funny thing. This article prompted me to check the health of my two Samsung SSD's (a 250GB 850 EVO SATA III, and a 970 EVO Plus 1TB NVMe), which were fine. But Samsung's Magician also listed my Seagate ST2000DM008-2FR102 2TB spinny disk. It found a SMART error. I ran a performance test and looked at SMART again, and the "Hardware ECC Recovered" value went from 80 to 81, with a threshold of 64. My other software labels this as "good". Nevertheless, this drive is now being replaced by a 4TB WD Blue. Thanks, article. Saved me some future troubles!
- arprocter 4y ago
  You might want to double-check that SMART status in CrystalDiskInfo I don't trust Magician to report correctly on other vendor's storage
  - geocrasher 4y ago
    Yep, I was wrong on this. I should have used my google-fu instead of jumping to a conclusion. I guess I just really wanted an excuse to upgrade to a 4TB :)
- opencl 4y ago
  Hardware ECC Recovered represents the amount of time between error correction events, so a higher number is better.
- dheera 4y ago
  How do I check health of a Samsung SSD on Linux?
  - javaunsafe2019 4y ago
    LMGTFY sudo smartctl -t long -a /dev/sdX
    - jeffbee 4y ago
      That's not the way for a modern SSD. Try `sudo nvme smart-log /dev/nvmeN`
aporetics 4y ago
Just a note as a happy customer of Puget Systems that my experience working with them has been excellent, they really seem to be expert in their field, and have been for many years. Also, their submerged in mineral oil aquarium computer was really cool, back in the day.
- danielodievich 4y ago
  I confirm. 2 Puget System workstations in this house. Just opened both of them up to add more hard drives for games and games work storage, and the cabling is so lovely. In my new gaming box I have a Samsung 980 Pro 1TB which according to this note is unaffected, and I couldn't find it on my motherboard. I created a support ticket and the support immediately responded with very clear explanation (it's under its own heatsink under a humongous heatsink/fan for the CPU. Duh!
- craigching 4y ago
  I use my Puget for build performance for a large mono-repo at Adobe. I could set my watch to the consistency in memory and CPU usage during a full build of our products.
KennyBlanken 4y ago
Ask anyone with a circa-2014-ish Macbook Pro about Samsung SSD reliability. The samsung-made drives lasted about 5-6 years. Everything seemed fine, and then one day you'd get a spinning pizza of death, power down, power it back on...and your SSD was...completely gone. Doesn't even enumerate on the PCIe bus. It's just gone. Screw the SSD chipset manufacturers for not making sure that their controllers can at least a)still show up on the bus b)be read-only in some sort of recovery mode.
- SV_BubbleTime 4y ago
  Not sure why downvoted. A recovery mode seems absolutely possible and I don’t care if it’s 100kbit over SPI. It’s better than losing everything.
  - ChewFarceSkunk 4y ago
    [dead]
bullen 4y ago
The real problem is OS:es that write to disk for no good reason. Windows 10 writes 100KB/s constantly. That should be illegal.
- walterbell 4y ago
  Do we know the location of those writes, perhaps they can be redirected to a ramdisk?
  - paranoidrobot 4y ago
    Sysinternals tools will show up the writes and what is causing them. I've dug down and found random things doing dumb stuff in the past. Verbose logging turned on by default for some services, for example.
  - rasz 4y ago
    management/windows logs about >100 active logs. Performance/data collector set about >50 active Event Trace Sessions running.
- rogers18445 4y ago
  This doesn't actually matter in a practical sense. Assuming 24/7, it's 3TB a year. Which is ~1% drive endurance. Also, if you are worried about overwriting the same files over and over, it also doesn't matter. Block device addresses are not physical addresses, controller maps them to wear the drive evenly.
  - matja 4y ago
    Except "evenly" is not a standard, or something that anyone other than the manufacturer can verify, it's hidden in the firmware so we have no idea really.
  - wtallis 4y ago
    On the other hand, lots of tiny writes scattered all over will tend to produce much higher write amplification than large sequential writes. So you'll get more actual wear to the drive from the 3TB of constant background churn than if you copied in 3TB of movies.
justinclift 4y ago
As a data point, the Linux kernel has a long list of workarounds for "ata" related devices (SSDs, HDDs, etc): https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4091ff99df26c67c9021/drivers/ata/libata-core.c#L3855-L4106 https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4... Can be a bit eye opening to look down that and see equipment you're using listed. ;)
- duckmysick 4y ago
  What's horkage?
  - jwatt 4y ago
    Brokenness.
  - deleted 4y ago
    [deleted]
- justinclift 4y ago
  This seems to be the same thing for NVMe devices (eg Samsung, SK Hynix, Micron, Kingston, ADATA, Intel, (etc) NVMe drives): https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4091ff99df26c67c9021/drivers/nvme/host/pci.c#L3357-L3497 https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4... Not quite as easy to read and understand as the ata driver code though. With the occasional further device specific workarounds in other parts of the code. eg for specific Toshiba, LiteON, and Kioxia devices: https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4091ff99df26c67c9021/drivers/nvme/host/core.c#L2641-L2672 https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4... While this seems to be special handling for Samsung X5 SSD external drives, and also Samsung 970 Evo Plus drives: https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4091ff99df26c67c9021/drivers/nvme/host/core.c#L2674-L2685 https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4...
gregatragenet3 4y ago
This posting came two months late for me. My 980 2TB crossed the river styx over the holiday break. Failure mode exactly as described. Nice Christmas present for me. Took 3 weeks to get the warranty replacement from Samsung.
dataflow 4y ago
Tangent: what's the cheapest 4TB+ PCIe SSD without significant known issues but with hardware encryption? The SN850x seems to have its own issues, and beyond these everything else seems so expensive.
- nine_k 4y ago
  Is hardware encryption a widespread feature? What are the benefits of it for you, compared to OS-level full disk encryption?
- wnevets 4y ago
  > The SN850x seems to have its own issues Such as?
  - dataflow 4y ago
    https://community.wd.com/t/sn850x-the-driver-detected-a-controller-error-on-device-raidport0/281205 https://community.wd.com/t/sn850x-the-driver-detected-a-cont...
evil-olive 4y ago
every machine I have that can fit 2 SSDs (basically, everything except the very slim laptops) I have converted over to running a ZFS mirror as its root filesystem. NixOS makes this very easy to do because the grub.mirroredBoots option [0] removes the need for a separate "bootpool" with limited ZFS feature flags. and crucially, I always make sure they're 2 drives from different manufacturers, so that a bug of this nature should never be able to take down both drives in a pool simultaneously. I think of this as the "if you're going to go to the trouble of wearing a belt and suspenders, make sure to buy them from separate brands" principle. 0: https://search.nixos.org/options?channel=22.11&show=boot.loader.grub.mirroredBoots https://search.nixos.org/options?channel=22.11&show=boot.loa...
- deleted 4y ago
  [deleted]
Havoc 4y ago
> high failure rates in the field Jikes. A lot of people were hoping that the high wear is just a reporting artifact
craigching 4y ago
I have a puget system with the Samsung SSDs mentioned and it locks up on me every 2-4 weeks. This sounds like it would explain the problem. Puget sent out a message this week to upgrade the Samsung firmware, but I am at the latest. I’ll be contacting Puget support on Monday so I’m on their radar. I will say I love my Puget, its performance has been killer other than these lockups. And I’ve heard only good things about Puget support. I should have reported this months ago, but it’s just now that I’m doing some critical work on Windows that it’s affecting me.
userbinator 4y ago
For the same price, you can get twice the space for 1/4 the endurance, thrice the space for 1/8th the endurance, and now four times the space for 1/16th the endurance. Most people don't realise that is a horrible tradeoff, because NAND flash marketing and terminology like "TLC" or "QLC" is intentionally deceptive and manufacturers have been very secretive about the true endurance specifications, as well as trying to overprice SLC out of production. If more people knew the truth of what they were trying to do, we wouldn't be in this situation.
- newZWhoDis 4y ago
  Can you explain your point further? Are you talking about competitors?
- wmf 4y ago
  While NAND endurance has certainly gone down, FTLs got much better during the same time so that SSD endurance is still fine for most people. And if the stock endurance isn't enough, a little overprovisioning is probably better than dropping back to very expensive MLC.
  - userbinator 4y ago
    If the number of firmware bugs in SSDs we've seen over time is any indication, I don't think things are really getting better... SLC needs almost no FTL. 100K endurance. Very low raw error rate that can be handled with basic ECC.
- wahern 4y ago
  > as well as trying to overprice SLC out of production. Is it even possible to buy SLC drives any more? For the past 5+ years the only outlet I've been able to find that even advertise SLC is https://www.delkin.com/ https://www.delkin.com/, and you need to speak to sales to even get a price. I just assumed they and any other similar suppliers bought giant lots of chips at the tail end of SLC production and jack up the price on every new order as their supply dwindles. Or maybe they cobble together drives from the tiny SLC chips used for cache on modern SSDs?
blindriver 4y ago
Is there any way to update my SSD on Windows without downloading Samsung Magician? I have the Samsung Evo across multiple systems and my Linux ones are okay but my Windows machines aren't okay unfortunately, they have the bad firmware.
- deleted 4y ago
  [deleted]
taspeotis 4y ago
We have Lenovo laptops at work with M.2’s that are OEM-branded Samsungs. One bricked itself in to read only mode after a few months. The other has been losing 1% health each week or so. I caught it losing 2% in just two days recently. These drives are older than the 990 model mentioned in the article but I have my suspicions anyway they’re dud drives. Nothing lost except time - they can be swapped under warranty. But I used to buy Intel exclusively before swapping to Samsung when Intel started selling rebranded drives. I guess the search for a reliable vendor starts again…
- jeffbee 4y ago
  These anecdotes are pretty frustrating without the other key piece of information. For the given lifetime indicators, how many writes were served? Are they wearing out faster than their TBW claims, or are they being written more than you expected?
  - taspeotis 4y ago
    It’s at 86% with 12.21TB written. Total power on time 68 days. Drive temp sits around 45 degrees celsius. I’m not paying great attention to all the SMART counters day by day. It’s for a dev workload so like … compiling code and stuff? I have the exact same workload on my desktop PC and its Samsung drive health is 99% after … years.
    - jeffbee 4y ago
      OK, so close to 1% lifetime per TBW, or lifetime approximately 100TBW. Thanks! That's consistent with their endurance claims for the smallest SSDs (128GB PM991 for example, or 256GB 960 Evo) but it would be poor for a larger one.
      - taspeotis 4y ago
        Cool well these are 2TB drives so…
- chemmail 4y ago
emodendroket 4y ago
Wouldn't be surprised if this is another pandemic supply chain casualty.
- jandrese 4y ago
  Sounds more like crappy firmware to me. This is not the first SSD to suffer from a crippling frimware flaw that is impossible to fix. I lost almost an entire computer lab of Dells thanks to the goddamn Sandforce firmware. One that the company acknowledged but refused to lift a finger to fix. Luckily it is possible to fix these yourself despite the vendor hostility towards the repair. Look how easy it is: https://computerlounge.it/how-to-unbrick-sandforce-ssd/ https://computerlounge.it/how-to-unbrick-sandforce-ssd/
mmis1000 4y ago
There is a chinese youtuber doing SSD durability tests on a few SSDs of different vendor. And one of the tested ssd is Samsung 980 pro. What is the funny thing? The Samsung 980 died before the wear test even start. https://youtu.be/tXYQZHz7u3w?t=898 https://youtu.be/tXYQZHz7u3w?t=898
JonathanFly 4y ago
Magician Smart Status reports 0 'media errors' on my 980 2TB. Is that suspiciously low or does it mean things are actually ok? Is there a different tool I should check with?
sebazzz 4y ago
SSDs are complex beasts :-) I've had an OCZ SSD in the past that also became read-only, in the sense that all changes after shutting down the computer were gone. If you rebooted everything was fine, but as soon as the computer was shut down and the SSD also ran out of power, it converted to the state before turning on the computer. That was so bizarre. Once I had a hefty Windows upgrade installed and it was gone after the reboot - as if I had never installed it. It also took me a while to realize it because first of course you start to doubt your own memory, you don't realize that the sector- mapping on the SSD has suddenly become read-only OCZ eventually solved this via an RMA and eventually OCZ also went upside down. The Time Warp bug it was called I think.
- mvdwoord 4y ago
  sounds ideal for a public terminal ;)
  - csdvrx 4y ago
    Marketing should make that a feature: hardware-guaranteed non-modifiable root filesystem! Better than Nix!
    - totalZero 4y ago
      No updates! Ever!
      - ashirviskas 4y ago
        Also, no malware!
  - sebazzz 4y ago
    There used to be a PCI module (iirc) that would assist in that. I believe it was called a Bo(u)rne Again module. It was used in school computers etc.
    - AdamJacobMuller 4y ago
talhah 4y ago
If you're using Linux you can use the following instructions to update the firmware. https://wiki.archlinux.org/title/Solid_state_drive#Update_under_Linux https://wiki.archlinux.org/title/Solid_state_drive#Update_un...
fulafel 4y ago
Anyone know if the failure mode is sudden total bricking or something more recoverable?
ThomasGlanzmann 4y ago
In 2021 I bought at least ten 870 EVO 4 TB SATA and six 980 Pro 2 TB NVMe. All devices failed within 6 months on barely used systems. Find some smart data here: <https://thomas.glanzmann.de/samsung/ https://thomas.glanzmann.de/samsung/> The pattern is always the same: I have them configured in a raid 1. Once a month debian does a raid check. During the raid check Debian reads all data from both devices. I get uncorrectable read errors. I no longer use Samsung SSDs and replaced them them with SSDSC2KB076T8, Micron SSDs and KC3000 Kingston NVMes. No failures since then. In 2021 I told a friend of mine about the issue. He also had a 870 EVO, issued a dd if=/dev/sdX of=/dev/null bs=8M and guess what, he got uncorrectable read errors. Due to running them in RAID 1 I caught the issue early and I had no data loss or downtime because the Linux software raid compensated for the bad hardware. However I replaced them in a hurry because I no longer trust Samsung SSDs. As you can see from the smart log they're barely used. Less than 4 months in service and 10 TB written. I also got uncorrectable read errors when evacuating data from the devices.
- lazyweb 4y ago
  Similar experience for me. Four 2TB 980pro in RAID-Z2. Since introducing the drives in August 2021 I've had to replace them five times. I think none of the original SSDs are left. Only between 5 - 25TB written on average. Usually individual uncorrectable errors caught by ZFS, but one drive just straight up died. I keep a cold spare these days. On the bright side, if the cycle continues, I'll never run out of warranty from Samsung or the vendor.
  - ThePowerOfFuet 4y ago
    > On the bright side, if the cycle continues, I'll never run out of warranty from Samsung or the vendor. The warranty is from the date of sale of the original unit. Replacing one doesn't reset the warranty end date.
    - gowld 4y ago
      If this really becomes an issue due to repeated failure, lemon law activates for a refund.
sireat 4y ago
I can live with my batch of 980pros and 870 EVOs failing. Bad batches happen to every manufacturer. However this seems like a systematic problem at Samsung. What really stinks is that I've been recommending Samsung EVOs and PROs to relatives and friends for some years. If I want to remain honest with myself I have to contact every one of them and have to run the SMART tests. So now there are NO reputable SSD manufacturers left. Only reliable SSDs are MLCs from mid 201Xs especially Intel and Samsung.
ChewFarceSkunk 4y ago
[dead]

487 points by Akharin 4y ago

gjsman-1000 4y ago

There is always the possibility that their S.M.A.R.T. implementation is borked...

TillE 4y ago
The article does say they've seen "abnormally high failure rates in the field", so it's not just that.
jpk 4y ago
If that's all it was, then it's likely a firmware update would not only prevent the issue, but also reverse it if the storage is actually healthy. That doesn't seem to be the case here, though.
- kmeisthax 4y ago
  Keep in mind that one of the things SSD firmware does is deliberately write-lock the drive if the media is too worn to erase. So buggy firmware overestimating media wear is also likely to cause a failed drive.

Dalewyn 4y ago

I wasn't aware the 980 Pro 2TB was also affected; I have four of those in a new machine I put together last year. Time to install some bloatware and see about updating their firmwares, I guess...

jeffbee 4y ago

Would like to know more. Were failures in the field from wear-out or sudden death? Are the health indicators losing 1% per week consistent with the datasheet TBW, or worse?

ssdpain 4y ago

I installed 2 x 980 Pro 2Tb in a laptop in Nov 2022. Running a daily Robocopy bat script to backup a folder in C: to D: would freeze a couple of times a week and lock the D: drive. After reboot, a drive check would find no errors and everything would work as normal. I've used the same script for years with no issues. Since the firmware update last week Robocopy has not frozen the drive at all this week.

issafram 4y ago
Could you provide that batch script please? Like in a GitHub Gist or something similar.
- ssdpain 4y ago
  Sure... @echo off pause robocopy "C:\Users\o\Desktop\2023" "D:\2023" /e /mir /np /v /tee /r:0 /w:0 /log+:"C:\Users\o\Desktop\log_robocopy.txt" pause @echo on
  - skissane 4y ago
    You don’t need to turn echo back on at the end of your batch file. That line is pointless.
    - thrdbndndn 4y ago
      And don't need /e since /mir=/e /purge
      - ssdpain 4y ago
        Thanks. This script has adapted over time and I need to lookup most of the switches these days to remind me what they do.
- deleted 4y ago
  [deleted]
4y ago

pifm_guy 4y ago

I really want to see ssd manufacturers offer a decent warranty... This drive costs $100, and will last 10 years or until 100TB has been written to it, as long as you keep it within the specified temperature/humidity/power conditions. If it fails to do that, we will return $1000 to you.

walterbell 4y ago
In theory, a 3rd party insurance equivalent to AppleCare could be constructed for some technology products, but this is hampered by short product lifecycles, lack of BOM transparency (e.g components changed within a single product generation) and ability of firmware updates to change product behavior and invalidate previously collected data on reliability. Open-source SSD firmware would provide more transparency on performance and reliability.
- CharlesW 4y ago
  > Open-source SSD firmware would provide more transparency on performance and reliability. This seems fantastic. Are you saying you could review the firmware source and know that the 980 Pro would lose ~1% of its endurance per week?
TacticalCoder 4y ago
Back when HDD would fail really a lot warranty was working. I'd happily fill an online form, Web 1.0 style, and then send my Seagate (I'm in Europe, was sending them to the Netherlands IIRC) disks and a few weeks later I'd receive a new drive. I probably still have a few screenshots of these forms somewhere.
mrtksn 4y ago
This sounds like an SLA agreement, its very unlikely you'll get that for 100 bucks. Even if this manufacturer somehow perfected their process and have zero defects, they are still acquiring a 10 years liability for 100 dollars of revenue.
- rlpb 4y ago

Aardwolf 4y ago

I have this exact model, 980 Pro 2TB. It says to update firmware, but how can you do that from Linux? The instructions are all about some Windows program. Thanks! EDIT: I'm quite happy with the warning from this article, fixed a potential future problem!

TacticalCoder 4y ago
From TFA it's not just the 980 Pro 2 TB but also all the newer 990, so it's problematic.
- Aardwolf 4y ago
  The article switches from the 980 issue to the 990 issue in a bit of an unclear way, but I think they're independent problems and the firmware update should fix the 980 one?
pentamassiv 4y ago
A few months ago I already updated my Samsung SSD by following this procedure: https://askubuntu.com/a/1386451 https://askubuntu.com/a/1386451. Theoretically they provide an image to boot from to do the update, but the image seems very outdated and did not recognize my keyboard so it was unusable.
- Aardwolf 4y ago
  I now found and followed this: https://blog.quindorian.org/2021/05/firmware-update-samsung-ssd-in-linux.html/ https://blog.quindorian.org/2021/05/firmware-update-samsung-... And it seems to have worked. After extracting this updater tool and running it, smartctl kept showing the old firmware version (3B2QGXA7), but after reboot it now shows the new version (5B2QGXA7). I took the risk of running this while the OS (Archlinux) was running with the disk mounted (this is the OS install disk), and at first sight this didn't cause issues. But still do it at your own risk!!
  - FullyFunctional 4y ago

adrenvi 4y ago

Samsung 870 EVO drives were also known to fail early including my 2TB model. https://www.techpowerup.com/forums/threads/samsung-870-evo-beware-certain-batches-prone-to-failure.291504/ https://www.techpowerup.com/forums/threads/samsung-870-evo-b...

acabal 4y ago
Yes, this bit me just last month. Around October 2022 I purchased 3 Samsung 870 EVO 2TBs for use in a RAID array. By January 2023, all three of them failed within a week of each other! Fortunately they failed one by one, so I was barely was able to recover my RAID array by pulling out one drive at a time, powering the computer off, and waiting for the RMA replacement to arrive. But imagine my shock to see one drive fail... only to replace it with an RMA... and then days later, seeing the next drive fail... and the next!
- bombcar 4y ago
  A perfect example why RAID ain’t backup
  - acabal 4y ago
    Indeed! Fortunately the RAID was also backed-up offsite. But, the entire process was shocking in several ways. At least Samsung was fairly speedy with the RMAs and it was basically no-questions-asked... because I imagine they're getting tons of these mailed back to them.
    - zargon 4y ago
      Did they require the failed drives to be returned? Did the secure erase function work on a failed drive? I guess this is a good reason for full disk encryption even on machines where I'm satisfied with their physical security.
      - acabal 4y ago

smiley1437 4y ago

I've been trying to find a decent endurance NVME in the m.2 form factor for write-heavy applications and it appears that true 2-bit MLC has all but disappeared, replaced by 3-bit TLC and higher (with commensurate loss of endurance) The high endurance SSDs appear to be only available in u.2\u.3\hhhl and god-help-me EDSFF form factors Any suggestions? Micron's 7450 isn't readily available

mnadkvlb 4y ago
i recommend samsung pm9a3 versions. not as popular, but are enterprise products and also the endurance is like 3 times 980 pro i believe (please check it, not 100% sure). Been using in my threadrupper workstation with a lot of vms which are put to sleep every day with around .25tb written and read each time the vms are started. keep in mind these are 22110 form factor
- jeffbee 4y ago
  These are not available with retail support so if you manage to acquire one (which may have been pulled from service with an unknown level of previous wear) you will get ZERO support from Samsung no matter what goes wrong.
  - mnadkvlb 4y ago
    well i bought from digitec here in switzerland and they take care of warranty. other than that i don't expect any support from any ssd vendor. I rely on the physical store i buy from where i live. that's the reason i only buy either physically or from amazon germnay (their support had been rock solid in the last 10 years i had been using them).
- csdvrx 4y ago
  SR-IOV support? Any firmware issue? (https://forums.servethehome.com/index.php?threads/pm9a3-firmware-errormod-related-to-fw-version.36822/ https://forums.servethehome.com/index.php?threads/pm9a3-firm...)
adgjlsfhk1 4y ago

flyinglizard 4y ago

Had issues with a 2TB 980 Pro. Things have stabilized with recent updates.

xoa 4y ago

Ars had a piece covering this as well, and I do wonder if there is something going on somewhere else in the Samsung stack, not just the NVMe 900 series line. Pure anecdote, but two years ago I did a NAS for a client using 24x 2TB Samsung 870 Evo drives (they'd gotten some incentive deal for it). While it was all one type vs mixed, there was the "luxury" of time because at that point getting the system they wanted together had a significant lead time. So I did ensure that the drives were purchased over the course of around 7 months, from multiple different reputable sellers (B&H, CDW, Provantage etc) in separate batches. System was solid, an Epyc 2 based SuperMicro server, running TrueNAS. And then last year with around 5500-7500 quite light hours of runtime (primarily reads, ~0.08 DWPD, well under official rating of 0.3 DWPD) drives started failing. These were definitely real failures, first indication came from regular automated ZFS scrubs and reporting increasing checksum errors and ATA errors. It was for so many drives and I'd always considered Samsung SSDs relatively reliable (even for consumer ones) that at first I thought it was a SATA controller failure, and our rep agreed and warranties back the server. They were great, gold plated support contracts pay off once in a while, and motherboard replacement and thorough testing later back in service. More drive problems. SMART short tests said everything was healthy, first longs did too. But then drives exceeded error limits and started getting faulted, and at last SMART long tests started failing. Digging in showed worrisome stats. So began swapping out and warrantying drives (cheers to the stress test to TrueNAS, in the end zero downtime or need to restore from backups). In the end, THIRTEEN (13) out of 24 failed. Brutal >50% dead drive rate. I talked to some others around and they'd seen <1 year rates also at 30-60%. Big :\. Rep also indicated they were hearing more about Samsung failures. Anyway, gave me a talking point going forward to really, really press management on "it's worth paying for drives from 3-4x brands and maybe splurging for higher rated vs consumer too", but also does made me wonder if there is something going on, or was (pandemic related?), at Samsung's storage division. It's definitely pure anecdote but still, I spread those drive purchases out reasonably hard, and they had radically different serial numbers. Same with other folks I know at other businesses using various Samsung drives, everyone has been going to real effort following decent practices to prevent buying drives all from a single lot. Even 10% failure rate for consumer drives I could have seen, but 54%? And not a bathtub curve all frontloaded in the first month or two but after 7-11 months? That feels high? Samsung did replace them all no questions asked, they paid for shipping too. I don't have any global insight into how this all looks and it could be just plain bad luck for all of us in the region, but still.

rekoil 4y ago

I've had 2x Samsung 980 Pro 2TB fail in as many years. Last time I buy Samsung.

sushid 4y ago
My Samsung 980s (1TB so not affected per article) are still going strong but Samsung has been dropping the ball across the board for me in recent years. I've had a bad experience with their Samsung Frame last year, their fridges are a nightmare to fix, and I'm now hearing bad things about their washing machine and their SSDs. I know these departments are not related but it's not a good look.

walrus01 4y ago

I wish more independent review organizations would conduct destructive "write lifespan until ultimate failure" real world tests on SSDs. With a mixture of real world large contiguous files and small random writes. Real ultimate write lifespan on 3-level-cell and QLC consumer grade SSDs varies wildly for things of the same capacity and similar price. Such as this series of tests from 7 years ago: https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/ https://techreport.com/review/27909/the-ssd-endurance-experi... It looks like the bar charts and other data in that URL are now broken, which is sad, because I recall reading it when it was first published and it shows some amazing differences between the drives that died first, and the ones that died last. another similar: https://www.guru3d.com/news-story/endurance-test-of-samsung-850-pro-comes-to-an-end-after-9100tb-of-writes.html https://www.guru3d.com/news-story/endurance-test-of-samsung-...

KennyBlanken 4y ago
What's even more important than durability is what the drive does when it runs out of write cycles. They should just become read-only, but it seems that in the vast majority, the controller just shuts off and bricks the drive.
rom-antics 4y ago
I'd like to see this too! The official line is that endurance should not matter for most people. For example the Samsung 990 Pro 4TB is rated for 2400TB TBW - which, if the drive has a service of 5 years, is 1.3TB of data written per day. The average user will need < 1% of that. Where that falls down though of course is cases like this. The point of a review is to show when real-world performance doesn't match the marketing. Tech reviewers seem to be blindly trusting the marketing on this one. They're really dropping the ball.
Havoc 4y ago
>destructive "write lifespan until ultimate failure" real world tests on SSDs >from 7 years ago It's from 7 years back for good reason. They stopped doing those tests when it became impractical as endurance increased. The drives are now good enough that you can't wear them out fast enough to make sense in a review setting ...unless fundamentally broken like these

mgdlbp 4y ago

Apparently a few months ago it became known on the Chinese internet that the 980 Pro, 970 Evo Plus with new controller, and OEM versions are prone to getting unreadable sectors, where SMART 'Media and Data Integrity Errors' increases on every read attempt. https://www.reddit.com/r/buildapc/comments/x82mwe/samsung_ssd_smart_0e_issue/ https://www.reddit.com/r/buildapc/comments/x82mwe/samsung_ss... https://www.reddit.com/r/DataHoarder/comments/x8arle/psa_samsung_980_pro_users_in_china_are_observing/ https://www.reddit.com/r/DataHoarder/comments/x8arle/psa_sam... How I came across this: Ran into this last week(!) on a 6-month old drive -- but I'm not in China....hmm. Not just one bad batch? Interestingly, it's non deterministic - the data is backed up but trying ddrescue, it occasionally succeeds at reading a few kilobytes from the 5 MB of several runs of 512-16384 bytes that can't be read or written. Curious to see what happens with a firmware update and secure erase.

voidfunc 4y ago
Hmm I'm going to need to check my Samsung ssd from oct 2021 that failed the first week of Jan 2023. I had started noticing some quirks in spring 2022 but it wasn't a super important drive so I ignored it.
- Dnguyen 4y ago
  I have similar issue. It started failing mid last year. Then it got more and more frequent toward the end of the year. Last month I got tired of reinstalling OS for the 4th time and got a new system.
jamhan 4y ago
My anecdata: tl;dr: All 3 of my Samsung M.2 NVMe SSDs have failed in less than 3 years. 100% failure rate. My first SSD was a 1TB Samsung 970 EVO. It failed after 2 years and 8 months. It was replaced under warranty with a 1TB 970 EVO Plus. That replacement has now also failed after 1 year and 9 months. I bought a 2nd 1TB 970 EVO Plus in May 2019. It has now also failed (2 years and 7 months). Both are expected to be replaced under warranty. The 2 970 EVO Plus SSDs clearly had hardware errors (that were not accurately reflected in SMART data) that caused everything from system hangs, game crashes to file corruption on OTHER drives. I couldn't believe it at first but after 5 days of testing and trial and error, I had it confirmed. As soon as I removed those SSDs, my PC was completely stable again. In the meantime, I have bought a Kingston KC3000 1TB drive as I no longer trust Samsung M.2 NVMe SSDs. On the other hand, I have a Samsung EVO 850 SATA drive which has been rock-solid.

geocrasher 4y ago

Funny thing. This article prompted me to check the health of my two Samsung SSD's (a 250GB 850 EVO SATA III, and a 970 EVO Plus 1TB NVMe), which were fine. But Samsung's Magician also listed my Seagate ST2000DM008-2FR102 2TB spinny disk. It found a SMART error. I ran a performance test and looked at SMART again, and the "Hardware ECC Recovered" value went from 80 to 81, with a threshold of 64. My other software labels this as "good". Nevertheless, this drive is now being replaced by a 4TB WD Blue. Thanks, article. Saved me some future troubles!

arprocter 4y ago
You might want to double-check that SMART status in CrystalDiskInfo I don't trust Magician to report correctly on other vendor's storage
- geocrasher 4y ago
  Yep, I was wrong on this. I should have used my google-fu instead of jumping to a conclusion. I guess I just really wanted an excuse to upgrade to a 4TB :)
opencl 4y ago
Hardware ECC Recovered represents the amount of time between error correction events, so a higher number is better.
dheera 4y ago
How do I check health of a Samsung SSD on Linux?
- javaunsafe2019 4y ago
  LMGTFY sudo smartctl -t long -a /dev/sdX
  - jeffbee 4y ago
    That's not the way for a modern SSD. Try `sudo nvme smart-log /dev/nvmeN`

aporetics 4y ago

Just a note as a happy customer of Puget Systems that my experience working with them has been excellent, they really seem to be expert in their field, and have been for many years. Also, their submerged in mineral oil aquarium computer was really cool, back in the day.

danielodievich 4y ago
I confirm. 2 Puget System workstations in this house. Just opened both of them up to add more hard drives for games and games work storage, and the cabling is so lovely. In my new gaming box I have a Samsung 980 Pro 1TB which according to this note is unaffected, and I couldn't find it on my motherboard. I created a support ticket and the support immediately responded with very clear explanation (it's under its own heatsink under a humongous heatsink/fan for the CPU. Duh!
craigching 4y ago
I use my Puget for build performance for a large mono-repo at Adobe. I could set my watch to the consistency in memory and CPU usage during a full build of our products.

KennyBlanken 4y ago

Ask anyone with a circa-2014-ish Macbook Pro about Samsung SSD reliability. The samsung-made drives lasted about 5-6 years. Everything seemed fine, and then one day you'd get a spinning pizza of death, power down, power it back on...and your SSD was...completely gone. Doesn't even enumerate on the PCIe bus. It's just gone. Screw the SSD chipset manufacturers for not making sure that their controllers can at least a)still show up on the bus b)be read-only in some sort of recovery mode.

SV_BubbleTime 4y ago
Not sure why downvoted. A recovery mode seems absolutely possible and I don’t care if it’s 100kbit over SPI. It’s better than losing everything.
- ChewFarceSkunk 4y ago
  [dead]

bullen 4y ago

The real problem is OS:es that write to disk for no good reason. Windows 10 writes 100KB/s constantly. That should be illegal.

walterbell 4y ago
Do we know the location of those writes, perhaps they can be redirected to a ramdisk?
- paranoidrobot 4y ago
  Sysinternals tools will show up the writes and what is causing them. I've dug down and found random things doing dumb stuff in the past. Verbose logging turned on by default for some services, for example.
- rasz 4y ago
  management/windows logs about >100 active logs. Performance/data collector set about >50 active Event Trace Sessions running.
rogers18445 4y ago
This doesn't actually matter in a practical sense. Assuming 24/7, it's 3TB a year. Which is ~1% drive endurance. Also, if you are worried about overwriting the same files over and over, it also doesn't matter. Block device addresses are not physical addresses, controller maps them to wear the drive evenly.
- matja 4y ago
  Except "evenly" is not a standard, or something that anyone other than the manufacturer can verify, it's hidden in the firmware so we have no idea really.
- wtallis 4y ago
  On the other hand, lots of tiny writes scattered all over will tend to produce much higher write amplification than large sequential writes. So you'll get more actual wear to the drive from the 3TB of constant background churn than if you copied in 3TB of movies.

justinclift 4y ago

As a data point, the Linux kernel has a long list of workarounds for "ata" related devices (SSDs, HDDs, etc): https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4091ff99df26c67c9021/drivers/ata/libata-core.c#L3855-L4106 https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4... Can be a bit eye opening to look down that and see equipment you're using listed. ;)

duckmysick 4y ago
What's horkage?
- jwatt 4y ago
  Brokenness.
- deleted 4y ago
  [deleted]
justinclift 4y ago
This seems to be the same thing for NVMe devices (eg Samsung, SK Hynix, Micron, Kingston, ADATA, Intel, (etc) NVMe drives): https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4091ff99df26c67c9021/drivers/nvme/host/pci.c#L3357-L3497 https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4... Not quite as easy to read and understand as the ata driver code though. With the occasional further device specific workarounds in other parts of the code. eg for specific Toshiba, LiteON, and Kioxia devices: https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4091ff99df26c67c9021/drivers/nvme/host/core.c#L2641-L2672 https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4... While this seems to be special handling for Samsung X5 SSD external drives, and also Samsung 970 Evo Plus drives: https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4091ff99df26c67c9021/drivers/nvme/host/core.c#L2674-L2685 https://github.com/torvalds/linux/blob/69f2c9346313ba3d3dfa4...

gregatragenet3 4y ago

This posting came two months late for me. My 980 2TB crossed the river styx over the holiday break. Failure mode exactly as described. Nice Christmas present for me. Took 3 weeks to get the warranty replacement from Samsung.

dataflow 4y ago

Tangent: what's the cheapest 4TB+ PCIe SSD without significant known issues but with hardware encryption? The SN850x seems to have its own issues, and beyond these everything else seems so expensive.

nine_k 4y ago
Is hardware encryption a widespread feature? What are the benefits of it for you, compared to OS-level full disk encryption?
wnevets 4y ago
> The SN850x seems to have its own issues Such as?
- dataflow 4y ago
  https://community.wd.com/t/sn850x-the-driver-detected-a-controller-error-on-device-raidport0/281205 https://community.wd.com/t/sn850x-the-driver-detected-a-cont...

evil-olive 4y ago

every machine I have that can fit 2 SSDs (basically, everything except the very slim laptops) I have converted over to running a ZFS mirror as its root filesystem. NixOS makes this very easy to do because the grub.mirroredBoots option [0] removes the need for a separate "bootpool" with limited ZFS feature flags. and crucially, I always make sure they're 2 drives from different manufacturers, so that a bug of this nature should never be able to take down both drives in a pool simultaneously. I think of this as the "if you're going to go to the trouble of wearing a belt and suspenders, make sure to buy them from separate brands" principle. 0: https://search.nixos.org/options?channel=22.11&show=boot.loader.grub.mirroredBoots https://search.nixos.org/options?channel=22.11&show=boot.loa...

deleted 4y ago
[deleted]

Havoc 4y ago

> high failure rates in the field Jikes. A lot of people were hoping that the high wear is just a reporting artifact

craigching 4y ago

I have a puget system with the Samsung SSDs mentioned and it locks up on me every 2-4 weeks. This sounds like it would explain the problem. Puget sent out a message this week to upgrade the Samsung firmware, but I am at the latest. I’ll be contacting Puget support on Monday so I’m on their radar. I will say I love my Puget, its performance has been killer other than these lockups. And I’ve heard only good things about Puget support. I should have reported this months ago, but it’s just now that I’m doing some critical work on Windows that it’s affecting me.

userbinator 4y ago

For the same price, you can get twice the space for 1/4 the endurance, thrice the space for 1/8th the endurance, and now four times the space for 1/16th the endurance. Most people don't realise that is a horrible tradeoff, because NAND flash marketing and terminology like "TLC" or "QLC" is intentionally deceptive and manufacturers have been very secretive about the true endurance specifications, as well as trying to overprice SLC out of production. If more people knew the truth of what they were trying to do, we wouldn't be in this situation.

newZWhoDis 4y ago
Can you explain your point further? Are you talking about competitors?
wmf 4y ago
While NAND endurance has certainly gone down, FTLs got much better during the same time so that SSD endurance is still fine for most people. And if the stock endurance isn't enough, a little overprovisioning is probably better than dropping back to very expensive MLC.
- userbinator 4y ago
  If the number of firmware bugs in SSDs we've seen over time is any indication, I don't think things are really getting better... SLC needs almost no FTL. 100K endurance. Very low raw error rate that can be handled with basic ECC.
wahern 4y ago
> as well as trying to overprice SLC out of production. Is it even possible to buy SLC drives any more? For the past 5+ years the only outlet I've been able to find that even advertise SLC is https://www.delkin.com/ https://www.delkin.com/, and you need to speak to sales to even get a price. I just assumed they and any other similar suppliers bought giant lots of chips at the tail end of SLC production and jack up the price on every new order as their supply dwindles. Or maybe they cobble together drives from the tiny SLC chips used for cache on modern SSDs?

blindriver 4y ago

Is there any way to update my SSD on Windows without downloading Samsung Magician? I have the Samsung Evo across multiple systems and my Linux ones are okay but my Windows machines aren't okay unfortunately, they have the bad firmware.

deleted 4y ago
[deleted]

taspeotis 4y ago

We have Lenovo laptops at work with M.2’s that are OEM-branded Samsungs. One bricked itself in to read only mode after a few months. The other has been losing 1% health each week or so. I caught it losing 2% in just two days recently. These drives are older than the 990 model mentioned in the article but I have my suspicions anyway they’re dud drives. Nothing lost except time - they can be swapped under warranty. But I used to buy Intel exclusively before swapping to Samsung when Intel started selling rebranded drives. I guess the search for a reliable vendor starts again…

jeffbee 4y ago
These anecdotes are pretty frustrating without the other key piece of information. For the given lifetime indicators, how many writes were served? Are they wearing out faster than their TBW claims, or are they being written more than you expected?
- taspeotis 4y ago
  It’s at 86% with 12.21TB written. Total power on time 68 days. Drive temp sits around 45 degrees celsius. I’m not paying great attention to all the SMART counters day by day. It’s for a dev workload so like … compiling code and stuff? I have the exact same workload on my desktop PC and its Samsung drive health is 99% after … years.
  - jeffbee 4y ago
    OK, so close to 1% lifetime per TBW, or lifetime approximately 100TBW. Thanks! That's consistent with their endurance claims for the smallest SSDs (128GB PM991 for example, or 256GB 960 Evo) but it would be poor for a larger one.
    - taspeotis 4y ago
      Cool well these are 2TB drives so…
chemmail 4y ago

emodendroket 4y ago

Wouldn't be surprised if this is another pandemic supply chain casualty.

jandrese 4y ago
Sounds more like crappy firmware to me. This is not the first SSD to suffer from a crippling frimware flaw that is impossible to fix. I lost almost an entire computer lab of Dells thanks to the goddamn Sandforce firmware. One that the company acknowledged but refused to lift a finger to fix. Luckily it is possible to fix these yourself despite the vendor hostility towards the repair. Look how easy it is: https://computerlounge.it/how-to-unbrick-sandforce-ssd/ https://computerlounge.it/how-to-unbrick-sandforce-ssd/

mmis1000 4y ago

There is a chinese youtuber doing SSD durability tests on a few SSDs of different vendor. And one of the tested ssd is Samsung 980 pro. What is the funny thing? The Samsung 980 died before the wear test even start. https://youtu.be/tXYQZHz7u3w?t=898 https://youtu.be/tXYQZHz7u3w?t=898

JonathanFly 4y ago

Magician Smart Status reports 0 'media errors' on my 980 2TB. Is that suspiciously low or does it mean things are actually ok? Is there a different tool I should check with?

sebazzz 4y ago

SSDs are complex beasts :-) I've had an OCZ SSD in the past that also became read-only, in the sense that all changes after shutting down the computer were gone. If you rebooted everything was fine, but as soon as the computer was shut down and the SSD also ran out of power, it converted to the state before turning on the computer. That was so bizarre. Once I had a hefty Windows upgrade installed and it was gone after the reboot - as if I had never installed it. It also took me a while to realize it because first of course you start to doubt your own memory, you don't realize that the sector- mapping on the SSD has suddenly become read-only OCZ eventually solved this via an RMA and eventually OCZ also went upside down. The Time Warp bug it was called I think.

mvdwoord 4y ago
sounds ideal for a public terminal ;)
- csdvrx 4y ago
  Marketing should make that a feature: hardware-guaranteed non-modifiable root filesystem! Better than Nix!
  - totalZero 4y ago
    No updates! Ever!
    - ashirviskas 4y ago
      Also, no malware!
- sebazzz 4y ago
  There used to be a PCI module (iirc) that would assist in that. I believe it was called a Bo(u)rne Again module. It was used in school computers etc.
  - AdamJacobMuller 4y ago

talhah 4y ago

If you're using Linux you can use the following instructions to update the firmware. https://wiki.archlinux.org/title/Solid_state_drive#Update_under_Linux https://wiki.archlinux.org/title/Solid_state_drive#Update_un...

fulafel 4y ago

Anyone know if the failure mode is sudden total bricking or something more recoverable?

ThomasGlanzmann 4y ago

In 2021 I bought at least ten 870 EVO 4 TB SATA and six 980 Pro 2 TB NVMe. All devices failed within 6 months on barely used systems. Find some smart data here: <https://thomas.glanzmann.de/samsung/ https://thomas.glanzmann.de/samsung/> The pattern is always the same: I have them configured in a raid 1. Once a month debian does a raid check. During the raid check Debian reads all data from both devices. I get uncorrectable read errors. I no longer use Samsung SSDs and replaced them them with SSDSC2KB076T8, Micron SSDs and KC3000 Kingston NVMes. No failures since then. In 2021 I told a friend of mine about the issue. He also had a 870 EVO, issued a dd if=/dev/sdX of=/dev/null bs=8M and guess what, he got uncorrectable read errors. Due to running them in RAID 1 I caught the issue early and I had no data loss or downtime because the Linux software raid compensated for the bad hardware. However I replaced them in a hurry because I no longer trust Samsung SSDs. As you can see from the smart log they're barely used. Less than 4 months in service and 10 TB written. I also got uncorrectable read errors when evacuating data from the devices.

lazyweb 4y ago
Similar experience for me. Four 2TB 980pro in RAID-Z2. Since introducing the drives in August 2021 I've had to replace them five times. I think none of the original SSDs are left. Only between 5 - 25TB written on average. Usually individual uncorrectable errors caught by ZFS, but one drive just straight up died. I keep a cold spare these days. On the bright side, if the cycle continues, I'll never run out of warranty from Samsung or the vendor.
- ThePowerOfFuet 4y ago
  > On the bright side, if the cycle continues, I'll never run out of warranty from Samsung or the vendor. The warranty is from the date of sale of the original unit. Replacing one doesn't reset the warranty end date.
  - gowld 4y ago
    If this really becomes an issue due to repeated failure, lemon law activates for a refund.

sireat 4y ago

I can live with my batch of 980pros and 870 EVOs failing. Bad batches happen to every manufacturer. However this seems like a systematic problem at Samsung. What really stinks is that I've been recommending Samsung EVOs and PROs to relatives and friends for some years. If I want to remain honest with myself I have to contact every one of them and have to run the SMART tests. So now there are NO reputable SSD manufacturers left. Only reliable SSDs are MLCs from mid 201Xs especially Intel and Samsung.

ChewFarceSkunk 4y ago

[dead]