3 ms·
Ars had a piece covering this as well, and I do wonder if there is something going on somewhere else in the Samsung stack, not just the NVMe 900 series line. Pu
by xoa 4y ago
Ars had a piece covering this as well, and I do wonder if there is something going on somewhere else in the Samsung stack, not just the NVMe 900 series line. Pure anecdote, but two years ago I did a NAS for a client using 24x 2TB Samsung 870 Evo drives (they'd gotten some incentive deal for it). While it was all one type vs mixed, there was the "luxury" of time because at that point getting the system they wanted together had a significant lead time. So I did ensure that the drives were purchased over the course of around 7 months, from multiple different reputable sellers (B&H, CDW, Provantage etc) in separate batches. System was solid, an Epyc 2 based SuperMicro server, running TrueNAS.
And then last year with around 5500-7500 quite light hours of runtime (primarily reads, ~0.08 DWPD, well under official rating of 0.3 DWPD) drives started failing. These were definitely real failures, first indication came from regular automated ZFS scrubs and reporting increasing checksum errors and ATA errors. It was for so many drives and I'd always considered Samsung SSDs relatively reliable (even for consumer ones) that at first I thought it was a SATA controller failure, and our rep agreed and warranties back the server. They were great, gold plated support contracts pay off once in a while, and motherboard replacement and thorough testing later back in service. More drive problems. SMART short tests said everything was healthy, first longs did too. But then drives exceeded error limits and started getting faulted, and at last SMART long tests started failing. Digging in showed worrisome stats. So began swapping out and warrantying drives (cheers to the stress test to TrueNAS, in the end zero downtime or need to restore from backups). In the end, THIRTEEN (13) out of 24 failed. Brutal >50% dead drive rate. I talked to some others around and they'd seen <1 year rates also at 30-60%. Big :\. Rep also indicated they were hearing more about Samsung failures.
Anyway, gave me a talking point going forward to really, really press management on "it's worth paying for drives from 3-4x brands and maybe splurging for higher rated vs consumer too", but also does made me wonder if there is something going on, or was (pandemic related?), at Samsung's storage division. It's definitely pure anecdote but still, I spread those drive purchases out reasonably hard, and they had radically different serial numbers. Same with other folks I know at other businesses using various Samsung drives, everyone has been going to real effort following decent practices to prevent buying drives all from a single lot. Even 10% failure rate for consumer drives I could have seen, but 54%? And not a bathtub curve all frontloaded in the first month or two but after 7-11 months? That feels high? Samsung did replace them all no questions asked, they paid for shipping too. I don't have any global insight into how this all looks and it could be just plain bad luck for all of us in the region, but still.
- paulmd 4y ago> Anyway, gave me a talking point going forward to really, really press management on "it's worth paying for drives from 3-4x brands and maybe splurging for higher rated vs consumer too", but also does made me wonder if there is something going on, or was (pandemic related?), at Samsung's storage division. What's going on at Samsung is the same thing that's always been going on at Samsung: they use their own flash and their own controllers and they have their own sets of problems as a result. It was a selling point in the early days of Sandforce and other turds (leading to things like the OCZ Vertex series) but now the commodity market has caught up and Samsung doing their own thing is kind of a negative. Like yeah it's fine as long as they don't screw up, but they're screwing up a lot more than they used to. I don't see any direct correlation between the various failures that have occurred over the years. 840 Evo had something wrong with the flash NANDs that caused them to lose charge over time (leading to data loss) so they put out a new firmware that would just continuously write the flash in a tape-loop sort of deal (lol) to avoid the data ever aging out. I don't count that as a controller flaw, that's a flash flaw that was fixed with a firmware patch. 870 Evo, 970 Evo, 970 Evo Plus, and 980 have all been accused of having problems over the years, in addition to the 980 Pro and now 990 Pro, but there's actually a pretty good variety of controller models as well as different flash types (from 64-layer to 236-layer) there. It's hard to know how much firmware they all share though... or whether it was more flash problems in certain batches, or what. But overall Samsung certainly has had a lot of failures in recent years and I think it all really comes back to the fact that they're using their own controllers and their own flash, while everyone else is pretty much commodity at this point... meaning they get their own bugs too. https://docs.google.com/spreadsheets/d/1B27_j9NDPU3cNlj2HKcrfpJKHkOf-Oi1DbuuQva2gT4 https://docs.google.com/spreadsheets/d/1B27_j9NDPU3cNlj2HKcr...