15 ms·

WD Red DM-SMR Update: 3 Vendors Bail and WD Knew of ZFS Issues

309 points by arm 6y ago

stingraycharles 6y ago
I’m very happy that this tactic has backfired so spectacularly for WD. I don’t know what the total impact on their bottom line is, but what I’m seeing is a huge reputation as damage for WD. People are avoiding putting WD in their NAS in general, even if they have plenty of drives that are fine with it!
- tjoff 6y ago
  I have such a hard time to understand how the risk vs. reward ratio for pulling this stunt made any sense. SMR does not have a great reputation, to put it in drives specifically targeted for NASes and enthusiasts is a very bold move.
  - kllrnohj 6y ago
    Given the price premium on these drives in the first place I don't even get why they'd do this? Stick the SMR drives in the EasyStore line or whatever, which is regularly half the $/gb of the bare RED drives. That'd probably also discourage some shucking.
    - smacktoward 6y ago
      In every company, no matter how obscene the margins they're making on a given product, there is someone arguing that those margins could be even larger if they just cut a few corners. Eventually someone listens to that person, and decades of painstakingly built-up brand value gets thrown out the window in order to bring home six nickels tomorrow instead of five.
      - deleted 6y ago
        [deleted]
    - cbsmith 6y ago
      As mentioned in the Manfred Berger talk, the tech is supposed to, in theory, get you a 25-50% cost effectiveness advantage. Obviously the market prices don't reflect that, but that explains why they'd do this.
      - loeg 6y ago
        Reality seems to be more like 10% on a capacity basis. On a performance basis, of course, it's extremely negative.
        cbsmith 6y ago
        Magnetic disks are generally valued on capacity, but at least as I heard the talk, he was talking about it growing to a 50% difference as they pushed the limits. I get that the reds are more performance oriented, but capacity is still a key factor (if not, you just go SSD and be done with it).
        loeg 6y ago
        I get that 50% has been the sort of marketing aim for "why bother support SMR at all," but until they actually get there, it's just marketing. Reality is ~10%. Magnetic disks are valued on capacity only because people considered CMR HDDs a performance floor. When they realize how bad SMR is, I think performance becomes a factor. If you're talking about GB per volume, SSDs beat HDDs of all stripes there: the biggest HDDs are 16 TB in a 3.5" form factor, while the biggest SSDs are 100 TB in the same form factor. 10 or 50% improvement in HDDs wouldn't begin to bridge that gap. SMR should be understood to be sort of a fast Tape, rather than slow CMR HDD.
        Dylan16807 6y ago
        > SMR should be understood to be sort of a fast Tape, rather than slow CMR HDD. I wouldn't go that far. If I can assume that the firmware won't cause a lot of problems all by itself, I think in my desktop I'd slightly prefer a 7200RPM SMR drive to an equal-price 5400RPM CMR drive. And no matter what kind of hard drive(s) you're using, take a $20 SSD and use half of it as a cache with writeback enabled.
    - kasabali 6y ago
      They do. Corresponding 2-6TB Blue drives are also SMR and that are shipped inside external drives. Only Reds got the attention because of ZFS issues. I suspect Blue and Red drive have a lot of common in manufacturing they switched all drives in same capacity together.
      - andoriyu 6y ago
        Red is getting attention not just because of ZFS, but because Reds are targeted and marketed for workloads that don't work well with SMR. It was well known before that Red are trash compared to Red Pro which IIRC are rebranded HGST DeskStar NAS.
  - rwmj 6y ago
    > SMR does not have a great reputation SMR is fine if you know what SMR is and what you're doing. It might even be fine given a sufficiently advanced translation layer (which apparently doesn't exist yet). The problem isn't SMR, it's selling a product which uses SMR and hiding that fact from the consumer, and I can't forgive WD for that.
    - yjftsjthsd-h 6y ago
      > It might even be fine given a sufficiently advanced translation layer (which apparently doesn't exist yet). Apparently f2fs does well with it?
      - simcop2387 6y ago
        Not terribly surprising, f2fs and similar filesystems are log-based systems. Rather than overwrite the old data they almost always try to write to the next new area. This ends up working really well for flash technology because writing to existing areas is expensive. You have to read in a 1-4MB area depending on the block size, make your changes to that, erase that area entirely, and then write the changed version. [Those sizes might be a little off but it's what I remember]. Some of that may be hidden by the disk's controller but it has to happen on every write to an area with existing data. SMR is similar in this respect because you have to read, change, and write large blocks together because of the way the tech works. So I'd expect anything that's directly optimized for low level flash memory to work similarly ok performance wise for it. The real problem comes when you can't work like that, in particular this happens when you're low on free space or if you're in a situation where relocating data isn't possible or known that it needs to happen. RAID setups are typically the worst case for this, since they'll effectively treat the entire disk as used and can't relocate data on it. ZFS is a bit more complicated with this but effectively ends up the same.
    - deleted 6y ago
      [deleted]
    - m4rtink 6y ago
      There have been some experiment with modified XFS using SMR drives natively: https://www.snia.org/sites/default/files/SDC15_presentations/smr/HannesReinecke_Strategies_for_running_unmodified_FS_SMR.pdf https://www.snia.org/sites/default/files/SDC15_presentations... You put the journal and metadata on the continuous part and data on the shingled part, making sure you only ever add to what was written previously in a shingled segment by using copy-on-write. And sporadically you reclaim a segment. This should be much more efficient than a block layer translation layer that knows nothing about the filesystem.
  - willcipriano 6y ago
    >I have such a hard time to understand how the risk vs. reward ratio for pulling this stunt made any sense. Classical economics likes to think of firms as rational entities working to maximize profit over the long term. In my experience that makes much of what companies do inexplicable. Instead I view firms as large groups of individuals all acting in their own self interest, this seems more complex but really all you have to know is what the incentive structure is to understand the choices a company makes. Similarly I've found you can also do this in reverse. It is a virtual certainty that from a compensation perspective moving to SMR was better for the individuals who made that choice.
    - TheSpiceIsLife 6y ago
      Fortunately, economic theory is more broad than classic economics. Managerial and behaviour theories here: https://en.wikipedia.org/wiki/Theory_of_the_firm https://en.wikipedia.org/wiki/Theory_of_the_firm Linked to from the Economics article under the Theory section: https://en.wikipedia.org/wiki/Economics https://en.wikipedia.org/wiki/Economics The theory your suggesting here has been known since at least the late 50s. You might also be aware of, or find interesting anyway, the principal–agent problem: https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble... Edit: typo
      - jacques_chester 6y ago
        I don't think this reply should be downvoted. It might be slightly spicy, but it does reflect that economics in the 20th century made enormous strides in understanding the observable behaviour of actual firms and markets. Transaction costs, agent problems, information economics, institutional economics, mechanism design and so on are powerful insights into the world around us. I've learned more about how software grows and spreads from economists than from software engineers.
        Dylan16807 6y ago
        It's a useful bit of information, but it's insulting the comment it replies to without actually disagreeing at all. There's a reason the initial comment said "classical" economics! I can see why it earned some downvotes.
  - tzs 6y ago
    Ars Technica did some detailed testing of these drives a couple weeks ago. See "We put Western Digital’s dreaded SMR Red drive to the test" [1]. They found that they won't work for ZFS but are actually OK in most NASes. Here is their conclusion: -------- begin quote We want to be very clear: we agree with Seagate's Greg Belloni, who stated on the company's behalf that they "do not recommend SMR for NAS applications." At absolute best, SMR disks underperform significantly in comparison to CMR disks; at their worst, they can fall flat on their face so badly that they may be mistakenly detected as failed hardware. With that said, we can see why Western Digital believed, after what we assume was a considerable amount of laboratory testing, that their disks would be "OK" for typical NAS usage. Although obviously slower than their Ironwolf competitors, they performed adequately both for conventional RAID rebuilds and for typical day-to-day NAS file-sharing workloads. We were genuinely impressed with how well the firmware adapted itself to most workloads—this is a clear example of RFC 1925 2.(3) in action, but the thrust does appear sufficient to the purpose. Unfortunately, it would appear that Western Digital did not test ZFS, which a substantial minority of their customer base depends upon. These tests may not be great news for either the American or Canadian class-action lawsuits currently underway against Western Digital, but they aren't the end of the line for those lawsuits, either. Even in the best case, the SMR models of WD Red underperform their earlier, non-SMR counterparts substantially—and consumers were not given clear notice of the downgrade. If the same firmware was being used to make substantially larger drives available to consumers than would otherwise be possible, and the limitations of those drives were adequately explained, we would probably be gushing over its utility and function. Unfortunately, Western Digital has so far only chosen to use it to cut manufacturing costs on small disks, without even passing the savings along to the consumer. -------- end quote That last paragraph may be key. It sounds like if these drives had been marketed a little different and priced a little different they could have been a good deal for many NAS users. I wonder what the chances are that they were originally intended for just that, and something got mixed up or miscommunication between design and release? [1] https://arstechnica.com/gadgets/2020/06/western-digitals-smr-disks-arent-great-but-theyre-not-garbage/ https://arstechnica.com/gadgets/2020/06/western-digitals-smr...
    - vetinari 6y ago
      I did a reshape, in phase 1 from 4-drive RAID5 to 5-drive RAID6 and then, in phase 2, from 5-drive RAID6 to 8-drive RAID6. Everything with Synology NAS, with typical Synology setup (mdraid, btrfs on top). The new 4 drives were the infamous 6TB EFAX. The phase 1 took 12 days, the phase 2 took 3 days. So my personal experience is radically different than Ars Technica's.
  - PaulHoule 6y ago
    SMR is not as bad for desktop-style workloads as you might think. If you have enough storage for cache and metadata, you can get great performance because all of your writes are sequentialized. If you had 32GB of Optane and 4 TB of SMR properly tuned that would be one heck of a desktop drive, but (1) you have to really eliminate all the bottlenecks and (2) who needs a desktop drive large enough that SMR is worth it?
    - StillBored 6y ago
      The problem is that once the buffers are full the drive effectively stops accepting commands while it rewrites a SMR region. When that happens depending on what your doing, the drive might just get kicked from an array, or your whole system might freeze if some critical page needs to be read back in and it can't get sufficient priority and a command slot. I guess the vast number of people won't ever write more than a buffers full of data, or the drive will never get fragmented sufficiently that even small write operations amplify into entire drive rewrites.....
      - londons_explore 6y ago
        Smarter firmware would detect the case you are discussing, and re-designate a part of the SMR drive to be regular(non-SMR space). For that to work, you need the host OS to be able to see that the drive has just "shrunk". Obviously, you still have data on it while it's shrinking, so the reality is the OS needs to give the drive a list of sectors which can be "shrunk away". One simple approach would be for a daemon to create a massive file filled with "MAGICBYTESMAGICBYTESMAGICBYTES...". As that data is 'written' to the drive, the drive sees that it's the magic bytes, and rather than storing the data, simply marks those sectors as no longer needed. As soon as enough sectors in a row have that designation, reformat them as non-smr, and use as regular (non-smr) disk space. Then, a few hours or days later, the drive can rewrite the data back to be SMR, then tell the daemon, which can remove the magic bytes and delete the files, and your free disk space increases again. The failure mode of this is that your "6TB" SMR disk ends up with a 3TB file filled with MAGICBYTES, and 3TB of your data. But for the same money, you'd only get to store 3TB of data on a non-SMR drive anyway...
- NortySpock 6y ago
  Western Digital-branded drives not showing on BackBlaze hard drive reliability tables probably left WD in a blindspot for people looking for reliable drives anyways. https://www.backblaze.com/blog/hard-drive-stats-q2-2019/ https://www.backblaze.com/blog/hard-drive-stats-q2-2019/ https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2020/ https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1...
  - ksec 6y ago
    HGST is under Western Digital.
    - syshum 6y ago
      HGST brand was killed, UltraStar is now WD Branded not HGST
  - arminiusreturns 6y ago
    I've been reading the backblaze report since it started coming out (thanks to the backblaze team as always for being so transparent!). It become obvious to me as far back as 2015 that WD was a brand to avoid... and it seems by the end of 2019 backblaze had phased them almost completely out... for a reason.
- sleepydog 6y ago
  The hard drive market has shrunk considerably in the past ~5 years. What other choices are there? I remember Seagate having a not-so-great reputation in the late 00s, but now they seem like the most obvious alternative. Are we going to be stuck oscillating between ~4 vendors?
  - tomatocracy 6y ago
    Toshiba are a reasonable bet now. I have several of the N300 drives which have performed pretty well so far.
  - occamrazor 6y ago
    There are actually only three producers left: WD, Seagate and Toshiba.
    - kevin_thibedeau 6y ago
      Toshiba is the only one without a tarnished reputation.
      - yellowapple 6y ago
        And Backblaze seemed to think highly of 'em as of 2015: https://www.backblaze.com/blog/a-look-at-backblazes-toshiba-hard-drives/ https://www.backblaze.com/blog/a-look-at-backblazes-toshiba-... It's interesting that Toshiba apparently acquired Hitachi's production assets during the HGST→WD acquisition. HGST had a pretty good reputation at the time, so I feel like this bodes well for Hitachi if their drives are indeed a continuation of that. If I do need some spinning rust at some point I guess I'll have to give 'em a whirl.
      - jclulow 6y ago
        Have had some serious and deeply negative experiences with mixed workload performance on Toshiba disks in the last five years. Allegedly enterprise-grade 2.5" spindles, becoming deeply confused when handed both reads and writes concurrently. Test with your workload, of course, but my default position is: beware!
- MintelIE 6y ago
  Personally I won't buy another WD or Sandisk product ever again. Companies like this don't ever really come off as being sorry for the scams they pull, so it's best to just cut them off completely. I de-Googled too because of their shady practices. EDIT: For those who can't use the Internet to look things up, Sandisk and G-Technology and some random thing called Upthere are all WD brands.
  - vl 6y ago
    What’s wrong with Sandisk?
  - mey 6y ago
    What has Sandisk been up to?
    - tw04 6y ago
      WD owns Sandisk. https://www.theverge.com/2016/5/12/11662018/western-digital-sandisk-deal-complete https://www.theverge.com/2016/5/12/11662018/western-digital-...
- washadjeffmad 6y ago
  A little too much outrage for my taste. Every major HDD manufacturer was found offering SMR. WD was just the first to be held to task because they're so incredibly popular with shuckers. Seagate SMRs were found before WDs because we only purchased ≥8TB EasyStores, which are unaffected. For the lines they were supplied under, SMR characteristics don't impede 95% of customers. The rest are small-timers trying to avoid paying for warranties. No one buying Enterprise NAS was affected. edit: I guess we all decided to disagree by downvote today. Words tomorrow, maybe?
  - jschwartzi 6y ago
    Actually I think you’re missing the point entirely. I have a 4 bay NAS that I use to back up all of my and my fiancé’s media so losing a phone isn't a catastrophe. I’m trusting that the NAS will actually rebuild in the event of a drive failure or it provides very little actual protection. WD shipped drives which were very unlikely to rebuild successfully. The upshot of that is that they lied to me and as a consequence have placed priceless data at risk from a drive failure. And had I known that I wouldn’t have bought their drives, so I’m materially harmed by it. I don’t shuck drives, I buy through retail channels like Newegg. I think you’re getting downvoted because you don’t understand the gravity of the problem and your victim blaming. Otherwise there’s nothing to disagree with because what you’re saying isn’t really supported in fact.
    - washadjeffmad 6y ago
      Thanks for helping me understand the demographic affected. This isn't a rebuttal, but I hope you don't lose anything, so you should know that a NAS is not a backup. They provide redundancy and availability, but no number or type of disks will save your data from being lost in a fire or flood if you keep it all in one place. SMR drives, if you know what you're buying them for, are perfect for backups and write few, read many applications, which it kind of sounds like you want. They're not inferior or at premature risk of failure, just specialized. Also, I'm not saying you have no right to be unhappy with finding out why, but were you really dissatisfied with the performance of your NAS before these hyperbolic exposes went out?
  - 6y ago
- m463 6y ago
  If only it had backfired for everyone when the marketing folks changed from hexadecimal capacity sizes to decimal. 1000000 = 1M
  - nayuki 6y ago
    1,000,000 is exactly equal to 1 mega. It's the other part of the computer industry (Windows Explorer, RAM sizes) that abused the notation to pretend 1 kilo = 1024. See: https://en.wikipedia.org/wiki/Binary_prefix https://en.wikipedia.org/wiki/Binary_prefix
  - mjevans 6y ago
    My argument has been: Can I store just ONE byte at a time? Can I expand by singular bytes in a normal system? For hard drives, and even flash memory, the physical size of a sector converged, rather naturally, to a multiple of system word size, and since that had also converged to a power of 2, to a larger power of 2. Thus hard drives have sectors that were 512 bytes (4096 or '4Kbits'), and also why the smallest write available for modern drives is a multiple of 2. Binary is the native nomenclature of computers. It only makes sense that related equipment should also use the native (binary) engineering unit approximations as a result. Though for communications equipment, they do have variable sized packets, which even though computers generate 8 bit 'octet' based signals might technically be used by some other format that doesn't; and talking about the actual speed on the wire conveying that in maximum possible raw bit states is a tiny bit confusing when the rest are binary units, but it's less strongly wrong than a system where the lowest addressable (real) unit is binary based.
    - nayuki 6y ago
      You're basically saying that prefixes like kilo- have a context-dependent meaning - usually 1000, but sometimes 1024. We know how well that worked out historically, right? e.g. https://en.wikipedia.org/wiki/Ounce https://en.wikipedia.org/wiki/Ounce "the international troy ounce is equal to exactly 31.1034768 grams", "The international avoirdupois ounce is defined as exactly 28.349523125 g", "A fluid ounce (abbreviated fl oz, fl. oz. or oz. fl.) is a unit of volume equal to about 28.4 ml in the imperial system or about 29.6 ml in the US system". By not respecting the definition that kilo = 1000, you are deliberately sowing confusion and undermining the metric system as a simple, consistent, universal measurement system.
- yellowapple 6y ago
  Yeah, that move turned me from being a more-or-less diehard WD fan (i.e. picking it as my first choice for spinning disks) to being really skeptical of it; if they're willing to cheap out on that, then for me it casts doubt on the reliability of the hardware, too (and reliability is literally my only concern, even for my own desktops and laptops; WD could sell drives at a tenth of the performance as, say, a Seagate, but if it's even a little bit more reliable - which, in my observation from a variety of different use cases, it historically has been - I'll pick it every time). The good news is that I've basically stopped buying spinning rust, so this is more of a "well bummer" type of mood than an "oh shit I gotta scramble for a better primary vendor" mood. Even for server/NAS use I'm more likely to go for a Crucial/Kingston mix nowadays than I am for a WD/Seagate mix, both because SSDs have matured enough to be more reliable for a lot of things (less moving parts = less mechanical wear/tear and less sensitivity to vibrations¹, and with RAID a dying flash module is less of a "game over" moment - though things have gotten a lot better on this front, too) and because they perform a lot better (both in terms of speed and in terms of thermal/energy efficiency). ---- ¹: https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
  - CyberDildonics 6y ago
    For SSDs you can look at the individual components they are made from, like the controller boards.
  - esyir 6y ago
    Unfortunately for bulk data storage, spinning rust is currently the only viable option at the consumer level. For now, until ssds reach competitive prices at the same scales, we're stuck with them.
    - yellowapple 6y ago
      For how I typically lean for storage (RAID1 "cells" with each disk from a separate manufacturer, combined into either a RAID10 or a JBOD), and thus how I typically buy drives, SSD v. HDD is actually surprisingly close price-wise, especially when looking at 2.5" SATA or SAS (which is what I typically buy, both because 3.5" SSDs strangely don't exist and because I can stuff more of 'em into an enclosure; plus, quite a few servers don't even take 3.5" drives). For example, consider two different 2TB cell setups: - SSD: Crucial MX500 + Samsung 860 QVO = $473.73 on Newegg - HDD: WD Blue WD20SPZX + Toshiba L200 = $168.57 on Newegg On the one hand, the SSD cell is a whole 2.81× more expensive. On the other hand, the SSD cell is only 2.81× more expensive. This narrows to 2× in the case of a 1TB (well, technically 960GB in the SSD case) cell setup: - SSD: Crucial MX500 + Kingston A400 = $236.54 on Newegg (this is my typical setup for a RAID1) - HDD: WD Blue WD10SPZX + Toshiba L200 = $117.13 on Newegg Now, to be fair, this is excluding Seagate, which is stupidly cheap and would widen that gap (and historically I've bought Seagate strictly as a secondary for these sorts of "cells", but given my bad experiences with Seagate I'm inclined to ignore them and go with WD+Toshiba if possible for a given target size). It's also excluding 3.5" drives (and therefore notably excluding WD Reds entirely), which doesn't help for a 1TB cell (Seagate Cheetah NS + WD Red = $118.75) or a 2TB cell (Seagate IronWolf + WD Red = $167.74), but does help for bigger cell sizes that wouldn't be possible with 2.5" at all without major sacrifices in vendor redundancy and price (for example, a 4TB cell w/ Toshiba N300 + WD Red = $225.91, v. Samsung 860 QVO v. WD Blue = $1049.17 - even higher if I wanted to keep Kingston in the mix - or two of the above 2TB cells for only slightly cheaper). On the other hand, this is including 5400 RPM drives; if I required 7200 RPM (as most NAS drives are) then the 1TB cell cost jumps up to $134.98 (WD Black + Seagate BarraCuda), and if I needed 10k RPM or SAS then I'd probably be springing for a WD XE WD9001BKHG + Toshiba AL13SEB900, which would bring the cell cost up even higher to $164.75 (while also losing 100GB of capacity), or even higher to $186.38 if I wanted to swap the WD for a Seagate Savvio ST9900705SS and have that 64MB of cache on both sides of the mirror (thus bringing the cost difference to a mere 1.27×). I'm also artificially keeping out a whole swath of SSD manufacturers in the above figures; never heard of "Team Group" or "Goldenfir", but if I had zero qualms about reliability I could put together a 1TB cell with those for $182.98 (on par w/ the 10k RPM cell), or for a 2TB cell I could go with a Crucial BX500 + Patriot P200 for $399.98. ---- That all being to say: SSDs are a lot cheaper nowadays than they were even a couple years ago, and at typical consumer scales they're plenty viable; "good enough" capacity, while also having better performance and lower power consumption.
AdmiralAsshat 6y ago
I imagine WD has said nothing so far because they're probably waiting for their lawyers to tell them exactly how much they can claim to have known/not known without opening themselves up to litigation.
- tssva 6y ago
  WD responded via a blog post when the use of DM-SMR for WD Red drives was first exposed. It basically consisted of an attempt to gaslight those that bought Red drives into believing that this is their fault for choosing drives inappropriate for the purpose they using them for. Who could think using Red NAS drives in a NAS would be an appropriate technology choice? /s
  - ashtonkem 6y ago
    That blog post will go down as one of the all time worst examples of corporate crisis management.
    - jldugger 6y ago
      I feel like boing is at least in the running here.
  - TheSpiceIsLife 6y ago
    I saw the headline at the time, but didn’t read the blog post because reasons. Does anyone have a link or archive link?
    - bonestamp2 6y ago
      I believe the April 20th (bottom) section of this blog post is the post in question: https://blog.westerndigital.com/wd-red-nas-drives/ https://blog.westerndigital.com/wd-red-nas-drives/ TLDR: They suggest different RED drives should be considered for different workloads. While that may have been their intention all along, that was not communicated well at in marketing or retail, where the drive colors (red, blue, purple, etc) used to be the appropriate method for selecting the right drive for your workload/purpose. The way they have worded it, the blog is not wrong, but they left out an important fact: consumers didn't have the necessary info to make the decision they're suggesting people should have made. Even if you read the data sheets that were available at the time, one would expect the listed transfer rate was a limit that the drive was capable of, not the rate that if exceeded would cause the device to perform very poorly. Worse, there would be no way to know that was the case because the fact that they were SMR technology was previously unreleased and the drives themselves actively hid that information from the host computer (since they were "device managed" SMR). The data sheets still aren't clear on how the performance is impacted with workload, but at least they're clear about the technology they use now.
S_A_P 6y ago
Interestingly these drives are on sale on newegg.com right now.
- mark-r 6y ago
  That's what happens when you have trouble selling something.
  - Hamuko 6y ago
    Do they actually have trouble selling? How many people shopping for these kind of drives understand what kind of a difference SMR has theoretically and what kind of a difference it has for them? When you Google "best NAS HDDs", you'll basically run to a bunch of articles recommending WD Reds alongside Seagate Ironwolves. I think only one link on the first page of results had a warning against the lower capacity WD Reds due to this.
- blibble 6y ago
  that's because they know they're going to be worth essentially zero very soon
  - metalliqaz 6y ago
    the SMR debacle has been going on for months now, the prices haven't taken much of a hit
- loeg 6y ago
  This is generally why shopping sales/discounts as a first-order criteria is not great for value. Find some acceptable parts first, then price hunt those parts.
- kabdib 6y ago
  Yup. I narrowly escaped buying one of these drives as a spare for a Synology. Anecdotal evidence is that the SMR drives will mostly work. It's not worth the time or trouble. WD is dead to me until the next cycle of vendor rage forces me to choose them again. :-/
duxup 6y ago
I'd really like to one day find out how this decision was made at Western Digital. My understanding is there wasn't much for WD to gain here and serious risks... Did they really feel this was worth it? How did this decision make its way through the management chain?
Teknoman117 6y ago
Maybe I need to try ZFS again now that I have different drives. I used to run it on 6x 3TB WD Red drives (raid-z2) and could never get ZFS to perform anywhere near what I expected. Ended up going with LVM on mdraid (raid 6) on dm-integrity and got tremendously better performance for my "home nas / vm image host" machine.
- nix23 6y ago
  Yes you should look at that numbers: https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eight-ironwolf-disks-two-filesystems-one-winner/ https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eigh...
- jhoechtl 6y ago
  Use Btrfs. Seriously. It's a tremendous fs. Read the caveats concerning raid56 and success.
  - Teknoman117 6y ago
    My stack at the moment is 6x [10 TB WD Red Pros -> dm-integrity] -> mdadm raid6 -> dm-crypt -> LVM. One of my LVM volumes is btrfs. I only use LVM because I've had terrible performance for VM images stored in btrfs volumes. You can always use fallocate and disable CoW for the image but that prevents taking snapshots. I'd if there was a zvol equivalent in btrfs.
    - cmurf 6y ago
      Neither fallocate nor nodatacow will prevent taking snapshots. The result of writing to nodatacow shared extents is that the write must be COW, not an overwrite. Same as a reflink copy of a file on XFS. New "overwrites" are COW, subsequent overwrites are in fact overwrites.
      - deleted 6y ago
        [deleted]
  - 6y ago
metalliqaz 6y ago
Why haven't the prices come down on these drives?
- executesorder66 6y ago
  Greed.
williesleg 6y ago
So tired of seeing this. The problem is you don't want to do ANY striping on ANY SMR drives. It's not a ZFS problem. It's not a SMR problem. It's STRIPING on SMR. PERIOD. Once upon a time people knew what they were talking about. Now everybody's an EXPERT NPC.
- effie 6y ago
  Could you explain why? Isn't the problem rather in the fact the filesystem does not know it is SMR and does not have a capability to work with such device optimally?
jhoechtl 6y ago
ZFS is like Bitcoin: Skyrocketing in popularity for no apparent reasons. Although I acknowledge that in this case it is WDs fault.
- fierarul 6y ago
  The reason is the growing data people save and experiencing bit rot. Why not use ZFS?
  - Dylan16807 6y ago
    Because it's absolutely awful at deduplicating. Or you're on windows. But sure yeah it's pretty good overall, I don't disagree with your main point.
    - bromonkey 6y ago
      Dedup is a pretty niche use case for most people.
    - modderation 6y ago
      ZFS on Windows is slowly getting better. I wouldn't actually trust it with data yet, but I suspect the developers would welcome any contributions or effort sent in their direction: https://github.com/openzfsonwindows/ZFSin https://github.com/openzfsonwindows/ZFSin (I'm not affiliated, but I think it's a cool project, and I really want it to succeed)
    - fierarul 6y ago
      I don't think deduplication can do much of a job on media files which are heavily compressed anyhow. As for Windows, that's a new one! When I think ZFS I think NAS although I did briefly had it on a laptop with FreeBSD I think.
StillBored 6y ago
Ok, so everyone has decided that SMR+ZFS is bad. Given that all traditional RAID arrays do online rebuild, what keeps a random application, lets say a database, from creating a write pattern during the rebuild which forces timeouts? AFAIK, nothing. And logically, it might be possible with just a single drive without the RAID given that WD hasn't published proper command timeout definitions for these drives either. So people are giving them a pass for finally admitting that the drives are SMR. That does nothing if they won't publish topology information, or buffer capacity so that the host can optimize write patterns. So, until the opensource community can adjust the worse case command timeouts, and preferably optimize write patterns based on drive topology, the hardware doesn't appear fit for any purpose.
- himinlomax 6y ago
  Putting a database on a spinning disk in 2020 is just completely idiotic.
  - zepearl 6y ago
    I don't think so - the decision which media to use is linked to a lot of factors (type of database, amount&size of data written/read, type of filesystem, costs, durability, expectations, etc...).
    - himinlomax 6y ago
      You don't think so, I know so. There are a few edge cases where spinning disks could be acceptable, but none where they are superior anymore. Cost? Maybe if you're not billed for electricity. And the cost of hardware is beer money compared to admin/dev/architect time. Durability? Drives of both technology vary in durability based on market class. Enterprise disks are superior to cheap consumer SSDs in that respect, but within the same class SSDs are superior. Type of database? Well if your database is used mostly for sequential reads, in other words when it's for most intents and purposes used like a flat file would be, you're in that edge case. Few individual databases are like that, though admittedly those that are can be very large. Data written/read? Early SSDs were admittedly prone to wearing off with heavy writes. We're long past that, except if you're talking about dodgy SD cards but you're not putting a database on that, are you? Types of filesystems? Just no. Hard drives are cheaper per byte. Significantly cheaper, but not even an order of magnitude cheaper anymore. They use more power, their random access time is orderS of magnitude slower, and their sequential transfer rate is slower. Hell even sequential write is not better any more. But I'll concede this, if you look long and hard enough, I'm sure you can find that one odd case where a database is marginally better served by spinning disks.
danShumway 6y ago
Open question: Speaking as someone who accidentally bought SMR drives from Seagate that weren't advertised as such, what's the primary danger of using these drives in a _new_ ZFS setup? From what I've seen, the write speed is bad enough that rebuilds might be problematic in ZFS. But since I'm not doing a rebuild, and since I can swap out the drive for a non-SMR alternative whenever one fails and I do need to do a rebuild, are they still relatively safe to use?
- magicalhippo 6y ago
  Depends on workload of course, but my buddy has used 8x Seagate Archive drives (which were the first SMRs) in a ZFS-based media/file server for years without issue. If any new disk you replace it with is non-SMR, then you shoud be good as far as I can see.
- seabrookmx 6y ago
  My anecdotal evidence, based on a fairly small (16TB) pool for personal use such as backups and media, is that the _Seagate_ SMR drives are fine. I've done multiple resilvers and they're definitely slower than conventional drives at that task, but the main issue people are seeing with WD drives is that in certain cases the drives take multiple seconds to respond to an ATA command, making the system think the drive is defective. The Seagate drives don't seem to suffer from this problem. There's lots of documentation out there on people using Seagate "Archive" drives in NAS's as well.. these were some of the earliest SMR drives and were on the market before the tech became more mainstream. Their results seem to align with what I'm seeing even though mine are branded "Barracuda Compute." So while I wouldn't be using _any_ SMR drives for a production workload, I'm perfectly happy with the 8TB Seagate SMR drives I picked up for $185 CAD each for my home use.
  - 65a 6y ago
    I ran a large array of 5TB Seagate SMR drives. Everything was fine until a disk failed, at which point the resilver took almost 24h. This was an uncomfortable window, but otherwise the array was fine, although write speeds were about 15-40MB/s in many cases.
Bud 6y ago
How can one tell for sure if a given WD Red drive is affected by this issue? I have some WD Red drives that are a couple years old, in my NAS.
- e40 6y ago
  Anything less than 8TB is SMR, that's what I read in various places. All of my 3TB ones where. 2 failed in a RAID 6 in the same week.
  - milofeynman 6y ago
    Not accurate. Some older reds are CMR. See the comment above. It depends on what model and how long ago you bought them.
- siera 6y ago
  WD has published a list of SMR drives : https://blog.westerndigital.com/wp-content/uploads/2020/04/2020_04_22_WD_SMR_SKUs_1Slide.pdf https://blog.westerndigital.com/wp-content/uploads/2020/04/2...
- berkut 6y ago
  They have different model numbers. i.e. the original 4 TB CMR drives were: WD40EFRX the new 4 TB SMR drives are: WD40EFAX They also have larger caches in most cases (i.e. 256 MB vs 64 MB).
  - at_a_remove 6y ago
    I checked and lucked out, I had WD4000FYYZs. Thank you!
aidenn0 6y ago
While WD's mismanagement of this is incredible, saying "WD knew about ZFS Issues" because one employee of WD said DM SMR was inappropriate for ZFS at a conference a few years ago overstates things. Even at relatively small companies, not all knowledge disseminates equally through the company, and WD isn't exactly a small company. It's almost certain that the team that was responsible for the WD Red SMR release was not aware of this issue or this wouldn't have happened. What probably happened was something more along the lines of: 1. Traditional wisdom says SMR can't be used in raid 2. DM-SMR firmware has gotten much better at hiding the performance issues of SMR since traditional wisdom 3. Let's actually measure this in some RAID setups and see what happens 4. Hmm, those numbers look reasonable, we can ship this! At no point did anyone think very hard about ZFS being an edge-case for DM-SMR. All that being said, the drives absolutely should have been represented as DM-SMR, along with some nice benchmark numbers to show that the performance hit isn't as bad as one might think. In the right RAID setup, these drives are at a reasonable performance/value point in the curve.
- syshum 6y ago
  It is not just ZFS edge cases seeing issues, ZFS was just the bell weather. Synology does not use ZFS at all, and they are seeing issues Also ZFS being an "edge case" is not going to be true long term as alot of SMB and Home NAS vendors (the very customers the WD Red Line targets) are starting to use ZFS on Linux more and more
  - boomboomsubban 6y ago
    I haven't seen the data for a few years, but I believe random Synology and other premade NAS dominated the market. To the point where ZFS would always be an edgecase.
    - syshum 6y ago
      Synology and QNap are probably the 2 biggest in this market segment, Synology already has issues and QNAP is bringing ZFS to market in their latest line of NAS devices Synology also has used btrfs in the past which I would suspect may suffer the same kind of issue as ZFS but dont know forsure
boomboomsubban 6y ago
Let me start by saying I also find WD's actions ridiculous and misleading. Has WD ever said Reds would work with ZFS? The Red spec sheet says performance isn't guaranteed on setups not listed as compatible. The compatibility list is a nightmare [1] to use, but I can't find ZFS and only Gold drives are listed for FreeBSD. None of the roughly ten ransom products I checked listed Red. https://www.westerndigital.com/support/partner-product-compatibility https://www.westerndigital.com/support/partner-product-compa... https://www.westerndigital.com/support/partner-product-compatibility https://www.westerndigital.com/support/partner-product-compa...
stock_toaster 6y ago
WD lost all their goodwill with me as a customer because of this. Shocking how fast a brand can go from trusted to distrusted once they are shown to be untrustworthy. ;)
- thijsvandien 6y ago
  Well, what brand does one buy today? It's not like there was that much choice anymore to begin with. Others had massive reliability issues (too). I don't consider Seagate much of an alternative, for example.
  - yellowapple 6y ago
    Apparently Toshiba's the way to go. I can't vouch for 'em (I haven't bought a Toshiba hard drive... ever; all of the ones I own are the ones that came in the various old laptops and desktops I've got in my garage), and I usually stick to SSDs nowadays, but next time I'm in the market for an HDD I guess that's the way I'll end up going (with WD and Seagate both relegated to secondaries, e.g. for RAID diversification).
pubutil 6y ago
I just ordered some WD Reds and a Synology box a couple of days ago, not knowing about this issue. Should I cancel the HDDs and go for another brand, or will I be okay if I don’t use ZFS?
- CrazyStat 6y ago
  Not all WD Reds are affected, only certain ones. Check first to see whether the ones you ordered are SMR or not.
- deleted 6y ago
  [deleted]
- thefz 6y ago
  IIRC EFAX are affected while EFRX are not. https://www.hwupgrade.it/immagini/wd-hdd-smr-25-04-2020.jpg https://www.hwupgrade.it/immagini/wd-hdd-smr-25-04-2020.jpg
charliebrownau 6y ago
So in the end , they lied, tired to be cheap, caught doing it Yet consumers dont get a refund/replaced product They are not forced to correctly label SMR on drives No one fired or arrested in WD and no ones offered other drive brands/makers as an replacement Maybe RAID/ZFS needs to be limited to SAS only and not retail drives at this stage

309 points by arm 6y ago

I’m very happy that this tactic has backfired so spectacularly for WD. I don’t know what the total impact on their bottom line is, but what I’m seeing is a huge reputation as damage for WD. People are avoiding putting WD in their NAS in general, even if they have plenty of drives that are fine with it!

tjoff 6y ago
I have such a hard time to understand how the risk vs. reward ratio for pulling this stunt made any sense. SMR does not have a great reputation, to put it in drives specifically targeted for NASes and enthusiasts is a very bold move.
- kllrnohj 6y ago
  Given the price premium on these drives in the first place I don't even get why they'd do this? Stick the SMR drives in the EasyStore line or whatever, which is regularly half the $/gb of the bare RED drives. That'd probably also discourage some shucking.
  - smacktoward 6y ago
    In every company, no matter how obscene the margins they're making on a given product, there is someone arguing that those margins could be even larger if they just cut a few corners. Eventually someone listens to that person, and decades of painstakingly built-up brand value gets thrown out the window in order to bring home six nickels tomorrow instead of five.
    - deleted 6y ago
      [deleted]
  - cbsmith 6y ago
    As mentioned in the Manfred Berger talk, the tech is supposed to, in theory, get you a 25-50% cost effectiveness advantage. Obviously the market prices don't reflect that, but that explains why they'd do this.
    - loeg 6y ago
      Reality seems to be more like 10% on a capacity basis. On a performance basis, of course, it's extremely negative.
      - cbsmith 6y ago
        Magnetic disks are generally valued on capacity, but at least as I heard the talk, he was talking about it growing to a 50% difference as they pushed the limits. I get that the reds are more performance oriented, but capacity is still a key factor (if not, you just go SSD and be done with it).
        loeg 6y ago
        I get that 50% has been the sort of marketing aim for "why bother support SMR at all," but until they actually get there, it's just marketing. Reality is ~10%. Magnetic disks are valued on capacity only because people considered CMR HDDs a performance floor. When they realize how bad SMR is, I think performance becomes a factor. If you're talking about GB per volume, SSDs beat HDDs of all stripes there: the biggest HDDs are 16 TB in a 3.5" form factor, while the biggest SSDs are 100 TB in the same form factor. 10 or 50% improvement in HDDs wouldn't begin to bridge that gap. SMR should be understood to be sort of a fast Tape, rather than slow CMR HDD.
        Dylan16807 6y ago
        > SMR should be understood to be sort of a fast Tape, rather than slow CMR HDD. I wouldn't go that far. If I can assume that the firmware won't cause a lot of problems all by itself, I think in my desktop I'd slightly prefer a 7200RPM SMR drive to an equal-price 5400RPM CMR drive. And no matter what kind of hard drive(s) you're using, take a $20 SSD and use half of it as a cache with writeback enabled.
  - kasabali 6y ago
    They do. Corresponding 2-6TB Blue drives are also SMR and that are shipped inside external drives. Only Reds got the attention because of ZFS issues. I suspect Blue and Red drive have a lot of common in manufacturing they switched all drives in same capacity together.
    - andoriyu 6y ago
      Red is getting attention not just because of ZFS, but because Reds are targeted and marketed for workloads that don't work well with SMR. It was well known before that Red are trash compared to Red Pro which IIRC are rebranded HGST DeskStar NAS.
- rwmj 6y ago
  > SMR does not have a great reputation SMR is fine if you know what SMR is and what you're doing. It might even be fine given a sufficiently advanced translation layer (which apparently doesn't exist yet). The problem isn't SMR, it's selling a product which uses SMR and hiding that fact from the consumer, and I can't forgive WD for that.
  - yjftsjthsd-h 6y ago
    > It might even be fine given a sufficiently advanced translation layer (which apparently doesn't exist yet). Apparently f2fs does well with it?
    - simcop2387 6y ago
      Not terribly surprising, f2fs and similar filesystems are log-based systems. Rather than overwrite the old data they almost always try to write to the next new area. This ends up working really well for flash technology because writing to existing areas is expensive. You have to read in a 1-4MB area depending on the block size, make your changes to that, erase that area entirely, and then write the changed version. [Those sizes might be a little off but it's what I remember]. Some of that may be hidden by the disk's controller but it has to happen on every write to an area with existing data. SMR is similar in this respect because you have to read, change, and write large blocks together because of the way the tech works. So I'd expect anything that's directly optimized for low level flash memory to work similarly ok performance wise for it. The real problem comes when you can't work like that, in particular this happens when you're low on free space or if you're in a situation where relocating data isn't possible or known that it needs to happen. RAID setups are typically the worst case for this, since they'll effectively treat the entire disk as used and can't relocate data on it. ZFS is a bit more complicated with this but effectively ends up the same.
  - deleted 6y ago
    [deleted]
  - m4rtink 6y ago
    There have been some experiment with modified XFS using SMR drives natively: https://www.snia.org/sites/default/files/SDC15_presentations/smr/HannesReinecke_Strategies_for_running_unmodified_FS_SMR.pdf https://www.snia.org/sites/default/files/SDC15_presentations... You put the journal and metadata on the continuous part and data on the shingled part, making sure you only ever add to what was written previously in a shingled segment by using copy-on-write. And sporadically you reclaim a segment. This should be much more efficient than a block layer translation layer that knows nothing about the filesystem.
- willcipriano 6y ago
  >I have such a hard time to understand how the risk vs. reward ratio for pulling this stunt made any sense. Classical economics likes to think of firms as rational entities working to maximize profit over the long term. In my experience that makes much of what companies do inexplicable. Instead I view firms as large groups of individuals all acting in their own self interest, this seems more complex but really all you have to know is what the incentive structure is to understand the choices a company makes. Similarly I've found you can also do this in reverse. It is a virtual certainty that from a compensation perspective moving to SMR was better for the individuals who made that choice.
  - TheSpiceIsLife 6y ago
    Fortunately, economic theory is more broad than classic economics. Managerial and behaviour theories here: https://en.wikipedia.org/wiki/Theory_of_the_firm https://en.wikipedia.org/wiki/Theory_of_the_firm Linked to from the Economics article under the Theory section: https://en.wikipedia.org/wiki/Economics https://en.wikipedia.org/wiki/Economics The theory your suggesting here has been known since at least the late 50s. You might also be aware of, or find interesting anyway, the principal–agent problem: https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble... Edit: typo
    - jacques_chester 6y ago
      I don't think this reply should be downvoted. It might be slightly spicy, but it does reflect that economics in the 20th century made enormous strides in understanding the observable behaviour of actual firms and markets. Transaction costs, agent problems, information economics, institutional economics, mechanism design and so on are powerful insights into the world around us. I've learned more about how software grows and spreads from economists than from software engineers.
      - Dylan16807 6y ago
        It's a useful bit of information, but it's insulting the comment it replies to without actually disagreeing at all. There's a reason the initial comment said "classical" economics! I can see why it earned some downvotes.
- tzs 6y ago
  Ars Technica did some detailed testing of these drives a couple weeks ago. See "We put Western Digital’s dreaded SMR Red drive to the test" [1]. They found that they won't work for ZFS but are actually OK in most NASes. Here is their conclusion: -------- begin quote We want to be very clear: we agree with Seagate's Greg Belloni, who stated on the company's behalf that they "do not recommend SMR for NAS applications." At absolute best, SMR disks underperform significantly in comparison to CMR disks; at their worst, they can fall flat on their face so badly that they may be mistakenly detected as failed hardware. With that said, we can see why Western Digital believed, after what we assume was a considerable amount of laboratory testing, that their disks would be "OK" for typical NAS usage. Although obviously slower than their Ironwolf competitors, they performed adequately both for conventional RAID rebuilds and for typical day-to-day NAS file-sharing workloads. We were genuinely impressed with how well the firmware adapted itself to most workloads—this is a clear example of RFC 1925 2.(3) in action, but the thrust does appear sufficient to the purpose. Unfortunately, it would appear that Western Digital did not test ZFS, which a substantial minority of their customer base depends upon. These tests may not be great news for either the American or Canadian class-action lawsuits currently underway against Western Digital, but they aren't the end of the line for those lawsuits, either. Even in the best case, the SMR models of WD Red underperform their earlier, non-SMR counterparts substantially—and consumers were not given clear notice of the downgrade. If the same firmware was being used to make substantially larger drives available to consumers than would otherwise be possible, and the limitations of those drives were adequately explained, we would probably be gushing over its utility and function. Unfortunately, Western Digital has so far only chosen to use it to cut manufacturing costs on small disks, without even passing the savings along to the consumer. -------- end quote That last paragraph may be key. It sounds like if these drives had been marketed a little different and priced a little different they could have been a good deal for many NAS users. I wonder what the chances are that they were originally intended for just that, and something got mixed up or miscommunication between design and release? [1] https://arstechnica.com/gadgets/2020/06/western-digitals-smr-disks-arent-great-but-theyre-not-garbage/ https://arstechnica.com/gadgets/2020/06/western-digitals-smr...
  - vetinari 6y ago
    I did a reshape, in phase 1 from 4-drive RAID5 to 5-drive RAID6 and then, in phase 2, from 5-drive RAID6 to 8-drive RAID6. Everything with Synology NAS, with typical Synology setup (mdraid, btrfs on top). The new 4 drives were the infamous 6TB EFAX. The phase 1 took 12 days, the phase 2 took 3 days. So my personal experience is radically different than Ars Technica's.
- PaulHoule 6y ago
  SMR is not as bad for desktop-style workloads as you might think. If you have enough storage for cache and metadata, you can get great performance because all of your writes are sequentialized. If you had 32GB of Optane and 4 TB of SMR properly tuned that would be one heck of a desktop drive, but (1) you have to really eliminate all the bottlenecks and (2) who needs a desktop drive large enough that SMR is worth it?
  - StillBored 6y ago
    The problem is that once the buffers are full the drive effectively stops accepting commands while it rewrites a SMR region. When that happens depending on what your doing, the drive might just get kicked from an array, or your whole system might freeze if some critical page needs to be read back in and it can't get sufficient priority and a command slot. I guess the vast number of people won't ever write more than a buffers full of data, or the drive will never get fragmented sufficiently that even small write operations amplify into entire drive rewrites.....
    - londons_explore 6y ago
      Smarter firmware would detect the case you are discussing, and re-designate a part of the SMR drive to be regular(non-SMR space). For that to work, you need the host OS to be able to see that the drive has just "shrunk". Obviously, you still have data on it while it's shrinking, so the reality is the OS needs to give the drive a list of sectors which can be "shrunk away". One simple approach would be for a daemon to create a massive file filled with "MAGICBYTESMAGICBYTESMAGICBYTES...". As that data is 'written' to the drive, the drive sees that it's the magic bytes, and rather than storing the data, simply marks those sectors as no longer needed. As soon as enough sectors in a row have that designation, reformat them as non-smr, and use as regular (non-smr) disk space. Then, a few hours or days later, the drive can rewrite the data back to be SMR, then tell the daemon, which can remove the magic bytes and delete the files, and your free disk space increases again. The failure mode of this is that your "6TB" SMR disk ends up with a 3TB file filled with MAGICBYTES, and 3TB of your data. But for the same money, you'd only get to store 3TB of data on a non-SMR drive anyway...
NortySpock 6y ago
Western Digital-branded drives not showing on BackBlaze hard drive reliability tables probably left WD in a blindspot for people looking for reliable drives anyways. https://www.backblaze.com/blog/hard-drive-stats-q2-2019/ https://www.backblaze.com/blog/hard-drive-stats-q2-2019/ https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1-2020/ https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1...
- ksec 6y ago
  HGST is under Western Digital.
  - syshum 6y ago
    HGST brand was killed, UltraStar is now WD Branded not HGST
- arminiusreturns 6y ago
  I've been reading the backblaze report since it started coming out (thanks to the backblaze team as always for being so transparent!). It become obvious to me as far back as 2015 that WD was a brand to avoid... and it seems by the end of 2019 backblaze had phased them almost completely out... for a reason.
sleepydog 6y ago
The hard drive market has shrunk considerably in the past ~5 years. What other choices are there? I remember Seagate having a not-so-great reputation in the late 00s, but now they seem like the most obvious alternative. Are we going to be stuck oscillating between ~4 vendors?
- tomatocracy 6y ago
  Toshiba are a reasonable bet now. I have several of the N300 drives which have performed pretty well so far.
- occamrazor 6y ago
  There are actually only three producers left: WD, Seagate and Toshiba.
  - kevin_thibedeau 6y ago
    Toshiba is the only one without a tarnished reputation.
    - yellowapple 6y ago
      And Backblaze seemed to think highly of 'em as of 2015: https://www.backblaze.com/blog/a-look-at-backblazes-toshiba-hard-drives/ https://www.backblaze.com/blog/a-look-at-backblazes-toshiba-... It's interesting that Toshiba apparently acquired Hitachi's production assets during the HGST→WD acquisition. HGST had a pretty good reputation at the time, so I feel like this bodes well for Hitachi if their drives are indeed a continuation of that. If I do need some spinning rust at some point I guess I'll have to give 'em a whirl.
    - jclulow 6y ago
      Have had some serious and deeply negative experiences with mixed workload performance on Toshiba disks in the last five years. Allegedly enterprise-grade 2.5" spindles, becoming deeply confused when handed both reads and writes concurrently. Test with your workload, of course, but my default position is: beware!
MintelIE 6y ago
Personally I won't buy another WD or Sandisk product ever again. Companies like this don't ever really come off as being sorry for the scams they pull, so it's best to just cut them off completely. I de-Googled too because of their shady practices. EDIT: For those who can't use the Internet to look things up, Sandisk and G-Technology and some random thing called Upthere are all WD brands.
- vl 6y ago
  What’s wrong with Sandisk?
- mey 6y ago
  What has Sandisk been up to?
  - tw04 6y ago
    WD owns Sandisk. https://www.theverge.com/2016/5/12/11662018/western-digital-sandisk-deal-complete https://www.theverge.com/2016/5/12/11662018/western-digital-...
washadjeffmad 6y ago
A little too much outrage for my taste. Every major HDD manufacturer was found offering SMR. WD was just the first to be held to task because they're so incredibly popular with shuckers. Seagate SMRs were found before WDs because we only purchased ≥8TB EasyStores, which are unaffected. For the lines they were supplied under, SMR characteristics don't impede 95% of customers. The rest are small-timers trying to avoid paying for warranties. No one buying Enterprise NAS was affected. edit: I guess we all decided to disagree by downvote today. Words tomorrow, maybe?
- jschwartzi 6y ago
  Actually I think you’re missing the point entirely. I have a 4 bay NAS that I use to back up all of my and my fiancé’s media so losing a phone isn't a catastrophe. I’m trusting that the NAS will actually rebuild in the event of a drive failure or it provides very little actual protection. WD shipped drives which were very unlikely to rebuild successfully. The upshot of that is that they lied to me and as a consequence have placed priceless data at risk from a drive failure. And had I known that I wouldn’t have bought their drives, so I’m materially harmed by it. I don’t shuck drives, I buy through retail channels like Newegg. I think you’re getting downvoted because you don’t understand the gravity of the problem and your victim blaming. Otherwise there’s nothing to disagree with because what you’re saying isn’t really supported in fact.
  - washadjeffmad 6y ago
    Thanks for helping me understand the demographic affected. This isn't a rebuttal, but I hope you don't lose anything, so you should know that a NAS is not a backup. They provide redundancy and availability, but no number or type of disks will save your data from being lost in a fire or flood if you keep it all in one place. SMR drives, if you know what you're buying them for, are perfect for backups and write few, read many applications, which it kind of sounds like you want. They're not inferior or at premature risk of failure, just specialized. Also, I'm not saying you have no right to be unhappy with finding out why, but were you really dissatisfied with the performance of your NAS before these hyperbolic exposes went out?
- 6y ago
m463 6y ago
If only it had backfired for everyone when the marketing folks changed from hexadecimal capacity sizes to decimal. 1000000 = 1M
- nayuki 6y ago
  1,000,000 is exactly equal to 1 mega. It's the other part of the computer industry (Windows Explorer, RAM sizes) that abused the notation to pretend 1 kilo = 1024. See: https://en.wikipedia.org/wiki/Binary_prefix https://en.wikipedia.org/wiki/Binary_prefix
- mjevans 6y ago
  My argument has been: Can I store just ONE byte at a time? Can I expand by singular bytes in a normal system? For hard drives, and even flash memory, the physical size of a sector converged, rather naturally, to a multiple of system word size, and since that had also converged to a power of 2, to a larger power of 2. Thus hard drives have sectors that were 512 bytes (4096 or '4Kbits'), and also why the smallest write available for modern drives is a multiple of 2. Binary is the native nomenclature of computers. It only makes sense that related equipment should also use the native (binary) engineering unit approximations as a result. Though for communications equipment, they do have variable sized packets, which even though computers generate 8 bit 'octet' based signals might technically be used by some other format that doesn't; and talking about the actual speed on the wire conveying that in maximum possible raw bit states is a tiny bit confusing when the rest are binary units, but it's less strongly wrong than a system where the lowest addressable (real) unit is binary based.
  - nayuki 6y ago
    You're basically saying that prefixes like kilo- have a context-dependent meaning - usually 1000, but sometimes 1024. We know how well that worked out historically, right? e.g. https://en.wikipedia.org/wiki/Ounce https://en.wikipedia.org/wiki/Ounce "the international troy ounce is equal to exactly 31.1034768 grams", "The international avoirdupois ounce is defined as exactly 28.349523125 g", "A fluid ounce (abbreviated fl oz, fl. oz. or oz. fl.) is a unit of volume equal to about 28.4 ml in the imperial system or about 29.6 ml in the US system". By not respecting the definition that kilo = 1000, you are deliberately sowing confusion and undermining the metric system as a simple, consistent, universal measurement system.
yellowapple 6y ago
Yeah, that move turned me from being a more-or-less diehard WD fan (i.e. picking it as my first choice for spinning disks) to being really skeptical of it; if they're willing to cheap out on that, then for me it casts doubt on the reliability of the hardware, too (and reliability is literally my only concern, even for my own desktops and laptops; WD could sell drives at a tenth of the performance as, say, a Seagate, but if it's even a little bit more reliable - which, in my observation from a variety of different use cases, it historically has been - I'll pick it every time). The good news is that I've basically stopped buying spinning rust, so this is more of a "well bummer" type of mood than an "oh shit I gotta scramble for a better primary vendor" mood. Even for server/NAS use I'm more likely to go for a Crucial/Kingston mix nowadays than I am for a WD/Seagate mix, both because SSDs have matured enough to be more reliable for a lot of things (less moving parts = less mechanical wear/tear and less sensitivity to vibrations¹, and with RAID a dying flash module is less of a "game over" moment - though things have gotten a lot better on this front, too) and because they perform a lot better (both in terms of speed and in terms of thermal/energy efficiency). ---- ¹: https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
- CyberDildonics 6y ago
  For SSDs you can look at the individual components they are made from, like the controller boards.
- esyir 6y ago
  Unfortunately for bulk data storage, spinning rust is currently the only viable option at the consumer level. For now, until ssds reach competitive prices at the same scales, we're stuck with them.
  - yellowapple 6y ago
    For how I typically lean for storage (RAID1 "cells" with each disk from a separate manufacturer, combined into either a RAID10 or a JBOD), and thus how I typically buy drives, SSD v. HDD is actually surprisingly close price-wise, especially when looking at 2.5" SATA or SAS (which is what I typically buy, both because 3.5" SSDs strangely don't exist and because I can stuff more of 'em into an enclosure; plus, quite a few servers don't even take 3.5" drives). For example, consider two different 2TB cell setups: - SSD: Crucial MX500 + Samsung 860 QVO = $473.73 on Newegg - HDD: WD Blue WD20SPZX + Toshiba L200 = $168.57 on Newegg On the one hand, the SSD cell is a whole 2.81× more expensive. On the other hand, the SSD cell is only 2.81× more expensive. This narrows to 2× in the case of a 1TB (well, technically 960GB in the SSD case) cell setup: - SSD: Crucial MX500 + Kingston A400 = $236.54 on Newegg (this is my typical setup for a RAID1) - HDD: WD Blue WD10SPZX + Toshiba L200 = $117.13 on Newegg Now, to be fair, this is excluding Seagate, which is stupidly cheap and would widen that gap (and historically I've bought Seagate strictly as a secondary for these sorts of "cells", but given my bad experiences with Seagate I'm inclined to ignore them and go with WD+Toshiba if possible for a given target size). It's also excluding 3.5" drives (and therefore notably excluding WD Reds entirely), which doesn't help for a 1TB cell (Seagate Cheetah NS + WD Red = $118.75) or a 2TB cell (Seagate IronWolf + WD Red = $167.74), but does help for bigger cell sizes that wouldn't be possible with 2.5" at all without major sacrifices in vendor redundancy and price (for example, a 4TB cell w/ Toshiba N300 + WD Red = $225.91, v. Samsung 860 QVO v. WD Blue = $1049.17 - even higher if I wanted to keep Kingston in the mix - or two of the above 2TB cells for only slightly cheaper). On the other hand, this is including 5400 RPM drives; if I required 7200 RPM (as most NAS drives are) then the 1TB cell cost jumps up to $134.98 (WD Black + Seagate BarraCuda), and if I needed 10k RPM or SAS then I'd probably be springing for a WD XE WD9001BKHG + Toshiba AL13SEB900, which would bring the cell cost up even higher to $164.75 (while also losing 100GB of capacity), or even higher to $186.38 if I wanted to swap the WD for a Seagate Savvio ST9900705SS and have that 64MB of cache on both sides of the mirror (thus bringing the cost difference to a mere 1.27×). I'm also artificially keeping out a whole swath of SSD manufacturers in the above figures; never heard of "Team Group" or "Goldenfir", but if I had zero qualms about reliability I could put together a 1TB cell with those for $182.98 (on par w/ the 10k RPM cell), or for a 2TB cell I could go with a Crucial BX500 + Patriot P200 for $399.98. ---- That all being to say: SSDs are a lot cheaper nowadays than they were even a couple years ago, and at typical consumer scales they're plenty viable; "good enough" capacity, while also having better performance and lower power consumption.

AdmiralAsshat 6y ago

I imagine WD has said nothing so far because they're probably waiting for their lawyers to tell them exactly how much they can claim to have known/not known without opening themselves up to litigation.

tssva 6y ago
WD responded via a blog post when the use of DM-SMR for WD Red drives was first exposed. It basically consisted of an attempt to gaslight those that bought Red drives into believing that this is their fault for choosing drives inappropriate for the purpose they using them for. Who could think using Red NAS drives in a NAS would be an appropriate technology choice? /s
- ashtonkem 6y ago
  That blog post will go down as one of the all time worst examples of corporate crisis management.
  - jldugger 6y ago
    I feel like boing is at least in the running here.
- TheSpiceIsLife 6y ago
  I saw the headline at the time, but didn’t read the blog post because reasons. Does anyone have a link or archive link?
  - bonestamp2 6y ago
    I believe the April 20th (bottom) section of this blog post is the post in question: https://blog.westerndigital.com/wd-red-nas-drives/ https://blog.westerndigital.com/wd-red-nas-drives/ TLDR: They suggest different RED drives should be considered for different workloads. While that may have been their intention all along, that was not communicated well at in marketing or retail, where the drive colors (red, blue, purple, etc) used to be the appropriate method for selecting the right drive for your workload/purpose. The way they have worded it, the blog is not wrong, but they left out an important fact: consumers didn't have the necessary info to make the decision they're suggesting people should have made. Even if you read the data sheets that were available at the time, one would expect the listed transfer rate was a limit that the drive was capable of, not the rate that if exceeded would cause the device to perform very poorly. Worse, there would be no way to know that was the case because the fact that they were SMR technology was previously unreleased and the drives themselves actively hid that information from the host computer (since they were "device managed" SMR). The data sheets still aren't clear on how the performance is impacted with workload, but at least they're clear about the technology they use now.

S_A_P 6y ago

Interestingly these drives are on sale on newegg.com right now.

mark-r 6y ago
That's what happens when you have trouble selling something.
- Hamuko 6y ago
  Do they actually have trouble selling? How many people shopping for these kind of drives understand what kind of a difference SMR has theoretically and what kind of a difference it has for them? When you Google "best NAS HDDs", you'll basically run to a bunch of articles recommending WD Reds alongside Seagate Ironwolves. I think only one link on the first page of results had a warning against the lower capacity WD Reds due to this.
blibble 6y ago
that's because they know they're going to be worth essentially zero very soon
- metalliqaz 6y ago
  the SMR debacle has been going on for months now, the prices haven't taken much of a hit
loeg 6y ago
This is generally why shopping sales/discounts as a first-order criteria is not great for value. Find some acceptable parts first, then price hunt those parts.
kabdib 6y ago
Yup. I narrowly escaped buying one of these drives as a spare for a Synology. Anecdotal evidence is that the SMR drives will mostly work. It's not worth the time or trouble. WD is dead to me until the next cycle of vendor rage forces me to choose them again. :-/

duxup 6y ago

I'd really like to one day find out how this decision was made at Western Digital. My understanding is there wasn't much for WD to gain here and serious risks... Did they really feel this was worth it? How did this decision make its way through the management chain?

Teknoman117 6y ago

Maybe I need to try ZFS again now that I have different drives. I used to run it on 6x 3TB WD Red drives (raid-z2) and could never get ZFS to perform anywhere near what I expected. Ended up going with LVM on mdraid (raid 6) on dm-integrity and got tremendously better performance for my "home nas / vm image host" machine.

nix23 6y ago
Yes you should look at that numbers: https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eight-ironwolf-disks-two-filesystems-one-winner/ https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eigh...
jhoechtl 6y ago
Use Btrfs. Seriously. It's a tremendous fs. Read the caveats concerning raid56 and success.
- Teknoman117 6y ago
  My stack at the moment is 6x [10 TB WD Red Pros -> dm-integrity] -> mdadm raid6 -> dm-crypt -> LVM. One of my LVM volumes is btrfs. I only use LVM because I've had terrible performance for VM images stored in btrfs volumes. You can always use fallocate and disable CoW for the image but that prevents taking snapshots. I'd if there was a zvol equivalent in btrfs.
  - cmurf 6y ago
    Neither fallocate nor nodatacow will prevent taking snapshots. The result of writing to nodatacow shared extents is that the write must be COW, not an overwrite. Same as a reflink copy of a file on XFS. New "overwrites" are COW, subsequent overwrites are in fact overwrites.
    - deleted 6y ago
      [deleted]
- 6y ago

metalliqaz 6y ago

Why haven't the prices come down on these drives?

executesorder66 6y ago
Greed.

williesleg 6y ago

So tired of seeing this. The problem is you don't want to do ANY striping on ANY SMR drives. It's not a ZFS problem. It's not a SMR problem. It's STRIPING on SMR. PERIOD. Once upon a time people knew what they were talking about. Now everybody's an EXPERT NPC.

effie 6y ago
Could you explain why? Isn't the problem rather in the fact the filesystem does not know it is SMR and does not have a capability to work with such device optimally?

jhoechtl 6y ago

ZFS is like Bitcoin: Skyrocketing in popularity for no apparent reasons. Although I acknowledge that in this case it is WDs fault.

fierarul 6y ago
The reason is the growing data people save and experiencing bit rot. Why not use ZFS?
- Dylan16807 6y ago
  Because it's absolutely awful at deduplicating. Or you're on windows. But sure yeah it's pretty good overall, I don't disagree with your main point.
  - bromonkey 6y ago
    Dedup is a pretty niche use case for most people.
  - modderation 6y ago
    ZFS on Windows is slowly getting better. I wouldn't actually trust it with data yet, but I suspect the developers would welcome any contributions or effort sent in their direction: https://github.com/openzfsonwindows/ZFSin https://github.com/openzfsonwindows/ZFSin (I'm not affiliated, but I think it's a cool project, and I really want it to succeed)
  - fierarul 6y ago
    I don't think deduplication can do much of a job on media files which are heavily compressed anyhow. As for Windows, that's a new one! When I think ZFS I think NAS although I did briefly had it on a laptop with FreeBSD I think.

StillBored 6y ago

Ok, so everyone has decided that SMR+ZFS is bad. Given that all traditional RAID arrays do online rebuild, what keeps a random application, lets say a database, from creating a write pattern during the rebuild which forces timeouts? AFAIK, nothing. And logically, it might be possible with just a single drive without the RAID given that WD hasn't published proper command timeout definitions for these drives either. So people are giving them a pass for finally admitting that the drives are SMR. That does nothing if they won't publish topology information, or buffer capacity so that the host can optimize write patterns. So, until the opensource community can adjust the worse case command timeouts, and preferably optimize write patterns based on drive topology, the hardware doesn't appear fit for any purpose.

himinlomax 6y ago
Putting a database on a spinning disk in 2020 is just completely idiotic.
- zepearl 6y ago
  I don't think so - the decision which media to use is linked to a lot of factors (type of database, amount&size of data written/read, type of filesystem, costs, durability, expectations, etc...).
  - himinlomax 6y ago
    You don't think so, I know so. There are a few edge cases where spinning disks could be acceptable, but none where they are superior anymore. Cost? Maybe if you're not billed for electricity. And the cost of hardware is beer money compared to admin/dev/architect time. Durability? Drives of both technology vary in durability based on market class. Enterprise disks are superior to cheap consumer SSDs in that respect, but within the same class SSDs are superior. Type of database? Well if your database is used mostly for sequential reads, in other words when it's for most intents and purposes used like a flat file would be, you're in that edge case. Few individual databases are like that, though admittedly those that are can be very large. Data written/read? Early SSDs were admittedly prone to wearing off with heavy writes. We're long past that, except if you're talking about dodgy SD cards but you're not putting a database on that, are you? Types of filesystems? Just no. Hard drives are cheaper per byte. Significantly cheaper, but not even an order of magnitude cheaper anymore. They use more power, their random access time is orderS of magnitude slower, and their sequential transfer rate is slower. Hell even sequential write is not better any more. But I'll concede this, if you look long and hard enough, I'm sure you can find that one odd case where a database is marginally better served by spinning disks.

danShumway 6y ago

Open question: Speaking as someone who accidentally bought SMR drives from Seagate that weren't advertised as such, what's the primary danger of using these drives in a _new_ ZFS setup? From what I've seen, the write speed is bad enough that rebuilds might be problematic in ZFS. But since I'm not doing a rebuild, and since I can swap out the drive for a non-SMR alternative whenever one fails and I do need to do a rebuild, are they still relatively safe to use?

magicalhippo 6y ago
Depends on workload of course, but my buddy has used 8x Seagate Archive drives (which were the first SMRs) in a ZFS-based media/file server for years without issue. If any new disk you replace it with is non-SMR, then you shoud be good as far as I can see.
seabrookmx 6y ago
My anecdotal evidence, based on a fairly small (16TB) pool for personal use such as backups and media, is that the _Seagate_ SMR drives are fine. I've done multiple resilvers and they're definitely slower than conventional drives at that task, but the main issue people are seeing with WD drives is that in certain cases the drives take multiple seconds to respond to an ATA command, making the system think the drive is defective. The Seagate drives don't seem to suffer from this problem. There's lots of documentation out there on people using Seagate "Archive" drives in NAS's as well.. these were some of the earliest SMR drives and were on the market before the tech became more mainstream. Their results seem to align with what I'm seeing even though mine are branded "Barracuda Compute." So while I wouldn't be using _any_ SMR drives for a production workload, I'm perfectly happy with the 8TB Seagate SMR drives I picked up for $185 CAD each for my home use.
- 65a 6y ago
  I ran a large array of 5TB Seagate SMR drives. Everything was fine until a disk failed, at which point the resilver took almost 24h. This was an uncomfortable window, but otherwise the array was fine, although write speeds were about 15-40MB/s in many cases.

Bud 6y ago

How can one tell for sure if a given WD Red drive is affected by this issue? I have some WD Red drives that are a couple years old, in my NAS.

e40 6y ago
Anything less than 8TB is SMR, that's what I read in various places. All of my 3TB ones where. 2 failed in a RAID 6 in the same week.
- milofeynman 6y ago
  Not accurate. Some older reds are CMR. See the comment above. It depends on what model and how long ago you bought them.
siera 6y ago
WD has published a list of SMR drives : https://blog.westerndigital.com/wp-content/uploads/2020/04/2020_04_22_WD_SMR_SKUs_1Slide.pdf https://blog.westerndigital.com/wp-content/uploads/2020/04/2...
berkut 6y ago
They have different model numbers. i.e. the original 4 TB CMR drives were: WD40EFRX the new 4 TB SMR drives are: WD40EFAX They also have larger caches in most cases (i.e. 256 MB vs 64 MB).
- at_a_remove 6y ago
  I checked and lucked out, I had WD4000FYYZs. Thank you!

aidenn0 6y ago

While WD's mismanagement of this is incredible, saying "WD knew about ZFS Issues" because one employee of WD said DM SMR was inappropriate for ZFS at a conference a few years ago overstates things. Even at relatively small companies, not all knowledge disseminates equally through the company, and WD isn't exactly a small company. It's almost certain that the team that was responsible for the WD Red SMR release was not aware of this issue or this wouldn't have happened. What probably happened was something more along the lines of: 1. Traditional wisdom says SMR can't be used in raid 2. DM-SMR firmware has gotten much better at hiding the performance issues of SMR since traditional wisdom 3. Let's actually measure this in some RAID setups and see what happens 4. Hmm, those numbers look reasonable, we can ship this! At no point did anyone think very hard about ZFS being an edge-case for DM-SMR. All that being said, the drives absolutely should have been represented as DM-SMR, along with some nice benchmark numbers to show that the performance hit isn't as bad as one might think. In the right RAID setup, these drives are at a reasonable performance/value point in the curve.

syshum 6y ago
It is not just ZFS edge cases seeing issues, ZFS was just the bell weather. Synology does not use ZFS at all, and they are seeing issues Also ZFS being an "edge case" is not going to be true long term as alot of SMB and Home NAS vendors (the very customers the WD Red Line targets) are starting to use ZFS on Linux more and more
- boomboomsubban 6y ago
  I haven't seen the data for a few years, but I believe random Synology and other premade NAS dominated the market. To the point where ZFS would always be an edgecase.
  - syshum 6y ago
    Synology and QNap are probably the 2 biggest in this market segment, Synology already has issues and QNAP is bringing ZFS to market in their latest line of NAS devices Synology also has used btrfs in the past which I would suspect may suffer the same kind of issue as ZFS but dont know forsure

boomboomsubban 6y ago

Let me start by saying I also find WD's actions ridiculous and misleading. Has WD ever said Reds would work with ZFS? The Red spec sheet says performance isn't guaranteed on setups not listed as compatible. The compatibility list is a nightmare [1] to use, but I can't find ZFS and only Gold drives are listed for FreeBSD. None of the roughly ten ransom products I checked listed Red. https://www.westerndigital.com/support/partner-product-compatibility https://www.westerndigital.com/support/partner-product-compa... https://www.westerndigital.com/support/partner-product-compatibility https://www.westerndigital.com/support/partner-product-compa...

stock_toaster 6y ago

WD lost all their goodwill with me as a customer because of this. Shocking how fast a brand can go from trusted to distrusted once they are shown to be untrustworthy. ;)

thijsvandien 6y ago
Well, what brand does one buy today? It's not like there was that much choice anymore to begin with. Others had massive reliability issues (too). I don't consider Seagate much of an alternative, for example.
- yellowapple 6y ago
  Apparently Toshiba's the way to go. I can't vouch for 'em (I haven't bought a Toshiba hard drive... ever; all of the ones I own are the ones that came in the various old laptops and desktops I've got in my garage), and I usually stick to SSDs nowadays, but next time I'm in the market for an HDD I guess that's the way I'll end up going (with WD and Seagate both relegated to secondaries, e.g. for RAID diversification).

pubutil 6y ago

I just ordered some WD Reds and a Synology box a couple of days ago, not knowing about this issue. Should I cancel the HDDs and go for another brand, or will I be okay if I don’t use ZFS?

CrazyStat 6y ago
Not all WD Reds are affected, only certain ones. Check first to see whether the ones you ordered are SMR or not.
deleted 6y ago
[deleted]
thefz 6y ago
IIRC EFAX are affected while EFRX are not. https://www.hwupgrade.it/immagini/wd-hdd-smr-25-04-2020.jpg https://www.hwupgrade.it/immagini/wd-hdd-smr-25-04-2020.jpg

charliebrownau 6y ago

So in the end , they lied, tired to be cheap, caught doing it Yet consumers dont get a refund/replaced product They are not forced to correctly label SMR on drives No one fired or arrested in WD and no ones offered other drive brands/makers as an replacement Maybe RAID/ZFS needs to be limited to SAS only and not retail drives at this stage