7 ms·

So is the independent product review industry so broken that nobody spotted this in testing?

• tibbydudeza 7y ago
  It is like those massive PC Magazine laser printer reviews decades back ... nobody cares anymore even the cheapest crud is good enough. Reviews these days are rather all about GPU's and CPU's.
  • jl6 7y ago
    My guess is that a decent review of a hard drive takes a non-trivial amount of work: review unit acquisition, test setup, test execution, test repetition, analysis, writeup, publishing, ... And that work needs to be paid for. And nobody buys review magazines any more. And advertising revenue on review sites can only pay for a small handful of SKUs to be reviewed. So only the sexiest, mass-marketest units get reviewed. And the vendors fly under the review radar by flooding the market with so many variants that there isn’t enough funding to review more than a tiny fraction of them. And by the time any headway is made into reviewing current models, new ones are released and suddenly nobody is interested in the old ones, further squeezing the window of opportunity for advertising revenue. And eliminating entirely the incentive for long-term testing. Meanwhile, review sites realise they can get paid more in bribes/freebies/access than in advertising, and either outright compromise their reviews or only report the hits and never the misses. And so consumers are flying blind, trying to extract signal from the noise of furious churn.
    • pjc50 7y ago
      > Meanwhile, review sites realise they can get paid more in bribes/freebies/access than in advertising, and either outright compromise their reviews or only report the hits and never the misses. Pretty much. Like the rest of news, accuracy is expensive, entertainment is cheaper, and anyway the audience prefer the latter.
    • ourlordcaffeine 7y ago
      >And so consumers are flying blind, trying to extract signal from the noise of furious churn. I think this is true for almost any product nowadays that isn't in a ridiculously small niche. Most non-niche products on Amazon I've found get a poor score on "FakeSpot". I believe someone even admitted on reddit that he/she was actively creating fake reviews and fake stories about certain boot brands on relevant subreddits. Unfortunately you can't trust anything online anymore. You need to spend hours or days researching the technology or product category in order to make a judgement on the products available.
    • tinus_hn 7y ago
      Why would the test be difficult and not just automatic as you push a button?
      • wtallis 7y ago
        Putting together a decent automated test suite is possible, but is a significant project for an individual drive reviewer. And any test suite you create is only good for a few years at most before you have to start worrying about new technologies that change performance or power characteristics in a way your tests can't capture. If you were putting together a hard drive test suite a few years ago to use for reviewing consumer/prosumer desktop and NAS drives, testing specifically for SMR behavior may not have been on your radar. If you want to get really thorough, automation starts to be infeasible and unreliable: http://blog.stuffedcow.net/2019/09/hard-disk-geometry-microbenchmarking/ http://blog.stuffedcow.net/2019/09/hard-disk-geometry-microb...
    • thanksforfish 7y ago
      I feel like "spec sheet omitted serious detail" is a good signal to avoid a specific brand even if that specific SKU isn't available.
      • pbhjpbhj 7y ago
        Yes, I've been buying WD for what, 15-20 years (I'm just a domestic user, fwiw) .. these sorts of things signal "we've decided to burn the brand to make short term gains", like when venture capitalists take over.
• syshum 7y ago
  Is there even independent reviews any more?
  • layoutIfNeeded 7y ago
    There are, but it's a moving goalpost. Consumers are chasing yet unbiased reviewers, while companies' marketing departments are busy bribing the next generation of reviewers. At least in IT we have Stallman-esque zealots who refuse to be bribed. I don't think we have the equivalent in other markets unfortunately.
  • pbhjpbhj 7y ago
    "Which?" in the UK do pretty good consumer reviews for domestic goods IMO. Though in general they have a problem of always picking the most expensive products, usually they give enough info that you can get a substantially cheaper item with practically no loss.
• skinkestek 7y ago
  The only valid disk reviews I know about are those from Backblaze and they take some time to do ;-)
  • cm2187 7y ago
    But that's reliability, not performance. For performance I think the problem is that not every single capacity is reviewed. Typically the top capacity in each range.
    • zozbot234 7y ago
      Well, the OP mentions that these drives are dropping out of RAID arrays during rebuilding operations. Some people might describe that as a reliability concern.
      • cm2187 7y ago
        Right. Though I doubt backblaze is using RAID in the first place.
        • zdkl 7y ago
          As a lay person I don't understand why they wouldn't, could you give some more information as to why?
          • cm2187 7y ago
            For these large scale infrastructures, they typically use JBOD, i.e. no RAID whatsoever and they achieve redundancy by maintaining multiple copies of the data over multiple disks spread around the datacentre. So it's not hardware RAID that requires a certain response time (I suspect the drives mentioned here drop because they timeout as it take a long time to random write on an SMR disk), more like a distributed software RAID. I think they likely also have their own file system so they are not dependent on a fixed block size, which allows them to save space if they have loads of small files.
• toolslive 7y ago
  Yes, you will notice this, even if you do a simple write test with only big writes at full speed. After a while (about 2h in our tests) these drives become "tired". You need to leave them be for a few hours and then they are fine again. An added problem is that the usual candidate file systems (ext4, xfs,...) do not combine well with SMR. Some people might be able to put these drives to good use, but only if they know up front that it's this kind of drive they are getting.
  • read_if_gay_ 7y ago
    > After a while (about 2h in our tests) these drives become "tired". You need to leave them be for a few hours and then they are fine again. Why does that happen?
    • jsjohnst 7y ago
      > Why does that happen? Because with DM-SMR drives there is usually a non-shingled area on the drive used as a buffer. Once the buffer is full, performance tanks dramatically as tracks are being rewritten. This is especially true in my experience when writing lots of smallish files vs several very large files.
  • MisterTea 7y ago
    That's a good way of explaining it and makes sense. I have 3x4TB WD Reds configured as an 8TB RAID 5 mdadm array in my Linux box. It's a sort of dumping grounds for backups/disk images, VMm images, etc. I would see initial write speeds of 150-200MB/sec but during large transfers moving 100's of GB, it would drop in write speed down to 50-60MB/sec. I thought it was a performance issue in the kernel or with mdadm and didn't bother to investigate further because I'm not a datacenter.
  • jjgreen 7y ago
    I've observed exactly this behaviour on TB volumes on AWS instances, is it known if they use this sort of drive?
    • 7y ago
• Aissen 7y ago
  Yes. I remember two years back when a small (really) independent french journal called CPC Hardware was able to publish a full review of the latest AMD CPU a month before everyone else in a bimonthly paper issue: they simply sourced the chip on eBay instead of signing NDAs like literally every other publication. https://www.guru3d.com/news-story/cpc-hardware-amd-ryzen-7-2700x-print-review-is-online-incl-benchmarks,9.html https://www.guru3d.com/news-story/cpc-hardware-amd-ryzen-7-2...
  • icefo 7y ago
    CPC rocks ! I should renew my subscription, I let it lapse
• sbx320 7y ago
  Aside from the other mentioned issues, a significant part is also the lack of retesting. WD Red drives aren't a fancy new thing and have been on the market for years now. As a result a test of the same drives is unlikely to give you a whole lot of revenue, since it's a known old product. People don't usually care a lot about yet another test with the same results. As a result retesting isn't really worth your time _unless_ you find something significantly changed (like in this case). Also running a useful test is non-trival as you'd need to source a significant amount of platforms, comparision products and setup a testing environment for it. Testing a brand new (CPU|GPU|SSD) instead would be much more likely to attract new readers and increase revenue. Manufacturers have slowly started realizing this circumstance and are obviously able to exploit it by making products cheaper to manufacture down the line.
  • fierarul 7y ago
    > Manufacturers have slowly started realizing this circumstance and are obviously able to exploit it by making products cheaper to manufacture down the line. I believe this is a known technique in mass-produced markets. Called 'debasing' or something like that. Even new cars have more bells and whistles when first released while as the model gets a stead consumer base the manufacturer might reduce accessories or get cheaper versions, etc.
    • aspenmayer 7y ago
      There’s an analogue here in the concept of shrinkflation. To my mind debasing occurs in capital purchase items that last years or for life to take advantage of the only opportunity to double dip on the single sale. Shrinkflation seems more applicable to commodity goods that are often consumable or single use, the hidden price increase of which causes larger impacts to ROI and and profitability as well as cost to the buyer. These factors combined make shrinkflation perhaps more impactful overall to the economy due to the frequency and necessity of such goods. If you pay too much for too little on your car, oh well. Its probably too late to do anything about that now or you would have returned it to the seller already. For most people they settle for enough car for enough money. You can always add aftermarket parts or customize it, because there is a large secondary market waiting to make your outside investment even more worthwhile to your own judgement. If you pay too much for soap, it’s not even worth it to return it to the store in your time or your money so the market simply wins. It’s a rigged game akin to Las Vegas house games. As buyers it’s just stressful being so wary all the time. I don’t know what the better way is but surely it must exist, we just haven’t made it profitable enough yet for the right people. https://en.wikipedia.org/wiki/Shrinkflation https://en.wikipedia.org/wiki/Shrinkflation
• frankharv 7y ago
  I dont know of any independent review site besides BackBlaze. Most review sites receive product samples from the manufacturer. The linked article shows that the disk manufacturers cannot be trusted. I bet they did not send out DM-SMR drives to review sites.
• duxup 7y ago
  How many product reviews occur for these drives? Were these new enough that anyone bothered?