4 ms·

The key flaw in this analysis is that the author is only using consumer grade drives from Amazon. And yes, I think what is going on is that consumers don't rea

by tytso 9y ago

The key flaw in this analysis is that the author is only using consumer grade drives from Amazon. And yes, I think what is going on is that consumers don't really need much more than 4TB. Heck, most consumers have trouble filling 1 or 2TB. It's for the same reason that drive managed SMR drives haven't really been a big thing, in the consumer market. Who cares about 15-20% more bytes when you can't use all of the bytes in a 4TB CMR drive? Companies at the "hyper-scale cloud companies" (e.g., Facebook, Amazon, Microsoft, Google) can actually use the extra bytes, and they are probably the ones that will be able to use the 10TB, 12TB, 14TB, 16TB etc. drive that are on the vendors' roadmap[1]. [1] http://www.pcworld.com/article/3162084/storage/seagates-roadmap-includes-14tb-16tb-hard-drives-within-18-months.html http://www.pcworld.com/article/3162084/storage/seagates-road... It may be that Amazon might not be interested in stocking these drives because they won't sell enough of them to consumers. And if some of the discussions proposed in Google's "Disks for Data Centers"[2] paper come to fruition, such as adding a second disk actuator arm, or changing the height of the disk drives, there might be a further diversion between drives optimized for the consumer market and drives optimized for the "hyper-scale cloud market". (Note: the intent of Google publishing this paper is to hopefully allow discussions amongst the community of hyper-scale cloud storage customers to work with the HDD vendors for something that helps the whole high capacity storage industry.) [2] https://research.google.com/pubs/pub44830.html https://research.google.com/pubs/pub44830.html

soared 9y ago
I think that is the point, rather than the flaw, of this article.
gwern 9y ago
> The key flaw in this analysis is that the author is only using consumer grade drives from Amazon ? This is Backblaze, a commercial storage company. They buy in bulk in thousands of drives, not from Amazon, and the prices are what they paid. As they say, "The price you might get at Costco or BestBuy, or on Amazon will most likely be higher." It's the trend and comparisons which matter, not so much the absolute price.
- tytso 9y ago
  In the article the author asserted that BackBlaze was typically only using consumer drives. So that makes them very different from the companies in the "hyper-scale cloud storage" market. And if the trends the author is talking about consumer disk drives, I would argue that's a very incomplete picture. It doesn't include the traditional enterprise users, and it doesn't include hyper-scale storage companies that are using the bigger disks. Or else why else do you think that Seagate is putting 14T and larger drives on their roadmap: http://www.pcworld.com/article/3162084/storage/seagates-roadmap-includes-14tb-16tb-hard-drives-within-18-months.html http://www.pcworld.com/article/3162084/storage/seagates-road... This is very different from what the article was asserting, which was that disks weren't getting much bigger, and that they aren't getting cheaper per gigabyte. Even from a trend perspective, it's just not correct. (For example, it's not taking SMR disks into account; but that's because the consumers will probably not be interested in SMR drives.)
  - brianwski 9y ago
    > BackBlaze was typically only using consumer drives Disclaimer: I work at Backblaze. If anybody is curious, in our blog posts we list the exact serial numbers of all of the drives. See here for an example: https://www.backblaze.com/blog/wp-content/uploads/2017/05/blog-hd-stats-q1-2017-table.jpg https://www.backblaze.com/blog/wp-content/uploads/2017/05/bl... We have been at this for a while, and the distinction between "consumer drive" and "enterprise drive" is not what most people think it is. But in the end it simply doesn't matter -> the differences between failure rates of individual MODELS of drives in one line is larger than any difference in failure rates of "consumer" vs "enterprise".
    - tytso 9y ago
      It's not just "consumer drive" versus "enterprise drive". It's also "consumer drive" versus "hyper-scale cloud storage drive". Google used to use consumer drives (and the latter category didn't used to exist) but the world has moved on, and if you are looking for the lowest storage TCO (again, please see Google's "Disks for Data Centers" paper), I'm going to suggest it's a lot more than just using 4T drives. Hint: you're assuming the only cost is just the raw spindle. You also need to take into account the cost of power, cooling, some portion of the server that the disk is attached to, the cost of the tray, etc., etc., etc. This is all in the "Disks for Data Center" paper, so I'm not giving away any corporate secrets. Disclaimer: I work at Google and I am a co-author (but only in a very small way) of the "Disks for Data Centers" paper. We have also been at this for a while...

by tytso 9y ago

soared 9y ago

I think that is the point, rather than the flaw, of this article.

gwern 9y ago

> The key flaw in this analysis is that the author is only using consumer grade drives from Amazon ? This is Backblaze, a commercial storage company. They buy in bulk in thousands of drives, not from Amazon, and the prices are what they paid. As they say, "The price you might get at Costco or BestBuy, or on Amazon will most likely be higher." It's the trend and comparisons which matter, not so much the absolute price.

tytso 9y ago
In the article the author asserted that BackBlaze was typically only using consumer drives. So that makes them very different from the companies in the "hyper-scale cloud storage" market. And if the trends the author is talking about consumer disk drives, I would argue that's a very incomplete picture. It doesn't include the traditional enterprise users, and it doesn't include hyper-scale storage companies that are using the bigger disks. Or else why else do you think that Seagate is putting 14T and larger drives on their roadmap: http://www.pcworld.com/article/3162084/storage/seagates-roadmap-includes-14tb-16tb-hard-drives-within-18-months.html http://www.pcworld.com/article/3162084/storage/seagates-road... This is very different from what the article was asserting, which was that disks weren't getting much bigger, and that they aren't getting cheaper per gigabyte. Even from a trend perspective, it's just not correct. (For example, it's not taking SMR disks into account; but that's because the consumers will probably not be interested in SMR drives.)
- brianwski 9y ago
  > BackBlaze was typically only using consumer drives Disclaimer: I work at Backblaze. If anybody is curious, in our blog posts we list the exact serial numbers of all of the drives. See here for an example: https://www.backblaze.com/blog/wp-content/uploads/2017/05/blog-hd-stats-q1-2017-table.jpg https://www.backblaze.com/blog/wp-content/uploads/2017/05/bl... We have been at this for a while, and the distinction between "consumer drive" and "enterprise drive" is not what most people think it is. But in the end it simply doesn't matter -> the differences between failure rates of individual MODELS of drives in one line is larger than any difference in failure rates of "consumer" vs "enterprise".
  - tytso 9y ago
    It's not just "consumer drive" versus "enterprise drive". It's also "consumer drive" versus "hyper-scale cloud storage drive". Google used to use consumer drives (and the latter category didn't used to exist) but the world has moved on, and if you are looking for the lowest storage TCO (again, please see Google's "Disks for Data Centers" paper), I'm going to suggest it's a lot more than just using 4T drives. Hint: you're assuming the only cost is just the raw spindle. You also need to take into account the cost of power, cooling, some portion of the server that the disk is attached to, the cost of the tray, etc., etc., etc. This is all in the "Disks for Data Center" paper, so I'm not giving away any corporate secrets. Disclaimer: I work at Google and I am a co-author (but only in a very small way) of the "Disks for Data Centers" paper. We have also been at this for a while...