4 ms·

I look forward to these reports with the same zeal and enthusiasm as I used to reserve for comic strips in Sunday newspapers. This data is significant and appr

I look forward to these reports with the same zeal and enthusiasm as I used to reserve for comic strips in Sunday newspapers. This data is significant and appreciated. Thank-you so much, and please keep up the good work.

sixothree 6y ago
While I don't _quite_ share your enthusiasm. I sure do share your appreciation. Nobody else at their level appears to be sharing this data. This is not something they had any obligation to share with us. It certainly is useful and interesting.
- DizzyDoo 6y ago
  Every time I read through their update articles I'm reminded that Backblaze is a service I could possibly buy! I only buy a hard drive once every few years, but I enjoy seeing the data. The time it takes to write up and share what they are no doubt tracking anyway is probably quite worthwhile, I'd guess this is an effective piece of marketing. I'd be quite interested to see how many new sign-ups they get each time they post these stats.
  - brianwski 6y ago
    Disclaimer: I work at Backblaze. > I'd guess this is an effective piece of marketing. Yes, it really has been good to us. :-) It is data we would collect internally for our own decision making and tracking even if we didn't release it, and it is only a small amount of work for us (mainly for Andy the author of that blog post) to format it up every 3 months and write some observations down. Since it isn't what we "sell", it would have just gone to waste being hidden. And it results in our name getting "out there" and inevitably every quarter somebody asks "hey, what do these people do that requires 130,000 hard drives anyway?" and then we get a little bump in customers. Cost vs. benefit has been WELL worth it to us. Along the way (unrelated to making money) it makes us happy that people get some use out of it. Like we find out somebody does their PhD thesis on the raw SMART numbers of the drives or something, we enjoy hearing about it. Until my dying day I will never understand why Google Storage and Amazon S3 and Microsoft Azure don't release their drive failure statistics. I just don't get it. Those companies have SO MANY DRIVES and they employ people with PhDs in statistics, they MUST have the same info internally but at 100x the scale. I don't get why they don't release the numbers?! But hey, if they want to continue to give us this marketing gift of exclusivity, I'll take it. Maybe they like us and are just doing us a solid.
    - RyJones 6y ago
      A long time ago, WANOPS put a couple racks of machines in the parking lot of the (former) Building 11 on Microsoft Campus. We had a lower failure rate across the board, for all components. I think the only protection that was in place was some blue tarps and a chain link fence. I can't find a public reference, sorry.
    - fomine3 6y ago
      Great work! I wish you have Asia region DC.
    - garaetjjte 6y ago
      >they MUST have the same info internally but at 100x the scale. I don't get why they don't release the numbers Could be that on that scale they're dealing more directly with manufacturers, and they fear sharing failure data would affect it negatively?
- aogl 6y ago
  .. it is however, very good cheap marketing
icelancer 6y ago
I agree. I circulate it every quarter in my company's Slack channel to show the importance of publishing open data and reports, not just as a steward of technology, but because it is actually useful marketing.
hobojones 6y ago
I concur - I used to work with a set of clusters totaling ~45k drives and it was always fun to compare failure rates (especially on a per model basis).
- icefo 6y ago
  Did you end up with significally different values sometimes ?
  - hobojones 6y ago
    Our failure rates were consistently much higher compared to Backblaze, usually 25-30% higher. I never did any detailed analysis, but I'd chalk it up to a couple items. First, the stock of drives were not rotated out as frequently as I understand Backblaze does. The overall workload on the clusters was extremely heavy, but not consistent across all drives in the cluster, resulting in localized 'bursts' of activity. Finally the datacenter where the clusters lived used evaporate cooling and was located in a hot, dry environment surrounded by very fine soil. During the summer the machine room was Georgia level humid, and powering on any piece of hardware would usually yield a small cloud of dust.
    - brandmeyer 6y ago
      Turns out that some outdoor environmental dust is ferromagnetic. A particular vertical-axis wind turbine project was destroyed by buildup of magnetic dust on the generator magnets. I'm not saying that's what was killing your drives directly. Normally HDDs are fully sealed. But the amount of dust you mention is awfully suspicious. It might change your investigative perspective a bit when you consider that some small percentage of all that dust is ferromagnetic.
    - brianwski 6y ago
      Disclaimer: I work at Backblaze. > Our failure rates were consistently much higher compared to Backblaze That's interesting. > The overall workload on the clusters was extremely heavy That is the most likely explanation. We see higher failure rates when we are writing to the drives, for example as a new vault fills with customer data. Backup (our oldest business) has a fairly easy work load, it is not as punishing as a database that is pummeling the drives. In certain circumstances when we are down more than 1 drive in a 20 drive Reed Solomon group we stop putting new data on that drive group until the parity is restored explicitly because this lowers the chances of an additional drive failing in the group. That gives us more time to rebuild the parity with less stress in our lives. When that last parity drive fails and one more drive failure means customers lose data the fun drains right out of this job. Red alerts are thrown, pagers go off in the middle of the night, datacenter techs start driving towards the datacenter at 3am to replace drives. We prefer the world nice and calm and relaxed with a good night's sleep.
ComodoHacker 6y ago
Some argue this data has mainly historical value. By the time it's available vendors may have changed many aspects of respective models, from geometry to controller chip to motors, let alone firmware. You might end up bying a very different product under the same SKU, to which this data has no relation at all.
- schmookeeg 6y ago
  At this point, Backblaze is a serious "influencer", and HDD companies would be smart to make sure they get the best possible drives, even if the rest of us get junk. I mean "doing it well for an influencer and shafting everyone else" is far cheaper and easier than just making a solid and reliable product consistently. That's such old-fashioned thinking these days. :)
  - ValentineC 6y ago
    > At this point, Backblaze is a serious "influencer", and HDD companies would be smart to make sure they get the best possible drives, even if the rest of us get junk. I don't know if they're as budget-conscious as they used to be, but Backblaze used to "shuck" drives from externals because they're cheaper [1]. If this is still the case, then their stats should be similar to hobbyists who are doing this too. [1] https://www.backblaze.com/blog/backblaze_drive_farming/ https://www.backblaze.com/blog/backblaze_drive_farming/
    - giantrobot 6y ago
      I think part of their shucking issue was from the floods in Thailand in 2011 that put a huge constraint on OEM drives. There were plenty of external drives in the channel so BackBlaze ordered a ton and shucked them. While the shucked drives might not have had the warranties or even the specs of OEM drives they might have ordered the aggregate nature of their storage meant they were a good replacement.
      - brianwski 6y ago

sixothree 6y ago

While I don't _quite_ share your enthusiasm. I sure do share your appreciation. Nobody else at their level appears to be sharing this data. This is not something they had any obligation to share with us. It certainly is useful and interesting.

DizzyDoo 6y ago
Every time I read through their update articles I'm reminded that Backblaze is a service I could possibly buy! I only buy a hard drive once every few years, but I enjoy seeing the data. The time it takes to write up and share what they are no doubt tracking anyway is probably quite worthwhile, I'd guess this is an effective piece of marketing. I'd be quite interested to see how many new sign-ups they get each time they post these stats.
- brianwski 6y ago
  Disclaimer: I work at Backblaze. > I'd guess this is an effective piece of marketing. Yes, it really has been good to us. :-) It is data we would collect internally for our own decision making and tracking even if we didn't release it, and it is only a small amount of work for us (mainly for Andy the author of that blog post) to format it up every 3 months and write some observations down. Since it isn't what we "sell", it would have just gone to waste being hidden. And it results in our name getting "out there" and inevitably every quarter somebody asks "hey, what do these people do that requires 130,000 hard drives anyway?" and then we get a little bump in customers. Cost vs. benefit has been WELL worth it to us. Along the way (unrelated to making money) it makes us happy that people get some use out of it. Like we find out somebody does their PhD thesis on the raw SMART numbers of the drives or something, we enjoy hearing about it. Until my dying day I will never understand why Google Storage and Amazon S3 and Microsoft Azure don't release their drive failure statistics. I just don't get it. Those companies have SO MANY DRIVES and they employ people with PhDs in statistics, they MUST have the same info internally but at 100x the scale. I don't get why they don't release the numbers?! But hey, if they want to continue to give us this marketing gift of exclusivity, I'll take it. Maybe they like us and are just doing us a solid.
  - RyJones 6y ago
    A long time ago, WANOPS put a couple racks of machines in the parking lot of the (former) Building 11 on Microsoft Campus. We had a lower failure rate across the board, for all components. I think the only protection that was in place was some blue tarps and a chain link fence. I can't find a public reference, sorry.
  - fomine3 6y ago
    Great work! I wish you have Asia region DC.
  - garaetjjte 6y ago
    >they MUST have the same info internally but at 100x the scale. I don't get why they don't release the numbers Could be that on that scale they're dealing more directly with manufacturers, and they fear sharing failure data would affect it negatively?
aogl 6y ago
.. it is however, very good cheap marketing

icelancer 6y ago

I agree. I circulate it every quarter in my company's Slack channel to show the importance of publishing open data and reports, not just as a steward of technology, but because it is actually useful marketing.

hobojones 6y ago

I concur - I used to work with a set of clusters totaling ~45k drives and it was always fun to compare failure rates (especially on a per model basis).

icefo 6y ago
Did you end up with significally different values sometimes ?
- hobojones 6y ago
  Our failure rates were consistently much higher compared to Backblaze, usually 25-30% higher. I never did any detailed analysis, but I'd chalk it up to a couple items. First, the stock of drives were not rotated out as frequently as I understand Backblaze does. The overall workload on the clusters was extremely heavy, but not consistent across all drives in the cluster, resulting in localized 'bursts' of activity. Finally the datacenter where the clusters lived used evaporate cooling and was located in a hot, dry environment surrounded by very fine soil. During the summer the machine room was Georgia level humid, and powering on any piece of hardware would usually yield a small cloud of dust.
  - brandmeyer 6y ago
    Turns out that some outdoor environmental dust is ferromagnetic. A particular vertical-axis wind turbine project was destroyed by buildup of magnetic dust on the generator magnets. I'm not saying that's what was killing your drives directly. Normally HDDs are fully sealed. But the amount of dust you mention is awfully suspicious. It might change your investigative perspective a bit when you consider that some small percentage of all that dust is ferromagnetic.
  - brianwski 6y ago
    Disclaimer: I work at Backblaze. > Our failure rates were consistently much higher compared to Backblaze That's interesting. > The overall workload on the clusters was extremely heavy That is the most likely explanation. We see higher failure rates when we are writing to the drives, for example as a new vault fills with customer data. Backup (our oldest business) has a fairly easy work load, it is not as punishing as a database that is pummeling the drives. In certain circumstances when we are down more than 1 drive in a 20 drive Reed Solomon group we stop putting new data on that drive group until the parity is restored explicitly because this lowers the chances of an additional drive failing in the group. That gives us more time to rebuild the parity with less stress in our lives. When that last parity drive fails and one more drive failure means customers lose data the fun drains right out of this job. Red alerts are thrown, pagers go off in the middle of the night, datacenter techs start driving towards the datacenter at 3am to replace drives. We prefer the world nice and calm and relaxed with a good night's sleep.

ComodoHacker 6y ago

Some argue this data has mainly historical value. By the time it's available vendors may have changed many aspects of respective models, from geometry to controller chip to motors, let alone firmware. You might end up bying a very different product under the same SKU, to which this data has no relation at all.

schmookeeg 6y ago
At this point, Backblaze is a serious "influencer", and HDD companies would be smart to make sure they get the best possible drives, even if the rest of us get junk. I mean "doing it well for an influencer and shafting everyone else" is far cheaper and easier than just making a solid and reliable product consistently. That's such old-fashioned thinking these days. :)
- ValentineC 6y ago
  > At this point, Backblaze is a serious "influencer", and HDD companies would be smart to make sure they get the best possible drives, even if the rest of us get junk. I don't know if they're as budget-conscious as they used to be, but Backblaze used to "shuck" drives from externals because they're cheaper [1]. If this is still the case, then their stats should be similar to hobbyists who are doing this too. [1] https://www.backblaze.com/blog/backblaze_drive_farming/ https://www.backblaze.com/blog/backblaze_drive_farming/
  - giantrobot 6y ago
    I think part of their shucking issue was from the floods in Thailand in 2011 that put a huge constraint on OEM drives. There were plenty of external drives in the channel so BackBlaze ordered a ton and shucked them. While the shucked drives might not have had the warranties or even the specs of OEM drives they might have ordered the aggregate nature of their storage meant they were a good replacement.
    - brianwski 6y ago