9 ms·

Backblaze Drive Stats: 2018 Q3 Hard Drive Failure Rates

175 points by LaSombra 8y ago

hanyoloren 8y ago
cool
deleted 8y ago
[deleted]
Dennip 8y ago
Do BB publish pricing of what they paid for drives? Sort of a TB/$/Failure rate. Those 12TB HGST drives must cost a ton
- geostyx 8y ago
  No they don't.
  - jason46 8y ago
    $33,500 if you bought them from amazon right now, plus whatever it costs to ship 79 hard drives.
    - brianwski 8y ago
      Disclaimer: I work at Backblaze. > $33,500 if you bought them from amazon right now About half of all of the money that flows into Backblaze from customers gets spent on the datacenter, and the largest component of that is drives. I think sales tax and property tax is the second largest datacenter component. It turns out you don't just owe property tax on buildings and land - if you own something of significant worth like $10 million worth of hard drives in a large pile, you owe the government "property tax" every month for owning that property.
      - lotsofpulp 8y ago
        Yes, called "tangible personal property tax" (even though it's used for business). Some states have it and some don't, with various exceptions and whatnot in all of them.
      - PhantomGremlin 8y ago
        if you own something of significant worth like $10 million worth of hard drives in a large pile, you owe the government "property tax" every month for owning that property. If you're Intel, you potentially owe "property tax" every month on all the equipment in multi-billion dollar fabs. That's why those fabs only get built in states where the tax doesn't apply or where the state/county agrees to forego it.
      - dorfsmay 8y ago
        I had never thought of income on capital vs income before Piketty made it the center of his theories, but since then realised that it is exactly what property tax is, which is weird that it is only in one type of capital.
- atYevP 8y ago
  Yev from Backblaze here -> We don't typically post the cost of the drives, but one thing you can be sure of, is that we go for as good of a price-to-density ratio as we can! So if they were much more expensive than their counterparts we would likely avoid them until they came down enough for us to test them out!
- brianwski 8y ago
  Disclaimer: I work at Backblaze. > Do BB publish pricing of what they paid for drives? ... a TB/$/Failure rate Thank you, that is EXACTLY the correct way to look at the failure statistics! So many people seem to sort the list by failure rate and think no matter what the cost, the lowest failure rate wins. For Backblaze, we just feed it back into the cost calculation. For example: If a drive fails 1% more often but is 2% cheaper in total cost of ownership, we buy that drive. Now, total cost of ownership includes the physical space rental so more dense drives can be more expensive per TByte in raw drive cost because we can make some of that back up in physical space rental. Also, most drives seem to take about the same amount of electricity unrelated to how many TBytes are contained inside, so double the drive density is like saying it takes half as much electricity over its 4 - 5 year lifespan. Electricity is one of our largest datacenter costs, so we keep an eye on that also. But to answer your very first question, unfortunately we cannot release the price we paid for the drives due to the vendors requesting we don't disclose prices. But don't think we have some magically huge discount or anything. Most of the time we are literally paying about retail, maybe a 3% - 5% discount for buying in bulk orders of over 10,000 drives. But for reasons nobody at Backblaze can figure out, sometimes a bunch of new drives appears randomly at a really good discount price, then returns to the original price the next month. It might be some attempt along the supply chain to boost monthly or quarterly top line at the expense of profits, I don't know.
  - Dylan16807 8y ago
    Cost adjusted by failure rate is definitely the best metric for a RAID. But when it's only one or two disks, and any failure means a giant hassle, it can be worth paying more to improve the odds.
  - brokenmachine 8y ago
    When I'm just buying drives for my home machines, I'd pay much more for a lower failure rate because a dead drive means a lot of time and massive inconvenience plus some research and a trip to the computer store to buy a replacement. I'd honestly pay double if I could be guaranteed to avoid all that.
    - 8y ago
jewel 8y ago
A total of 704 petabytes of raw space, which I think is 600 petabytes of usable space. Looks like backblaze will cross the exabyte mark very soon. They crossed 500 PB in February [1]. Yev, in case you're around, I was curious what your plan was for a potential future shortage of supply of hard drives due to another disaster, political problem, etc. Perhaps just grow far enough ahead of your needs to have a buffer? I imagine you're now at a scale where you can't farm hard drives from costco anymore. Also what happens to all of those functional, but non-profitable 4TB drives? [1]: https://www.backblaze.com/blog/wp-content/uploads/2018/01/500pb_growth.jpg https://www.backblaze.com/blog/wp-content/uploads/2018/01/50...
- atYevP 8y ago
  Yev from Backblaze here -> of course I'm around :D Good questions! We recycle the drives that get removed from service. As for future shortages, that's one of the reasons we took a small round back in 2012 (https://www.backblaze.com/blog/backblaze-raises-5-million-why-we-took-funding-after-5-years-of-bootstrapping/ https://www.backblaze.com/blog/backblaze-raises-5-million-wh...). The majority of it was to spend on marketing experiments, but a little portion was to be used for buffer if an event like that ever happened again and hard drive prices spiked 300%. We're a lot larger now and do a pretty good job with forecasting and having rainy day funds, but we've also gotten to the point where we are now a bit less price sensitive, so we'd be able to handle the fluctuation. That said, if everything goes horribly wrong...https://www.backblaze.com/blog/backblaze_drive_farming/ https://www.backblaze.com/blog/backblaze_drive_farming/
  - wlesieutre 8y ago
    You might not be the right person to do anything about this, but I have a bookmarklet for hiding position:fixed elements when I'm trying to read something, and every time I scroll your blog it recreates the obnoxious share bar. Whyyyyyyy? EDIT: PS- fun story! I enjoyed reading it except for all the social media buttons.
    - atYevP 8y ago
      Hahahaha! Social sharing is great! Do not fear the share bar! Seriously though, sorry about that - not sure if we can do anything on our end as that's the way that widget works :( We might be able to add a "print" button so you can just print the post out, would that help?
vxNsr 8y ago
What do you guys do with decommissioned but functional hard-drives?
- deleted 8y ago
  [deleted]
- rtkwe 8y ago
  Like google probably wipe and maybe shred. I thought they were all shredded but it turns out they do sell decommissioned drives if they can be verified as 100% wiped. Google actually recently upgraded to using a robotic arm to manage the wiping and shredding of drives. [0] [0] https://www.datacenterknowledge.com/google-alphabet/robots-now-annihilate-hard-drives-google-data-centers https://www.datacenterknowledge.com/google-alphabet/robots-n...
  - briffle 8y ago
    I wonder where I can purchase those.. I have a personal Plex Server that could use a few 4TB used drives that are cheap.
    - rtkwe 8y ago
      I'm betting they probably sell them to a bulk reseller of which there are many. If you're looking to buy from one they're out there but may or may not be looking for small customers.
  - WaxProlix 8y ago
    Little confusing, are you referring to shred the tool or some physical destruction process? https://linux.die.net/man/1/shred https://linux.die.net/man/1/shred
    - rtkwe 8y ago
      Physically shredding. https://youtu.be/TQoKFovvigI?t=98 https://youtu.be/TQoKFovvigI?t=98
    - 8y ago
swang 8y ago
What does Backblaze do with all its older hard drives? Resell? Recycle?
- brianwski 8y ago
  Disclaimer: I work at Backblaze. > What does Backblaze do with all its older hard drives? Resell? Recycle? If the drive is still working (like you can read and write to it) when we cycle it out, we securely wipe it and then sometimes resell them in "bulk" to places like "Weird Stuff Warehouse" in the San Francisco area, or other places. If the drive is not working, we physically destroy the hard drives (special equipment) and dispose/recycle the electronic waste of the carcass.
  - Rafuino 8y ago
    I think the Weird Stuff Warehouse closed down this past year. What are other examples of where you sell these cycled out but still operational drives?
  - corobo 8y ago
    > physically destroy the hard drives (special equipment) Is that as fun a day as I'm picturing?
    - dsfyu404ed 8y ago
      When you've got hundreds or thousands of drives to destroy it's not worth the man hours or consumables to cut them or drill holes in them. Even shooting them gets old quickly. When I worked for a defense contractor we started one of our daily infosec briefings with a video of one of our shipyards destroying a few hundred hard drives. They put them all in a shipyard size press brake. The end result was very long line of C shaped hard drives. Anyone who deals with enough drives and doesn't have other business (like a shipyard) that they can borrow suitable equipment from will likely wind up buying an industrial shredder.[1] [1] https://www.youtube.com/watch?v=sQYPCPB1g3o https://www.youtube.com/watch?v=sQYPCPB1g3o
    - jaclaz 8y ago
chrisper 8y ago
Any plans on the European Datacenter?
- atYevP 8y ago
  Yev from Backblaze here -> yes but no ETAs at the moment!
  - mjevans 8y ago
    That would be spectacular; I know this is obvious, but please do make sure that it actually can operate independently of the other locations and that end users have an easy way of controlling which region their data sits in. There are legal reasons some companies might need to keep their data within the EU, and for others having multi-region backups are a checkbox to tick off on disaster preparations. Both are things the marketing department should use.
deleted 8y ago
[deleted]
arminiusreturns 8y ago
As a sysadmin I can't say how much I appreciate BB publishing these each year. it helps me make informed decisions on drive selection backed by data.
- atYevP 8y ago
  Yev from Backblaze -> Glad you like it! *Edit -> I was just at SpiceWorld in Austin and a lot of the sys admins who were there enjoyed it quiet a bit! I'm headed to JNUC in Minneapolis next week and last year we talked a lot about hard drive stats as well!
xmichael999 8y ago
I always enjoy this site for anyone in the market. https://diskprices.com/?locale=us https://diskprices.com/?locale=us FYI I have nothing to do with the site, just a user.
- jaclaz 8y ago
  Sure, but, at least for my local, ALL links are to Amazon.
antman 8y ago
Something seems to go wrong with WDC 6TB.
- breakingcups 8y ago
  It's a relatively small sample though. 5 failed drives.

175 points by LaSombra 8y ago

hanyoloren 8y ago

cool

deleted 8y ago

[deleted]

Dennip 8y ago

Do BB publish pricing of what they paid for drives? Sort of a TB/$/Failure rate. Those 12TB HGST drives must cost a ton

geostyx 8y ago
No they don't.
- jason46 8y ago
  $33,500 if you bought them from amazon right now, plus whatever it costs to ship 79 hard drives.
  - brianwski 8y ago
    Disclaimer: I work at Backblaze. > $33,500 if you bought them from amazon right now About half of all of the money that flows into Backblaze from customers gets spent on the datacenter, and the largest component of that is drives. I think sales tax and property tax is the second largest datacenter component. It turns out you don't just owe property tax on buildings and land - if you own something of significant worth like $10 million worth of hard drives in a large pile, you owe the government "property tax" every month for owning that property.
    - lotsofpulp 8y ago
      Yes, called "tangible personal property tax" (even though it's used for business). Some states have it and some don't, with various exceptions and whatnot in all of them.
    - PhantomGremlin 8y ago
      if you own something of significant worth like $10 million worth of hard drives in a large pile, you owe the government "property tax" every month for owning that property. If you're Intel, you potentially owe "property tax" every month on all the equipment in multi-billion dollar fabs. That's why those fabs only get built in states where the tax doesn't apply or where the state/county agrees to forego it.
    - dorfsmay 8y ago
      I had never thought of income on capital vs income before Piketty made it the center of his theories, but since then realised that it is exactly what property tax is, which is weird that it is only in one type of capital.
atYevP 8y ago
Yev from Backblaze here -> We don't typically post the cost of the drives, but one thing you can be sure of, is that we go for as good of a price-to-density ratio as we can! So if they were much more expensive than their counterparts we would likely avoid them until they came down enough for us to test them out!
brianwski 8y ago
Disclaimer: I work at Backblaze. > Do BB publish pricing of what they paid for drives? ... a TB/$/Failure rate Thank you, that is EXACTLY the correct way to look at the failure statistics! So many people seem to sort the list by failure rate and think no matter what the cost, the lowest failure rate wins. For Backblaze, we just feed it back into the cost calculation. For example: If a drive fails 1% more often but is 2% cheaper in total cost of ownership, we buy that drive. Now, total cost of ownership includes the physical space rental so more dense drives can be more expensive per TByte in raw drive cost because we can make some of that back up in physical space rental. Also, most drives seem to take about the same amount of electricity unrelated to how many TBytes are contained inside, so double the drive density is like saying it takes half as much electricity over its 4 - 5 year lifespan. Electricity is one of our largest datacenter costs, so we keep an eye on that also. But to answer your very first question, unfortunately we cannot release the price we paid for the drives due to the vendors requesting we don't disclose prices. But don't think we have some magically huge discount or anything. Most of the time we are literally paying about retail, maybe a 3% - 5% discount for buying in bulk orders of over 10,000 drives. But for reasons nobody at Backblaze can figure out, sometimes a bunch of new drives appears randomly at a really good discount price, then returns to the original price the next month. It might be some attempt along the supply chain to boost monthly or quarterly top line at the expense of profits, I don't know.
- Dylan16807 8y ago
  Cost adjusted by failure rate is definitely the best metric for a RAID. But when it's only one or two disks, and any failure means a giant hassle, it can be worth paying more to improve the odds.
- brokenmachine 8y ago
  When I'm just buying drives for my home machines, I'd pay much more for a lower failure rate because a dead drive means a lot of time and massive inconvenience plus some research and a trip to the computer store to buy a replacement. I'd honestly pay double if I could be guaranteed to avoid all that.
  - 8y ago

jewel 8y ago

A total of 704 petabytes of raw space, which I think is 600 petabytes of usable space. Looks like backblaze will cross the exabyte mark very soon. They crossed 500 PB in February [1]. Yev, in case you're around, I was curious what your plan was for a potential future shortage of supply of hard drives due to another disaster, political problem, etc. Perhaps just grow far enough ahead of your needs to have a buffer? I imagine you're now at a scale where you can't farm hard drives from costco anymore. Also what happens to all of those functional, but non-profitable 4TB drives? [1]: https://www.backblaze.com/blog/wp-content/uploads/2018/01/500pb_growth.jpg https://www.backblaze.com/blog/wp-content/uploads/2018/01/50...

atYevP 8y ago
Yev from Backblaze here -> of course I'm around :D Good questions! We recycle the drives that get removed from service. As for future shortages, that's one of the reasons we took a small round back in 2012 (https://www.backblaze.com/blog/backblaze-raises-5-million-why-we-took-funding-after-5-years-of-bootstrapping/ https://www.backblaze.com/blog/backblaze-raises-5-million-wh...). The majority of it was to spend on marketing experiments, but a little portion was to be used for buffer if an event like that ever happened again and hard drive prices spiked 300%. We're a lot larger now and do a pretty good job with forecasting and having rainy day funds, but we've also gotten to the point where we are now a bit less price sensitive, so we'd be able to handle the fluctuation. That said, if everything goes horribly wrong...https://www.backblaze.com/blog/backblaze_drive_farming/ https://www.backblaze.com/blog/backblaze_drive_farming/
- wlesieutre 8y ago
  You might not be the right person to do anything about this, but I have a bookmarklet for hiding position:fixed elements when I'm trying to read something, and every time I scroll your blog it recreates the obnoxious share bar. Whyyyyyyy? EDIT: PS- fun story! I enjoyed reading it except for all the social media buttons.
  - atYevP 8y ago
    Hahahaha! Social sharing is great! Do not fear the share bar! Seriously though, sorry about that - not sure if we can do anything on our end as that's the way that widget works :( We might be able to add a "print" button so you can just print the post out, would that help?

vxNsr 8y ago

What do you guys do with decommissioned but functional hard-drives?

deleted 8y ago
[deleted]
rtkwe 8y ago
Like google probably wipe and maybe shred. I thought they were all shredded but it turns out they do sell decommissioned drives if they can be verified as 100% wiped. Google actually recently upgraded to using a robotic arm to manage the wiping and shredding of drives. [0] [0] https://www.datacenterknowledge.com/google-alphabet/robots-now-annihilate-hard-drives-google-data-centers https://www.datacenterknowledge.com/google-alphabet/robots-n...
- briffle 8y ago
  I wonder where I can purchase those.. I have a personal Plex Server that could use a few 4TB used drives that are cheap.
  - rtkwe 8y ago
    I'm betting they probably sell them to a bulk reseller of which there are many. If you're looking to buy from one they're out there but may or may not be looking for small customers.
- WaxProlix 8y ago
  Little confusing, are you referring to shred the tool or some physical destruction process? https://linux.die.net/man/1/shred https://linux.die.net/man/1/shred
  - rtkwe 8y ago
    Physically shredding. https://youtu.be/TQoKFovvigI?t=98 https://youtu.be/TQoKFovvigI?t=98
  - 8y ago

swang 8y ago

What does Backblaze do with all its older hard drives? Resell? Recycle?

brianwski 8y ago
Disclaimer: I work at Backblaze. > What does Backblaze do with all its older hard drives? Resell? Recycle? If the drive is still working (like you can read and write to it) when we cycle it out, we securely wipe it and then sometimes resell them in "bulk" to places like "Weird Stuff Warehouse" in the San Francisco area, or other places. If the drive is not working, we physically destroy the hard drives (special equipment) and dispose/recycle the electronic waste of the carcass.
- Rafuino 8y ago
  I think the Weird Stuff Warehouse closed down this past year. What are other examples of where you sell these cycled out but still operational drives?
- corobo 8y ago
  > physically destroy the hard drives (special equipment) Is that as fun a day as I'm picturing?
  - dsfyu404ed 8y ago
    When you've got hundreds or thousands of drives to destroy it's not worth the man hours or consumables to cut them or drill holes in them. Even shooting them gets old quickly. When I worked for a defense contractor we started one of our daily infosec briefings with a video of one of our shipyards destroying a few hundred hard drives. They put them all in a shipyard size press brake. The end result was very long line of C shaped hard drives. Anyone who deals with enough drives and doesn't have other business (like a shipyard) that they can borrow suitable equipment from will likely wind up buying an industrial shredder.[1] [1] https://www.youtube.com/watch?v=sQYPCPB1g3o https://www.youtube.com/watch?v=sQYPCPB1g3o
  - jaclaz 8y ago

chrisper 8y ago

Any plans on the European Datacenter?

atYevP 8y ago
Yev from Backblaze here -> yes but no ETAs at the moment!
- mjevans 8y ago
  That would be spectacular; I know this is obvious, but please do make sure that it actually can operate independently of the other locations and that end users have an easy way of controlling which region their data sits in. There are legal reasons some companies might need to keep their data within the EU, and for others having multi-region backups are a checkbox to tick off on disaster preparations. Both are things the marketing department should use.

deleted 8y ago

[deleted]

arminiusreturns 8y ago

As a sysadmin I can't say how much I appreciate BB publishing these each year. it helps me make informed decisions on drive selection backed by data.

atYevP 8y ago
Yev from Backblaze -> Glad you like it! *Edit -> I was just at SpiceWorld in Austin and a lot of the sys admins who were there enjoyed it quiet a bit! I'm headed to JNUC in Minneapolis next week and last year we talked a lot about hard drive stats as well!

xmichael999 8y ago

I always enjoy this site for anyone in the market. https://diskprices.com/?locale=us https://diskprices.com/?locale=us FYI I have nothing to do with the site, just a user.

jaclaz 8y ago
Sure, but, at least for my local, ALL links are to Amazon.

antman 8y ago

Something seems to go wrong with WDC 6TB.

breakingcups 8y ago
It's a relatively small sample though. 5 failed drives.