9 ms·

The fact that external drives that have to be "shucked" of their casing and circuit boards (at what environmental expense?) were cheaper than the same internal

The fact that external drives that have to be "shucked" of their casing and circuit boards (at what environmental expense?) were cheaper than the same internal drives makes you wonder whether the internal drive prices were realistic. It can't be cheaper to produce the external drives, so the internal drive prices must have been raised above the actual cost to the manufacturers, presumably because the internal drive demand was guaranteed. Consumers might hold off purchasing an external disk if the prices doubled, but Apple aren't going to stop selling Macs because of it. Sometimes capitalism isn't perfect.

DanBC 14y ago
Were they the same as the internal drives? Or just the same capacity? I get the impression they just kludged as many 3 Tb drives in as they could, and are ignoring any other specs. Maybe I'm missing something.
- bruceboughton 14y ago
  The article implies that all internal drives rocketed in price. The drives used in the external drives will most likely be a subset of the internal drives available.
  - brianwski 14y ago
    (Backblaze engineer here.) Exactly, we chose the particular external drives to contain the exact type of drives that we prefer in our datacenter.
- aes256 14y ago
  They may have different branding or model numbers, but in terms of technical specifications they are no different to internal drives.
- dsr_ 14y ago
  The prices on external drives lagged the prices on internal. The externals are generally sold with reduced warranty -- 1 year, or sometimes as low as 90 days -- and my suspicion has always been that many of them failed an internal longevity test and were binned.
  - bruceboughton 14y ago
    So Backblaze is facing a huge increase in drive failure rates in the near future? That's reassuring.
    - stephengillie 14y ago
      That could be a lot of liability they're holding, since they voided the warranties on the externals by shucking them. So they could wind up paying twice for failed HDDs = $130 + $130 + $25 in gas and wage to pay someone to drive around to pick these up = $285 or so per HDD? Plus 2 people got banned from Costco.
      - aes256 14y ago
        Failure rates are still likely to be low (<5% per annum) I've only ever bought five external drives in my lifetime. Incidentally, all of them have been "shucked" over the years, and none of them have failed. They vary from 3-5 years old. In fact, the only two drives I've had fail on me were OEM packaged, so based on my anecdotal, statistically insignificant personal experience you could even say the opposite, that external drives are more reliable than their OEM counterparts. You never know. Perhaps the additional packaging prevents damage to the drives in transit, and the process of installing software on the drives (as I believe some manufacturers do with external drives) provides another opportunity to quality check the drives.
      - atYevP 14y ago
        *Yev from Backblaze here: I can now purchase from Costco again, so all is well on that front. Even with a price at or around your calculated $285, that is still half as expensive as the $600 per drive we were once quoted during the crisis.
    - sjwright 14y ago
      Another way to look at it is they potentially found an inexpensive form of credit. They save money now and might have to repay it back later -- the interesting question will be how much "credit" they'll have to "repay" because of a higher failure rate. I hope they're keeping some basic metadata about the shucked drives, so they can compile some statistics for us. It would be fascinating to see what the lifespan of these drives will be over the next few years!
- brigade 14y ago
  MyBook Essentials (pictured) use the same Caviar Green internal hard drive, with the same warranty, which is still priced $10 more on amazon.
  - r00fus 14y ago
    Yes, think of the bare drive (if you don't need the enclosure) as "frustration free packaging" - ie, the kind that Amazon charges a good % markup - and people willingly pay for.
- budmang 14y ago
  Gleb from Backblaze here. The drives were literally the same inside the external enclosures. We bought a few to start with to check. We use consumer-grade drives across our entire 40+ petabyte storage farm because we do not need high-performance drives. Instead, we need low-cost, low-power, high-density drives to put into our custom-designed Storage Pods: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v... We tested "enterprise drives" and found that from a pure reliability perspective, they actually performed worse in our testing than the consumer drives. With about 5 years and 40+ petabytes of data, we're fairly confident this holds true. Finally, we assume that all drives will fail and replace about 10 drives every week. The software is written such that the system takes drive failures into account and data is stored redundantly and continuously cross-checked.
  - stephengillie 14y ago
    It's interesting to find that "enterprise drives" performed worse than consumer. Recently, a report came from a Microsoft datacenter, where they found most errors weren't from faulty hardware, but firmware incompatibilities between array controller and drive controller. I wonder if this type of incompatibility occurs more often for enterprise drives than consumer.
3pt14159 14y ago
Sure it is. You are watching capitalism in action. The company that could effectively utilize shucked hard drives did so. Larger companies that don't have the management oversight to follow this type of detail just let them be fore sale at what they already were.
sebtoast 14y ago
Most of the drives in external devices are "green" drives which has higher seek time and/or lower RPM. The reason why these drives' price have not increased as much/as fast is because the demand is lower than the "regular" internal drives. I did the same thing when the "crisis" happened, I bought a load of external drive and removed the hard disk from them.
- bruceboughton 14y ago
  What is a "green" drive?
  - stephengillie 14y ago
    It's a HDD which is supposed to use less electricity because it has a lower or variable spin speed. It's probably mostly a marketing gimmick.
    - rsync 14y ago
      ... and you should avoid them at all costs for any serious application. To this day there is still confusion between host adaptors (raid controllers) and green drives, and the firmware management process (you will absolutely need to manage the firmware on these) is atrocious.
    - jrabone 14y ago
      It's not a gimmick, but it's also a double-edged sword. They also do fun things like spinning down on an aggressive timeout, which can lead to early SMART warnings about head load/unload cycles if you keep waking them up to write periodic log files to them. I had to "fix" this with a nasty little DOS utility on a home Linux box I built using Seagate's version of this.
    - budmang 14y ago
      The "green" is certainly a gimmick. But the lower power usage is not. 10 Backblaze Storage Pods full of "green" drives use the same amount of power as 8 pods full of "regular" drives.
patio11 14y ago
Capitalism doesn't guarantee that things are priced in order of production costs. In particular, after these goods were produced, the ongoing inventory situation for hard drives had a wrench thrown into the works. One would hope the market then allocated a once-commodity component efficiently, such that users with the highest need for hard drives received them and users with less need made the decision to purchase hard drives later. For social (non-market) reasons, some people who sold hard drives decided to essentially sell hard drives below their market value, with rationing, to a) maintain availability of supply for a market segment with a relatively low need for hard drives while b) avoiding selling those hard drives into a market segment with a relatively high need for hard drives. Economics (not quite capitalism) says that if you attempt to replace a market with a rationing system, you will get a market anyhow, because markets are emergent features of reality and as soon as you have multiple agents with differing values for the same goods trade happens. (Capitalism would add "... and that's a good thing!") Rationing will generally result in there being an artificial consumer surplus created in the system versus a functioning market, and that consumer surplus will generally be captured by a market-maker rather than either any of the intended beneficiaries of the rationing scheme or the entity who has the product requiring rationing in the first place. In the absence of a rationing scheme, a) Backblaze would have had as many drives as it wished to purchase at the market price (slightly lower than the price they think is unconscionable in their normal channels) and b) most home users of hard drives would have been unable to source new hard drives at rates they were willing to pay, and would have turned to substitutes like "use refurbished hard drives", "decrease consumption of hard drive space", and "defer hard drive purchases."
stephengillie 14y ago
Those are loss leaders. Much like the 29c/lb turkey at Thanksgiving, or the $29 laptops the day after, these deals are designed to get consumers into the store or website, so each will buy 1 or 2 and several other items and accessories. High-margin accessories like USB cables and wireless mice, or potatoes and stuffing, easily offset those loss leaders. The entire point of the 2 item limit on deals like those is to prevent companies from doing exactly what Backblaze has done. By snapping up all of these deals, they're reducing the number available for consumer purchase, which impacts the revenues of these big-box retailers. Basically, Backblaze absorbed the (economic) consumer and retailer surpluses for HDDs in the Bay area and passed it on to their consumers.
rsync 14y ago
I agree with the gist of this, but it's worth noting that the drives that you typically buy for arrays like this are enterprise SATA drives (or at least 24/7 duty cycle desktop drives) and the drives inside the enclosures are the bottom of the barrel ...
- brianwski 14y ago
  (Backblaze engineer here.) Online backup is a unique application for hard drives. This is not a high transaction environment like a database getting pummeled with SQL queries. In online backup, we write the encrypted data only once, remember a SHA-1 to verify the data later, then we read it once a week to make sure the data still has perfect integrity (the SHA-1 must match). We now have over 10,000 hard drives spinning in our datacenter, some for over five years, and we've collected some pretty detailed statistics on drive failure rates vs many parameters. For our application (online backup) drives from external enclosures and consumer drives last every bit as long in our environment as "enterprise" drives.
  - toomuchtodo 14y ago
    I'm sure you guys keep track of read/write statistics as well; have you thought about spinning drives down on full BB pods until a restore is requested (similar to how Amazon Glacier's storage system works)?
    - mdaniel 14y ago
      I read once that the majority of drive failure happens during transitions to or from a running state, and thus it is counter-intuitively safer to just leave them running all the time. I am sure there is some mathematical formula that will express the risk based on how infrequently one expects to spin the drive up (and then down again), but I don't know it.
      - toomuchtodo 14y ago
        Backblaze is hosted out of a SF datacenter, where power is much more expensive than other US locations; I assume someone has done the math whether its cheaper to replace failed drives from frequent spinups/downs vs burning the power to keep them spinning.
shard 14y ago
Partly because Costco charges only a 15% markup to the cost they acquired the goods at (http://en.wikipedia.org/wiki/Costco#Sales_model http://en.wikipedia.org/wiki/Costco#Sales_model), which means that the external drives, purchased before the flood, were able to be sold at prices below the newly raised internal drive prices.

DanBC 14y ago

Were they the same as the internal drives? Or just the same capacity? I get the impression they just kludged as many 3 Tb drives in as they could, and are ignoring any other specs. Maybe I'm missing something.

bruceboughton 14y ago
The article implies that all internal drives rocketed in price. The drives used in the external drives will most likely be a subset of the internal drives available.
- brianwski 14y ago
  (Backblaze engineer here.) Exactly, we chose the particular external drives to contain the exact type of drives that we prefer in our datacenter.
aes256 14y ago
They may have different branding or model numbers, but in terms of technical specifications they are no different to internal drives.
dsr_ 14y ago
The prices on external drives lagged the prices on internal. The externals are generally sold with reduced warranty -- 1 year, or sometimes as low as 90 days -- and my suspicion has always been that many of them failed an internal longevity test and were binned.
- bruceboughton 14y ago
  So Backblaze is facing a huge increase in drive failure rates in the near future? That's reassuring.
  - stephengillie 14y ago
    That could be a lot of liability they're holding, since they voided the warranties on the externals by shucking them. So they could wind up paying twice for failed HDDs = $130 + $130 + $25 in gas and wage to pay someone to drive around to pick these up = $285 or so per HDD? Plus 2 people got banned from Costco.
    - aes256 14y ago
      Failure rates are still likely to be low (<5% per annum) I've only ever bought five external drives in my lifetime. Incidentally, all of them have been "shucked" over the years, and none of them have failed. They vary from 3-5 years old. In fact, the only two drives I've had fail on me were OEM packaged, so based on my anecdotal, statistically insignificant personal experience you could even say the opposite, that external drives are more reliable than their OEM counterparts. You never know. Perhaps the additional packaging prevents damage to the drives in transit, and the process of installing software on the drives (as I believe some manufacturers do with external drives) provides another opportunity to quality check the drives.
    - atYevP 14y ago
      *Yev from Backblaze here: I can now purchase from Costco again, so all is well on that front. Even with a price at or around your calculated $285, that is still half as expensive as the $600 per drive we were once quoted during the crisis.
  - sjwright 14y ago
    Another way to look at it is they potentially found an inexpensive form of credit. They save money now and might have to repay it back later -- the interesting question will be how much "credit" they'll have to "repay" because of a higher failure rate. I hope they're keeping some basic metadata about the shucked drives, so they can compile some statistics for us. It would be fascinating to see what the lifespan of these drives will be over the next few years!
brigade 14y ago
MyBook Essentials (pictured) use the same Caviar Green internal hard drive, with the same warranty, which is still priced $10 more on amazon.
- r00fus 14y ago
  Yes, think of the bare drive (if you don't need the enclosure) as "frustration free packaging" - ie, the kind that Amazon charges a good % markup - and people willingly pay for.
budmang 14y ago
Gleb from Backblaze here. The drives were literally the same inside the external enclosures. We bought a few to start with to check. We use consumer-grade drives across our entire 40+ petabyte storage farm because we do not need high-performance drives. Instead, we need low-cost, low-power, high-density drives to put into our custom-designed Storage Pods: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v... We tested "enterprise drives" and found that from a pure reliability perspective, they actually performed worse in our testing than the consumer drives. With about 5 years and 40+ petabytes of data, we're fairly confident this holds true. Finally, we assume that all drives will fail and replace about 10 drives every week. The software is written such that the system takes drive failures into account and data is stored redundantly and continuously cross-checked.
- stephengillie 14y ago
  It's interesting to find that "enterprise drives" performed worse than consumer. Recently, a report came from a Microsoft datacenter, where they found most errors weren't from faulty hardware, but firmware incompatibilities between array controller and drive controller. I wonder if this type of incompatibility occurs more often for enterprise drives than consumer.

3pt14159 14y ago

Sure it is. You are watching capitalism in action. The company that could effectively utilize shucked hard drives did so. Larger companies that don't have the management oversight to follow this type of detail just let them be fore sale at what they already were.

sebtoast 14y ago

Most of the drives in external devices are "green" drives which has higher seek time and/or lower RPM. The reason why these drives' price have not increased as much/as fast is because the demand is lower than the "regular" internal drives. I did the same thing when the "crisis" happened, I bought a load of external drive and removed the hard disk from them.

bruceboughton 14y ago
What is a "green" drive?
- stephengillie 14y ago
  It's a HDD which is supposed to use less electricity because it has a lower or variable spin speed. It's probably mostly a marketing gimmick.
  - rsync 14y ago
    ... and you should avoid them at all costs for any serious application. To this day there is still confusion between host adaptors (raid controllers) and green drives, and the firmware management process (you will absolutely need to manage the firmware on these) is atrocious.
  - jrabone 14y ago
    It's not a gimmick, but it's also a double-edged sword. They also do fun things like spinning down on an aggressive timeout, which can lead to early SMART warnings about head load/unload cycles if you keep waking them up to write periodic log files to them. I had to "fix" this with a nasty little DOS utility on a home Linux box I built using Seagate's version of this.
  - budmang 14y ago
    The "green" is certainly a gimmick. But the lower power usage is not. 10 Backblaze Storage Pods full of "green" drives use the same amount of power as 8 pods full of "regular" drives.

patio11 14y ago

Capitalism doesn't guarantee that things are priced in order of production costs. In particular, after these goods were produced, the ongoing inventory situation for hard drives had a wrench thrown into the works. One would hope the market then allocated a once-commodity component efficiently, such that users with the highest need for hard drives received them and users with less need made the decision to purchase hard drives later. For social (non-market) reasons, some people who sold hard drives decided to essentially sell hard drives below their market value, with rationing, to a) maintain availability of supply for a market segment with a relatively low need for hard drives while b) avoiding selling those hard drives into a market segment with a relatively high need for hard drives. Economics (not quite capitalism) says that if you attempt to replace a market with a rationing system, you will get a market anyhow, because markets are emergent features of reality and as soon as you have multiple agents with differing values for the same goods trade happens. (Capitalism would add "... and that's a good thing!") Rationing will generally result in there being an artificial consumer surplus created in the system versus a functioning market, and that consumer surplus will generally be captured by a market-maker rather than either any of the intended beneficiaries of the rationing scheme or the entity who has the product requiring rationing in the first place. In the absence of a rationing scheme, a) Backblaze would have had as many drives as it wished to purchase at the market price (slightly lower than the price they think is unconscionable in their normal channels) and b) most home users of hard drives would have been unable to source new hard drives at rates they were willing to pay, and would have turned to substitutes like "use refurbished hard drives", "decrease consumption of hard drive space", and "defer hard drive purchases."

stephengillie 14y ago

Those are loss leaders. Much like the 29c/lb turkey at Thanksgiving, or the $29 laptops the day after, these deals are designed to get consumers into the store or website, so each will buy 1 or 2 and several other items and accessories. High-margin accessories like USB cables and wireless mice, or potatoes and stuffing, easily offset those loss leaders. The entire point of the 2 item limit on deals like those is to prevent companies from doing exactly what Backblaze has done. By snapping up all of these deals, they're reducing the number available for consumer purchase, which impacts the revenues of these big-box retailers. Basically, Backblaze absorbed the (economic) consumer and retailer surpluses for HDDs in the Bay area and passed it on to their consumers.

rsync 14y ago

I agree with the gist of this, but it's worth noting that the drives that you typically buy for arrays like this are enterprise SATA drives (or at least 24/7 duty cycle desktop drives) and the drives inside the enclosures are the bottom of the barrel ...

brianwski 14y ago
(Backblaze engineer here.) Online backup is a unique application for hard drives. This is not a high transaction environment like a database getting pummeled with SQL queries. In online backup, we write the encrypted data only once, remember a SHA-1 to verify the data later, then we read it once a week to make sure the data still has perfect integrity (the SHA-1 must match). We now have over 10,000 hard drives spinning in our datacenter, some for over five years, and we've collected some pretty detailed statistics on drive failure rates vs many parameters. For our application (online backup) drives from external enclosures and consumer drives last every bit as long in our environment as "enterprise" drives.
- toomuchtodo 14y ago
  I'm sure you guys keep track of read/write statistics as well; have you thought about spinning drives down on full BB pods until a restore is requested (similar to how Amazon Glacier's storage system works)?
  - mdaniel 14y ago
    I read once that the majority of drive failure happens during transitions to or from a running state, and thus it is counter-intuitively safer to just leave them running all the time. I am sure there is some mathematical formula that will express the risk based on how infrequently one expects to spin the drive up (and then down again), but I don't know it.
    - toomuchtodo 14y ago
      Backblaze is hosted out of a SF datacenter, where power is much more expensive than other US locations; I assume someone has done the math whether its cheaper to replace failed drives from frequent spinups/downs vs burning the power to keep them spinning.

shard 14y ago

Partly because Costco charges only a 15% markup to the cost they acquired the goods at (http://en.wikipedia.org/wiki/Costco#Sales_model http://en.wikipedia.org/wiki/Costco#Sales_model), which means that the external drives, purchased before the flood, were able to be sold at prices below the newly raised internal drive prices.