12 ms·

Farming hard drives: how Backblaze weathered the Thailand drive crisis

187 points by zorlem 14y ago

bruceboughton 14y ago
If BackBlaze are adding 50TB of data a day, are they really buying drives from NewEgg?
- zorlem 14y ago
  That's one of the things that caught my attention as well. It seems strange, I would've imagined that at 50TB/day (~500 x 3TB HDDs/month) they could get a nice deal with a direct importer. I guess there is a small PR twist to the whole story ;-) with newegg being a well-known name. I don't know if the 50TB/day figure includes the replacement drives or only expanding the capacity as I guess their failure rate is quite high. edit: fixed a typo.
  - bruceboughton 14y ago
    My guess would be that the 50TB/day figure is before de-duplication and not at all relevant to the actual storage, but even at 10% of that I would expect them to be sourcing them wholesale, not retail. In fact, this whole article is making me doubt my faith in Backblaze as an off-site backup solution.
    - recuter 14y ago
      I remember a reddit AMA where they mentioned that they don't actually do any de-duplication. This isn't Dropbox and nowhere as reliable as S3 - its literally as if you're renting remote consumer external drives for offsite backup, for that usage pattern Backblaze makes some sense. And those pods are cool. But I agree, a somewhat flawed approach.
      - setec 14y ago
        I don't think it should be assumed that there is no data redundancy between pods, afterall from prior blog posts it seems like the pods can be taken offline. I'd also posit that a properly constructed RAID array can mitigate the risks of using consumer parts; afterall that is the "I" in RAID.
      - zorlem 14y ago
        >This isn't Dropbox and nowhere as reliable as S3 - its literally as if you're renting remote consumer external drives for offsite backup What are their reliability promises? I think that they're doing some mirroring of the data, but I'm still not sure about the rest of their service details. I'm considering using them for backups, but have not decided yet. I came upon this [1] discussion here, from 3 years ago. I'm diving in :) Yeah, the pods are really sweet and they provide all the design documents [2]! [1]: http://news.ycombinator.com/item?id=803136 http://news.ycombinator.com/item?id=803136 [2]: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v... edit: added additional details about the design
  - deleted 14y ago
    [deleted]
- smackfu 14y ago
  Why not let NewEgg hold the inventory if their prices are good?
- mbell 14y ago
  NewEgg isn't just retail anymore. They have a 'NewEgg Business' program with pretty solid volume discounts.
- michaelt 14y ago
  A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams. I could order them individually from Amazon for £27 each with free delivery. I phoned up all the official Microsoft distributors in my country, who phoned up Microsoft and whatnot. Not one of them could offer me a price below about £35. And getting in contact with Microsoft hardware directly? Ha, I should be so lucky.
  - andrewf 14y ago
    A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams. I'll bite. Why? :)
    - michaelt 14y ago
      Asked to quote to make a system like [1] for a fleet of 1000 vans. Buying in USB cameras would reduce engineering costs compared to making our own. Eventually went for a brand that cost less and could accept wide angle lenses. [1] http://www.smartdrive.net http://www.smartdrive.net
  - corin_ 14y ago
    More often than not it's a case of who you know when it comes to stuff like this.
- brianwski 14y ago
  (Backblaze engineer here.) We have been told that until we buy blocks of 10,000 drives, we cannot deal directly with the drive manufacturers. We're not there yet, thus we NORMALLY bid out each drive purchase (about one large order per month) to four or five regular "resellers" and "distributors" like CDW, Ingram, B&H, etc and take the best price that month. We were down at $90 or so for individual Hitachi 3 TBytes at the low point before the crisis. We're currently paying around $160 from CDW and friends and we can afford that without raising our prices. During the Thailand flood crisis, CDW offered "not enough drives" (half our order) for $350 per drive. We said "no", because we couldn't afford "yes". :-) Anybody can win our business by under bidding the other resellers, and we're ALWAYS open to suggestions. If you know any way to achieve lower drive prices, let us know! PLEASE!!
- polyfractal 14y ago
  I don't understand this comment at all. If anything, it has redoubled my interest in remaining a (paying) customer of BackBlaze. Look at the lengths they will go to securing drives so that I can continue to have stupid-cheap backup! I can't ask for more. And realistically, if push comes to shove, I'm sure BackBlaze would send out a heartfelt email to their loyal customers explaining the situation and kindly requesting a price increase. Especially after reading a post like the one they just submitted...I'm sure no one would have a problem ponying up a few more dollars.
pella 14y ago
same news ; and comments: http://news.ycombinator.com/item?id=4630910 http://news.ycombinator.com/item?id=4630910
- damncabbage 14y ago
  This one is the original article, and is better-written too. (Gigaom blogspammed it.)
yock 14y ago
If I were their customer, I would be looking for alternatives right about now...
- zorlem 14y ago
  Why would you do that? Due to reliability concerns, since they're using 3TB SATA drives, sourced by Cotsco and Best Buy?
  - bitJericho 14y ago
    I'd be shocked if they were not the same drives you buy from a supplier.
    - zorlem 14y ago
      I mostly agree with this. That's why I asked about GP's motivation. I suspect, though, without any actual data to back this up, that the drives that go into (low-cost) external HDD enclosures have lower MTBF ratings than the desktop-oriented internal drives. I think that the usage pattern for an external HDD is very different in terms of longevity in operation than desktop hard drives (and these have lower guarantees than eg. server-grade SATA drives). If I was a manufacturer of such enclosures I'd be willing to buy cheaper drives with lower promised MTBF for embedding them in enclosures that stay off most of the time. So, even if they're the same, I guess the manufacturer performs some tests and sells drives that fail some non-critical ones at a lower price (pretty much what Intel and AMD do with their chips - those that can't sustain the highest frequencies are downgraded to clock rates that are stable). edit: The comment from @dsr_ tends to support my hypothesis, although I haven't looked at the differences in guarantees of external enclosures, it's true that internal desktop HDDs can have up to 5 years of guarantee (i.e. WD's Black series has a 5 years guarantee).
      - alexkus 14y ago
        But then Backblaze's use of HDDs is not the same as an HDD stuck in a desktop machine and used as a primary/secondary drive. It's why their restore isn't instantaneous. You have to ask for the files and they'll let you know when they're ready, this takes time (up to several hours). AIUI they essentially fill up each Pod with incoming data and, when it's full it goes into a low power mode where they spin down all of the disks in that Pod. Restore requests (from many users) will usually span quite a few Pods so they batch up requests for each Pod so that can minimise the number of times that drives in a Pod need to be spun up to copy off files. Rather than spinning up and down once every 10 minutes to service 36 requests over 6 hours they wait 6 hours, spin up once and service all 36 requests for that pod in one go. (Figures are for example only.) (Small restores may be instantaneous if the Pod is still being filled or just happens to be spun up for restore soon. Large restores will more than likely span multiple pods and so there's more of a delay until all parts of the restore have been obtained from all of the pods involved.) If anything this maps pretty well to the expected usage pattern of an HDD in an external enclosure as many of them are used for infrequent backups rather than extending constant use storage. It's how I expect Backblaze will be hoping to get much more than the usual MTBF out of their drives they have purchased. HDD guarantees may be time based, but HDD failure rates are linked much more closely with usage patterns; number of spin-ups/downs, number of seeks, number of sector writes, etc. A drive in constant use on a desktop machine is going to be hammered a lot more than one that's essentially written to just once (as the pod fills up) and then spun up every now and again for partial reads.
bruceboughton 14y ago
The fact that external drives that have to be "shucked" of their casing and circuit boards (at what environmental expense?) were cheaper than the same internal drives makes you wonder whether the internal drive prices were realistic. It can't be cheaper to produce the external drives, so the internal drive prices must have been raised above the actual cost to the manufacturers, presumably because the internal drive demand was guaranteed. Consumers might hold off purchasing an external disk if the prices doubled, but Apple aren't going to stop selling Macs because of it. Sometimes capitalism isn't perfect.
- DanBC 14y ago
  Were they the same as the internal drives? Or just the same capacity? I get the impression they just kludged as many 3 Tb drives in as they could, and are ignoring any other specs. Maybe I'm missing something.
  - bruceboughton 14y ago
    The article implies that all internal drives rocketed in price. The drives used in the external drives will most likely be a subset of the internal drives available.
    - brianwski 14y ago
      (Backblaze engineer here.) Exactly, we chose the particular external drives to contain the exact type of drives that we prefer in our datacenter.
  - aes256 14y ago
    They may have different branding or model numbers, but in terms of technical specifications they are no different to internal drives.
  - dsr_ 14y ago
    The prices on external drives lagged the prices on internal. The externals are generally sold with reduced warranty -- 1 year, or sometimes as low as 90 days -- and my suspicion has always been that many of them failed an internal longevity test and were binned.
JakeFratelli 14y ago
Their execution was poor. Start by depleting SF and then look elsewhere? Short-sighted at best. They should have moved quickly before Best Buy/Costco implemented constraints that should have been anticipated. During one weekend, via train or one-way flights, employees could have been mobilized to cities across the US. Utilizing USPS flat rate boxes or low-cost equivalent, hard drives could have been shipped back to HQ en masse. Three days, disaster is over. I give them a C- on execution.
- rtkwe 14y ago
  That's predicated on them being able to spare many employees just for drive sourcing, which as a relatively small company they probably couldn't. They went with the lower cost stop gap at first, if the shortage hadn't lasted as long they would have been better off with how they did it.
- unreal37 14y ago
  Yeah sure. In a weekend they could have purchased 1500 hard drives when there is a shortage and stores are limiting them to 2? I would challenge you to go to retail stores in a single day under those conditions and buy 25.
andrewcooke 14y ago
what happened to the post that raised ethical questions about this? i think it had a valid point. the hard drive shortage hit us all. this one company coordinated a mass buying operation from commercial stores when the rest of use were looking for personal use. while i don't think this is illegal, it doesn't make me feel warm and fuzzy inside either. when there's a shortage, the "noise" in traditional distribution means that small buyers (like me, and many others) can still find what we need if we put in some extra effort. it functions like a "soft buffer", making the transition to a complete lack of resources more gradual. action like this removes that buffer. this feels like one group of people - driven by their desire to do business - beat out a load of other people who didn't have the same logistical resources. there's something uncool about it. i am not sure i have explained very well, but it's to do with crossing the line from business to personal. [edit: ah, thanks - separate thread: http://news.ycombinator.com/item?id=4630910 http://news.ycombinator.com/item?id=4630910] [edit2: wtf? i thought someone replied here? was about to upvote you. thanks anyway.]
- gadders 14y ago
  I believe this is called "hustle".
  - finnw 14y ago
    It does not fit any definition of "hustle" that I am aware of, including these: http://en.wiktionary.org/wiki/hustle http://en.wiktionary.org/wiki/hustle
    - cdmoyer 14y ago
      It's definition 2a[1] on merriam-webster.com. a : to obtain by energetic activity <hustle up new customers> Seems like a pretty common use of the term in the sense of "that scrappy start-up showed quite a bit of hustle to pull it off." [1] http://www.merriam-webster.com/dictionary/hustle http://www.merriam-webster.com/dictionary/hustle
- deleted 14y ago
  [deleted]
rsync 14y ago
We had this problem at rsync.net ... the price of our main consumable tripling overnight, etc. We really never considered "shucking" external drives, since the drives inside these enclosures are the worst spec'd, highest failure rate drives ... never mind the (usually) nonsensical enterprise vs. non-enterprise, they're just crummy drives. Good for BB for making it work ... BTW, hard drive price weirdness continues to this day - 3TB Hitachi enterprise drives two months ago were $198 (or so) and last week they were $248 (or so).
- mrb 14y ago
  But Blackblaze's infrastructure is properly designed to tolerate hw failures. They use RAID6 and replicate your data to 2 or 3 different storage pods. Smarter than beint forced to buy "better" expensive drives IMHO when your primary business expense is raw storage.
  - luser001 14y ago
    If a disk fails, it still needs to be replaced. RAID etc lets you avoid data loss until you replace the drive.
w1ntermute 14y ago
I wonder if the hard drive manufacturers will reconsider locating their entire supply chain in a third world country with frequent natural disasters after this incident. With rising wages in the third world and stagnating wages in America, perhaps the financial benefits of outsourcing aren't so apparent anymore.
- rogerbinns 14y ago
  They have more data about all the pieces and processes leading to their ultimate market share and profitability so I have no doubt they try to maximize those. I suspect the biggest issues are that all their suppliers and customers are in that region. Think fabs to make the chips, the various motors and related electronics, coating processes, and then sending them off to the places that make HP/Dell/etc systems. If just the drive factory were moved then you'd still have the drive parts crossing the ocean to get to the US and the completed drives crossing once again to go back into the systems. Usually the relocations happen when a disruptive solution comes along. You get the advantage of a somewhat clean sheet. For example SSDs don't need all the mechanical components and fine tuning, so they can be made with a completely different supply chain. It won't happen in hard drives but for other products customization is what can spur relocation. For example Lenovo is opening a manufacturing plant in the US to deal with customization which allows quicker response times. (Customized orders often can charge a little more which offsets increased costs.)
granite 14y ago
I was impressed with the effort to get those drives. If the massive companies who produce hard drives were unprepared for such a natural disaster I can't say I expect more from a small company.
abalone 14y ago
Now that's a startup story. Backblaze has some of the best down home old fashioned spit n glue startup stories on hacker news. Like the one where they invented their own storage pod out of commodity components to beat amazon's pricing. And now this stay-alive-against-all-odds story. Fantastic.
deleted 14y ago
[deleted]

187 points by zorlem 14y ago

bruceboughton 14y ago

If BackBlaze are adding 50TB of data a day, are they really buying drives from NewEgg?

zorlem 14y ago
That's one of the things that caught my attention as well. It seems strange, I would've imagined that at 50TB/day (~500 x 3TB HDDs/month) they could get a nice deal with a direct importer. I guess there is a small PR twist to the whole story ;-) with newegg being a well-known name. I don't know if the 50TB/day figure includes the replacement drives or only expanding the capacity as I guess their failure rate is quite high. edit: fixed a typo.
- bruceboughton 14y ago
  My guess would be that the 50TB/day figure is before de-duplication and not at all relevant to the actual storage, but even at 10% of that I would expect them to be sourcing them wholesale, not retail. In fact, this whole article is making me doubt my faith in Backblaze as an off-site backup solution.
  - recuter 14y ago
    I remember a reddit AMA where they mentioned that they don't actually do any de-duplication. This isn't Dropbox and nowhere as reliable as S3 - its literally as if you're renting remote consumer external drives for offsite backup, for that usage pattern Backblaze makes some sense. And those pods are cool. But I agree, a somewhat flawed approach.
    - setec 14y ago
      I don't think it should be assumed that there is no data redundancy between pods, afterall from prior blog posts it seems like the pods can be taken offline. I'd also posit that a properly constructed RAID array can mitigate the risks of using consumer parts; afterall that is the "I" in RAID.
    - zorlem 14y ago
      >This isn't Dropbox and nowhere as reliable as S3 - its literally as if you're renting remote consumer external drives for offsite backup What are their reliability promises? I think that they're doing some mirroring of the data, but I'm still not sure about the rest of their service details. I'm considering using them for backups, but have not decided yet. I came upon this [1] discussion here, from 3 years ago. I'm diving in :) Yeah, the pods are really sweet and they provide all the design documents [2]! [1]: http://news.ycombinator.com/item?id=803136 http://news.ycombinator.com/item?id=803136 [2]: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v... edit: added additional details about the design
- deleted 14y ago
  [deleted]
smackfu 14y ago
Why not let NewEgg hold the inventory if their prices are good?
mbell 14y ago
NewEgg isn't just retail anymore. They have a 'NewEgg Business' program with pretty solid volume discounts.
michaelt 14y ago
A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams. I could order them individually from Amazon for £27 each with free delivery. I phoned up all the official Microsoft distributors in my country, who phoned up Microsoft and whatnot. Not one of them could offer me a price below about £35. And getting in contact with Microsoft hardware directly? Ha, I should be so lucky.
- andrewf 14y ago
  A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams. I'll bite. Why? :)
  - michaelt 14y ago
    Asked to quote to make a system like [1] for a fleet of 1000 vans. Buying in USB cameras would reduce engineering costs compared to making our own. Eventually went for a brand that cost less and could accept wide angle lenses. [1] http://www.smartdrive.net http://www.smartdrive.net
- corin_ 14y ago
  More often than not it's a case of who you know when it comes to stuff like this.
brianwski 14y ago
(Backblaze engineer here.) We have been told that until we buy blocks of 10,000 drives, we cannot deal directly with the drive manufacturers. We're not there yet, thus we NORMALLY bid out each drive purchase (about one large order per month) to four or five regular "resellers" and "distributors" like CDW, Ingram, B&H, etc and take the best price that month. We were down at $90 or so for individual Hitachi 3 TBytes at the low point before the crisis. We're currently paying around $160 from CDW and friends and we can afford that without raising our prices. During the Thailand flood crisis, CDW offered "not enough drives" (half our order) for $350 per drive. We said "no", because we couldn't afford "yes". :-) Anybody can win our business by under bidding the other resellers, and we're ALWAYS open to suggestions. If you know any way to achieve lower drive prices, let us know! PLEASE!!
polyfractal 14y ago
I don't understand this comment at all. If anything, it has redoubled my interest in remaining a (paying) customer of BackBlaze. Look at the lengths they will go to securing drives so that I can continue to have stupid-cheap backup! I can't ask for more. And realistically, if push comes to shove, I'm sure BackBlaze would send out a heartfelt email to their loyal customers explaining the situation and kindly requesting a price increase. Especially after reading a post like the one they just submitted...I'm sure no one would have a problem ponying up a few more dollars.

pella 14y ago

same news ; and comments: http://news.ycombinator.com/item?id=4630910 http://news.ycombinator.com/item?id=4630910

damncabbage 14y ago
This one is the original article, and is better-written too. (Gigaom blogspammed it.)

yock 14y ago

If I were their customer, I would be looking for alternatives right about now...

zorlem 14y ago
Why would you do that? Due to reliability concerns, since they're using 3TB SATA drives, sourced by Cotsco and Best Buy?
- bitJericho 14y ago
  I'd be shocked if they were not the same drives you buy from a supplier.
  - zorlem 14y ago
    I mostly agree with this. That's why I asked about GP's motivation. I suspect, though, without any actual data to back this up, that the drives that go into (low-cost) external HDD enclosures have lower MTBF ratings than the desktop-oriented internal drives. I think that the usage pattern for an external HDD is very different in terms of longevity in operation than desktop hard drives (and these have lower guarantees than eg. server-grade SATA drives). If I was a manufacturer of such enclosures I'd be willing to buy cheaper drives with lower promised MTBF for embedding them in enclosures that stay off most of the time. So, even if they're the same, I guess the manufacturer performs some tests and sells drives that fail some non-critical ones at a lower price (pretty much what Intel and AMD do with their chips - those that can't sustain the highest frequencies are downgraded to clock rates that are stable). edit: The comment from @dsr_ tends to support my hypothesis, although I haven't looked at the differences in guarantees of external enclosures, it's true that internal desktop HDDs can have up to 5 years of guarantee (i.e. WD's Black series has a 5 years guarantee).
    - alexkus 14y ago
      But then Backblaze's use of HDDs is not the same as an HDD stuck in a desktop machine and used as a primary/secondary drive. It's why their restore isn't instantaneous. You have to ask for the files and they'll let you know when they're ready, this takes time (up to several hours). AIUI they essentially fill up each Pod with incoming data and, when it's full it goes into a low power mode where they spin down all of the disks in that Pod. Restore requests (from many users) will usually span quite a few Pods so they batch up requests for each Pod so that can minimise the number of times that drives in a Pod need to be spun up to copy off files. Rather than spinning up and down once every 10 minutes to service 36 requests over 6 hours they wait 6 hours, spin up once and service all 36 requests for that pod in one go. (Figures are for example only.) (Small restores may be instantaneous if the Pod is still being filled or just happens to be spun up for restore soon. Large restores will more than likely span multiple pods and so there's more of a delay until all parts of the restore have been obtained from all of the pods involved.) If anything this maps pretty well to the expected usage pattern of an HDD in an external enclosure as many of them are used for infrequent backups rather than extending constant use storage. It's how I expect Backblaze will be hoping to get much more than the usual MTBF out of their drives they have purchased. HDD guarantees may be time based, but HDD failure rates are linked much more closely with usage patterns; number of spin-ups/downs, number of seeks, number of sector writes, etc. A drive in constant use on a desktop machine is going to be hammered a lot more than one that's essentially written to just once (as the pod fills up) and then spun up every now and again for partial reads.

bruceboughton 14y ago

The fact that external drives that have to be "shucked" of their casing and circuit boards (at what environmental expense?) were cheaper than the same internal drives makes you wonder whether the internal drive prices were realistic. It can't be cheaper to produce the external drives, so the internal drive prices must have been raised above the actual cost to the manufacturers, presumably because the internal drive demand was guaranteed. Consumers might hold off purchasing an external disk if the prices doubled, but Apple aren't going to stop selling Macs because of it. Sometimes capitalism isn't perfect.

DanBC 14y ago
Were they the same as the internal drives? Or just the same capacity? I get the impression they just kludged as many 3 Tb drives in as they could, and are ignoring any other specs. Maybe I'm missing something.
- bruceboughton 14y ago
  The article implies that all internal drives rocketed in price. The drives used in the external drives will most likely be a subset of the internal drives available.
  - brianwski 14y ago
    (Backblaze engineer here.) Exactly, we chose the particular external drives to contain the exact type of drives that we prefer in our datacenter.
- aes256 14y ago
  They may have different branding or model numbers, but in terms of technical specifications they are no different to internal drives.
- dsr_ 14y ago
  The prices on external drives lagged the prices on internal. The externals are generally sold with reduced warranty -- 1 year, or sometimes as low as 90 days -- and my suspicion has always been that many of them failed an internal longevity test and were binned.

JakeFratelli 14y ago

Their execution was poor. Start by depleting SF and then look elsewhere? Short-sighted at best. They should have moved quickly before Best Buy/Costco implemented constraints that should have been anticipated. During one weekend, via train or one-way flights, employees could have been mobilized to cities across the US. Utilizing USPS flat rate boxes or low-cost equivalent, hard drives could have been shipped back to HQ en masse. Three days, disaster is over. I give them a C- on execution.

rtkwe 14y ago
That's predicated on them being able to spare many employees just for drive sourcing, which as a relatively small company they probably couldn't. They went with the lower cost stop gap at first, if the shortage hadn't lasted as long they would have been better off with how they did it.
unreal37 14y ago
Yeah sure. In a weekend they could have purchased 1500 hard drives when there is a shortage and stores are limiting them to 2? I would challenge you to go to retail stores in a single day under those conditions and buy 25.

andrewcooke 14y ago

what happened to the post that raised ethical questions about this? i think it had a valid point. the hard drive shortage hit us all. this one company coordinated a mass buying operation from commercial stores when the rest of use were looking for personal use. while i don't think this is illegal, it doesn't make me feel warm and fuzzy inside either. when there's a shortage, the "noise" in traditional distribution means that small buyers (like me, and many others) can still find what we need if we put in some extra effort. it functions like a "soft buffer", making the transition to a complete lack of resources more gradual. action like this removes that buffer. this feels like one group of people - driven by their desire to do business - beat out a load of other people who didn't have the same logistical resources. there's something uncool about it. i am not sure i have explained very well, but it's to do with crossing the line from business to personal. [edit: ah, thanks - separate thread: http://news.ycombinator.com/item?id=4630910 http://news.ycombinator.com/item?id=4630910] [edit2: wtf? i thought someone replied here? was about to upvote you. thanks anyway.]

gadders 14y ago
I believe this is called "hustle".
- finnw 14y ago
  It does not fit any definition of "hustle" that I am aware of, including these: http://en.wiktionary.org/wiki/hustle http://en.wiktionary.org/wiki/hustle
  - cdmoyer 14y ago
    It's definition 2a[1] on merriam-webster.com. a : to obtain by energetic activity <hustle up new customers> Seems like a pretty common use of the term in the sense of "that scrappy start-up showed quite a bit of hustle to pull it off." [1] http://www.merriam-webster.com/dictionary/hustle http://www.merriam-webster.com/dictionary/hustle
deleted 14y ago
[deleted]

rsync 14y ago

We had this problem at rsync.net ... the price of our main consumable tripling overnight, etc. We really never considered "shucking" external drives, since the drives inside these enclosures are the worst spec'd, highest failure rate drives ... never mind the (usually) nonsensical enterprise vs. non-enterprise, they're just crummy drives. Good for BB for making it work ... BTW, hard drive price weirdness continues to this day - 3TB Hitachi enterprise drives two months ago were $198 (or so) and last week they were $248 (or so).

mrb 14y ago
But Blackblaze's infrastructure is properly designed to tolerate hw failures. They use RAID6 and replicate your data to 2 or 3 different storage pods. Smarter than beint forced to buy "better" expensive drives IMHO when your primary business expense is raw storage.
- luser001 14y ago
  If a disk fails, it still needs to be replaced. RAID etc lets you avoid data loss until you replace the drive.

w1ntermute 14y ago

I wonder if the hard drive manufacturers will reconsider locating their entire supply chain in a third world country with frequent natural disasters after this incident. With rising wages in the third world and stagnating wages in America, perhaps the financial benefits of outsourcing aren't so apparent anymore.

rogerbinns 14y ago
They have more data about all the pieces and processes leading to their ultimate market share and profitability so I have no doubt they try to maximize those. I suspect the biggest issues are that all their suppliers and customers are in that region. Think fabs to make the chips, the various motors and related electronics, coating processes, and then sending them off to the places that make HP/Dell/etc systems. If just the drive factory were moved then you'd still have the drive parts crossing the ocean to get to the US and the completed drives crossing once again to go back into the systems. Usually the relocations happen when a disruptive solution comes along. You get the advantage of a somewhat clean sheet. For example SSDs don't need all the mechanical components and fine tuning, so they can be made with a completely different supply chain. It won't happen in hard drives but for other products customization is what can spur relocation. For example Lenovo is opening a manufacturing plant in the US to deal with customization which allows quicker response times. (Customized orders often can charge a little more which offsets increased costs.)

granite 14y ago

I was impressed with the effort to get those drives. If the massive companies who produce hard drives were unprepared for such a natural disaster I can't say I expect more from a small company.

abalone 14y ago

Now that's a startup story. Backblaze has some of the best down home old fashioned spit n glue startup stories on hacker news. Like the one where they invented their own storage pod out of commodity components to beat amazon's pricing. And now this stay-alive-against-all-odds story. Fantastic.

deleted 14y ago

[deleted]