5 ms·

If I were their customer, I would be looking for alternatives right about now...

by yock 14y ago

zorlem 14y ago
Why would you do that? Due to reliability concerns, since they're using 3TB SATA drives, sourced by Cotsco and Best Buy?
- bitJericho 14y ago
  I'd be shocked if they were not the same drives you buy from a supplier.
  - zorlem 14y ago
    I mostly agree with this. That's why I asked about GP's motivation. I suspect, though, without any actual data to back this up, that the drives that go into (low-cost) external HDD enclosures have lower MTBF ratings than the desktop-oriented internal drives. I think that the usage pattern for an external HDD is very different in terms of longevity in operation than desktop hard drives (and these have lower guarantees than eg. server-grade SATA drives). If I was a manufacturer of such enclosures I'd be willing to buy cheaper drives with lower promised MTBF for embedding them in enclosures that stay off most of the time. So, even if they're the same, I guess the manufacturer performs some tests and sells drives that fail some non-critical ones at a lower price (pretty much what Intel and AMD do with their chips - those that can't sustain the highest frequencies are downgraded to clock rates that are stable). edit: The comment from @dsr_ tends to support my hypothesis, although I haven't looked at the differences in guarantees of external enclosures, it's true that internal desktop HDDs can have up to 5 years of guarantee (i.e. WD's Black series has a 5 years guarantee).
    - alexkus 14y ago
      But then Backblaze's use of HDDs is not the same as an HDD stuck in a desktop machine and used as a primary/secondary drive. It's why their restore isn't instantaneous. You have to ask for the files and they'll let you know when they're ready, this takes time (up to several hours). AIUI they essentially fill up each Pod with incoming data and, when it's full it goes into a low power mode where they spin down all of the disks in that Pod. Restore requests (from many users) will usually span quite a few Pods so they batch up requests for each Pod so that can minimise the number of times that drives in a Pod need to be spun up to copy off files. Rather than spinning up and down once every 10 minutes to service 36 requests over 6 hours they wait 6 hours, spin up once and service all 36 requests for that pod in one go. (Figures are for example only.) (Small restores may be instantaneous if the Pod is still being filled or just happens to be spun up for restore soon. Large restores will more than likely span multiple pods and so there's more of a delay until all parts of the restore have been obtained from all of the pods involved.) If anything this maps pretty well to the expected usage pattern of an HDD in an external enclosure as many of them are used for infrequent backups rather than extending constant use storage. It's how I expect Backblaze will be hoping to get much more than the usual MTBF out of their drives they have purchased. HDD guarantees may be time based, but HDD failure rates are linked much more closely with usage patterns; number of spin-ups/downs, number of seeks, number of sector writes, etc. A drive in constant use on a desktop machine is going to be hammered a lot more than one that's essentially written to just once (as the pod fills up) and then spun up every now and again for partial reads.
      - revelation 14y ago
        If thats the access pattern, couldn't they just use tape? Of course you can't easily bootstrap an automated warehouse for these things, but once you have it, you enjoy the drastically lower prices and better availability of tape. (I know nothing about using tapes to seriously store data, so this is conjecture.)
        toomuchtodo 14y ago
        Having a tape libarary/silo is much harder to scale vs building pods of hard drives. Experience: I helped route and process data for the Large Hadron's CMS detector.
  - atYevP 14y ago
    Yev from Backblaze here -> The drives were EXACTLY the same as the ones we would put in to our storage pods prior to the crisis. We selected specific external drives to break open and check because we thought they might be similar, and we were right. The drives ended up being the same, just put in to an external shell. We've always used consumer level products to avoid the "enterprise tax", so reliability more or less stayed constant.
aes256 14y ago
The post struck me as slightly unprofessional, but from a technical standpoint I don't see anything that would change my opinion of the service they provide. They achieve the best bang for buck by using ordinary consumer hard drives in RAID arrays. During the hard drive crisis, the cheapest way of procuring these was by purchasing external drives from ordinary stores. There's nothing wrong with that. If anything, it shows agility and ingenuity.
- yock 14y ago
  And, I would add, intense vulnerability to future market fluctuations. If this was their contingency plan, they've exhausted it. What happens next time?
  - jws 14y ago
    Did you miss the part where they had a month's supply and the option of just paying more on the OEM market? This is just how they made best use of the market by going to nontraditional channels, not their only option.
  - aes256 14y ago
    This was a one-off event that I'm sure the hard drive manufacturers themselves will be taking steps to prevent reoccurring. What else can Backblaze do? Stockpile drives? The hard drive industry moves fast, with major product releases every 6-12 months and constant price reductions; this would almost certainly increase the cost to the end user. No business can adequately prepare for a supply shock like this. Imagine airlines facing a 300% increase in the price of fuel. Almost every single one would go out of business. In this case, would you be berating the ingenious airline that goes out of its way to track down fuel at a cheaper price, finding a low-cost fuel that they can easily refine themselves to aviation fuel, so as to honour their commitments to their customers and avoiding raising their prices?
    - anonymouz 14y ago
      > Imagine airlines facing a 300% increase in the price of fuel. Almost every single one would go out of business. No. They use hedges to control their cost (this is the original point of hedging...). If a company is reliant on a constant supply of hard-drives and doesn't want to be exposed to the risk of extreme price jumps it should probably do the same.
nl 14y ago
Why? They have built a backup architecture designed to use low reliability commodity hard drives. This post shows the lengths they go to just to keep growing. Imagine what they'll do to keep the system running!
pgrote 14y ago
I am a customer of theirs both of their backup service and their plans for machines. The backup service has never let me down and at work we've built two of their machines using their plans saving over 10K. The machine plans are on their site and they explain why they chose off the shelf components. My only regret is that no one on our team has welding skills, so we couldn't make awesome face plates. lol Backblaze is one of the more transparent companies operating in the space, which increased the trust level.
- budmang 14y ago
  Gleb from Backblaze here. Glad you're enjoying the service and the spec design. As for the faceplates, our welding skills aren't great either. However, the company we had cut our steel case for us (www.protocase.com) will cut a custom logo for you into your faceplate as well.
tallanvor 14y ago
Although I am in the process of moving away from Backblaze, it's not because of this. Rather it's because they don't provide the flexibility I've come to need for prioritizing content to backup, and when I recently ran into a problem after swapping out hard drives, the error message Backblaze provided was the most unhelpful piece of drivel I've ever seen. --I was told to contact support with the error code "horse" rather than just being told that it was a timeout issue!
- atYevP 14y ago
  *Yev from Backblaze here -> Yea, that code was very early and was intended for support to gather data from the field as we were collecting data on what was causing the timeout, the message was to prompt folks to write in so they could collect some additional information. We'll probably be changing it in the near future!
mikeash 14y ago
I am a Backblaze customer and this post made me admire their commitment to provide their service even in times of crisis. I won't be switching away, and I don't see why I would.

If I were their customer, I would be looking for alternatives right about now...

by yock 14y ago

If I were their customer, I would be looking for alternatives right about now...

Why would you do that? Due to reliability concerns, since they're using 3TB SATA drives, sourced by Cotsco and Best Buy?

bitJericho 14y ago
I'd be shocked if they were not the same drives you buy from a supplier.
- zorlem 14y ago
  I mostly agree with this. That's why I asked about GP's motivation. I suspect, though, without any actual data to back this up, that the drives that go into (low-cost) external HDD enclosures have lower MTBF ratings than the desktop-oriented internal drives. I think that the usage pattern for an external HDD is very different in terms of longevity in operation than desktop hard drives (and these have lower guarantees than eg. server-grade SATA drives). If I was a manufacturer of such enclosures I'd be willing to buy cheaper drives with lower promised MTBF for embedding them in enclosures that stay off most of the time. So, even if they're the same, I guess the manufacturer performs some tests and sells drives that fail some non-critical ones at a lower price (pretty much what Intel and AMD do with their chips - those that can't sustain the highest frequencies are downgraded to clock rates that are stable). edit: The comment from @dsr_ tends to support my hypothesis, although I haven't looked at the differences in guarantees of external enclosures, it's true that internal desktop HDDs can have up to 5 years of guarantee (i.e. WD's Black series has a 5 years guarantee).
  - alexkus 14y ago
    But then Backblaze's use of HDDs is not the same as an HDD stuck in a desktop machine and used as a primary/secondary drive. It's why their restore isn't instantaneous. You have to ask for the files and they'll let you know when they're ready, this takes time (up to several hours). AIUI they essentially fill up each Pod with incoming data and, when it's full it goes into a low power mode where they spin down all of the disks in that Pod. Restore requests (from many users) will usually span quite a few Pods so they batch up requests for each Pod so that can minimise the number of times that drives in a Pod need to be spun up to copy off files. Rather than spinning up and down once every 10 minutes to service 36 requests over 6 hours they wait 6 hours, spin up once and service all 36 requests for that pod in one go. (Figures are for example only.) (Small restores may be instantaneous if the Pod is still being filled or just happens to be spun up for restore soon. Large restores will more than likely span multiple pods and so there's more of a delay until all parts of the restore have been obtained from all of the pods involved.) If anything this maps pretty well to the expected usage pattern of an HDD in an external enclosure as many of them are used for infrequent backups rather than extending constant use storage. It's how I expect Backblaze will be hoping to get much more than the usual MTBF out of their drives they have purchased. HDD guarantees may be time based, but HDD failure rates are linked much more closely with usage patterns; number of spin-ups/downs, number of seeks, number of sector writes, etc. A drive in constant use on a desktop machine is going to be hammered a lot more than one that's essentially written to just once (as the pod fills up) and then spun up every now and again for partial reads.
    - revelation 14y ago
      If thats the access pattern, couldn't they just use tape? Of course you can't easily bootstrap an automated warehouse for these things, but once you have it, you enjoy the drastically lower prices and better availability of tape. (I know nothing about using tapes to seriously store data, so this is conjecture.)
      - toomuchtodo 14y ago
        Having a tape libarary/silo is much harder to scale vs building pods of hard drives. Experience: I helped route and process data for the Large Hadron's CMS detector.
- atYevP 14y ago
  Yev from Backblaze here -> The drives were EXACTLY the same as the ones we would put in to our storage pods prior to the crisis. We selected specific external drives to break open and check because we thought they might be similar, and we were right. The drives ended up being the same, just put in to an external shell. We've always used consumer level products to avoid the "enterprise tax", so reliability more or less stayed constant.

aes256 14y ago

The post struck me as slightly unprofessional, but from a technical standpoint I don't see anything that would change my opinion of the service they provide. They achieve the best bang for buck by using ordinary consumer hard drives in RAID arrays. During the hard drive crisis, the cheapest way of procuring these was by purchasing external drives from ordinary stores. There's nothing wrong with that. If anything, it shows agility and ingenuity.

yock 14y ago
And, I would add, intense vulnerability to future market fluctuations. If this was their contingency plan, they've exhausted it. What happens next time?
- jws 14y ago
  Did you miss the part where they had a month's supply and the option of just paying more on the OEM market? This is just how they made best use of the market by going to nontraditional channels, not their only option.
- aes256 14y ago
  This was a one-off event that I'm sure the hard drive manufacturers themselves will be taking steps to prevent reoccurring. What else can Backblaze do? Stockpile drives? The hard drive industry moves fast, with major product releases every 6-12 months and constant price reductions; this would almost certainly increase the cost to the end user. No business can adequately prepare for a supply shock like this. Imagine airlines facing a 300% increase in the price of fuel. Almost every single one would go out of business. In this case, would you be berating the ingenious airline that goes out of its way to track down fuel at a cheaper price, finding a low-cost fuel that they can easily refine themselves to aviation fuel, so as to honour their commitments to their customers and avoiding raising their prices?
  - anonymouz 14y ago
    > Imagine airlines facing a 300% increase in the price of fuel. Almost every single one would go out of business. No. They use hedges to control their cost (this is the original point of hedging...). If a company is reliant on a constant supply of hard-drives and doesn't want to be exposed to the risk of extreme price jumps it should probably do the same.

nl 14y ago

Why? They have built a backup architecture designed to use low reliability commodity hard drives. This post shows the lengths they go to just to keep growing. Imagine what they'll do to keep the system running!

pgrote 14y ago

I am a customer of theirs both of their backup service and their plans for machines. The backup service has never let me down and at work we've built two of their machines using their plans saving over 10K. The machine plans are on their site and they explain why they chose off the shelf components. My only regret is that no one on our team has welding skills, so we couldn't make awesome face plates. lol Backblaze is one of the more transparent companies operating in the space, which increased the trust level.

budmang 14y ago
Gleb from Backblaze here. Glad you're enjoying the service and the spec design. As for the faceplates, our welding skills aren't great either. However, the company we had cut our steel case for us (www.protocase.com) will cut a custom logo for you into your faceplate as well.

tallanvor 14y ago

Although I am in the process of moving away from Backblaze, it's not because of this. Rather it's because they don't provide the flexibility I've come to need for prioritizing content to backup, and when I recently ran into a problem after swapping out hard drives, the error message Backblaze provided was the most unhelpful piece of drivel I've ever seen. --I was told to contact support with the error code "horse" rather than just being told that it was a timeout issue!

atYevP 14y ago
*Yev from Backblaze here -> Yea, that code was very early and was intended for support to gather data from the field as we were collecting data on what was causing the timeout, the message was to prompt folks to write in so they could collect some additional information. We'll probably be changing it in the near future!

mikeash 14y ago

I am a Backblaze customer and this post made me admire their commitment to provide their service even in times of crisis. I won't be switching away, and I don't see why I would.