8 ms·

What do you do when a drive goes bad? Do you move ~50TB of data, pull the entire pod out of the rack, and then try to determine which of 45 drives is bad?

X-Istence 12y ago
I don't know what Backblaze does, but if I were building their infrastructure I wouldn't care if a disk goes bad. They are a backup solution, it is a write once, read almost never. It's pure cold storage. 99.99% of their time after they are filled they will be powered down sitting in a rack waiting for someone to come fetch the data. As long as they have stored my data across multiple replicas, then there is no caring if one of the drives goes bad...
- us0r 12y ago
  From another blog post: "Funny thing about providing an online backup service that is currently storing over 100 Petabytes of customer data, sometimes people need their data back. In fact, nearly half (46%) of our customers restored one or more of their files in 2013 and that adds up." https://www.backblaze.com/blog/six-billion-files-restored-and-counting/ https://www.backblaze.com/blog/six-billion-files-restored-an...
  - vidarh 12y ago
    That doesn't contradict the earlier point. 46% of customers restoring one or more of their files is practically nothing; assume that many customers only restores a few files and that most customers do nightly backups, and the vast volume of backed up data is likely never touched. So as was said earlier: As long as there are replicas (and presumably a repair process to maintain a reasonable replica count), a dead disk can sit in the enclosure until a high enough percentage is dead that it's cost effective to pull a a couple of them and consolidate into a new one. Not just because it's awkward to pull them, but data centre visits gets expensive when you factor in the cost of ops people so there are definite tradeoffs in terms of how often it is cost effective to deal with.
    - brianwski 12y ago
      Brian from Backblaze here. > data centre visits gets expensive Early on going to the datacenter was expensive (and time consuming, the datacenter was a 1 hour drive away from the office). We had some success with training "remote hands" (the datacenter provides some basic services like rebooting servers for hourly fees). Now we employ several full time people who go to work in the datacenter (they don't even have desks at the corporate office). It makes our world SO much better and smoother now that there is enough work that they can be there full time.
  - Spooky23 12y ago
    I use Backblaze to backup something like 400,000 files on my Mac. Worst case scenario, I only care about maybe 100,000 files total. The rest is associated with the OS, apps, etc
rythie 12y ago
Why would you need to do any of that? RAID6 can tolerate 2 drive failures and Linux will tell you which drive is bad. Just slide the pod out and replace the drive, no data lost, very little down time.
- sneak 12y ago
  I don't understand why they're using raid6 instead of file-level replication/integrity. They're already running an application over top of it - do the replication there and skip replacing disks...
  - rythie 12y ago
    I don't think they are doing replication at all, don't see it mentioned. RAID6 is cheaper than replicating the files, which would mean twice as many servers.
    - aftbit 12y ago
      But you'll need some level of multi-data-center durability (or at least across racks), so you'll want to replicate user content anyway. Otherwise a dead server could prevent a restore.
      - rythie 12y ago
        I see no mention that they do that, clearly they trying to make this as cheap as possible. If the server is down people can wait for their recovery, many tape based backup systems require you to wait for 30 mins or more to get the backup which covers most outages. Even waiting a days for a recovery isn't the end of the world, especially given very few recoveries are being done and big recoveries require sending of a drive that takes days anyway. The worst case of course is that actually lose your data, probably as the result of a data center fire/explosion, though people should, in most cases, still have the primary copy of the data on their machines. However re-backing this up would take a long time.
    - vidarh 12y ago
atYevP 12y ago
Yev from Backblaze here -> If a disk goes bad we put the pod in a read-only mode, which limits the stress on the pod, and allows restores to continue. We then swap the drive, usually fairly quickly, but because we designed the system around failure, it's usually not critical, though we keep an eye on each one.
bluedino 12y ago
In their 2.0 blog post, they said they had a guy spending 1 day a week replacing approximately 10 drives. I would imagine that consists of identifying the drives and pods, issuing software commands to power them down, physically replacing the drive(s), then re-assembling and powering the unit back up and then bringing the storage software back online and issuing a rebuild command. They only had 9,000 drives back then.
- atYevP 12y ago
  Yev from Backblaze -> We've staffed up ;-)
  - acmecorps 12y ago
    Hi there Yev, I'm a long time crashplan user and the reason why is because they have a family plan ($15 for 10machines). I have multiple machines that I need backed up, so that's why I go with this. I wished i can use backblaze because it looks very good and have native apps. If you have a family plan, I'll switch in a heartbeat. :)
    - brianwski 12y ago
      Brian from Backblaze here. CrashPlan is awesome, I have ALWAYS liked those guys. Backblaze focuses on being really simple, we already have 3 different pricing choices which is probably two more than we should have: 1) $5/month, 2) $50/year, or 3) $95/two years. All are unlimited, none have throttles or caps, we don't even price businesses differently. Simple. Our theory is (may not be correct) that if you offer customers too many complicated choices, they get stuck in the decision making process and don't buy any of the choices. If we offer a family plan that is MORE expensive for two computers but LESS expensive for three computers, blah blah, too much thought. It's $5/computer/month - done. You can get started IMMEDIATELY with one computer, and add them as you go, you don't have to decide to commit to 10 licenses to get the best price, etc, etc. We don't have any magic knowledge more than anybody, so internally we sometimes angst over stuff like a family plan, but for now we just want it to be simple and straight-forward.
keypusher 12y ago
No, you just replace the drive. I'm am sure they have automated means to notify them of exactly which drive has failed and this is a routine occurrence.

X-Istence 12y ago

I don't know what Backblaze does, but if I were building their infrastructure I wouldn't care if a disk goes bad. They are a backup solution, it is a write once, read almost never. It's pure cold storage. 99.99% of their time after they are filled they will be powered down sitting in a rack waiting for someone to come fetch the data. As long as they have stored my data across multiple replicas, then there is no caring if one of the drives goes bad...

us0r 12y ago
From another blog post: "Funny thing about providing an online backup service that is currently storing over 100 Petabytes of customer data, sometimes people need their data back. In fact, nearly half (46%) of our customers restored one or more of their files in 2013 and that adds up." https://www.backblaze.com/blog/six-billion-files-restored-and-counting/ https://www.backblaze.com/blog/six-billion-files-restored-an...
- vidarh 12y ago
  That doesn't contradict the earlier point. 46% of customers restoring one or more of their files is practically nothing; assume that many customers only restores a few files and that most customers do nightly backups, and the vast volume of backed up data is likely never touched. So as was said earlier: As long as there are replicas (and presumably a repair process to maintain a reasonable replica count), a dead disk can sit in the enclosure until a high enough percentage is dead that it's cost effective to pull a a couple of them and consolidate into a new one. Not just because it's awkward to pull them, but data centre visits gets expensive when you factor in the cost of ops people so there are definite tradeoffs in terms of how often it is cost effective to deal with.
  - brianwski 12y ago
    Brian from Backblaze here. > data centre visits gets expensive Early on going to the datacenter was expensive (and time consuming, the datacenter was a 1 hour drive away from the office). We had some success with training "remote hands" (the datacenter provides some basic services like rebooting servers for hourly fees). Now we employ several full time people who go to work in the datacenter (they don't even have desks at the corporate office). It makes our world SO much better and smoother now that there is enough work that they can be there full time.
- Spooky23 12y ago
  I use Backblaze to backup something like 400,000 files on my Mac. Worst case scenario, I only care about maybe 100,000 files total. The rest is associated with the OS, apps, etc

rythie 12y ago

Why would you need to do any of that? RAID6 can tolerate 2 drive failures and Linux will tell you which drive is bad. Just slide the pod out and replace the drive, no data lost, very little down time.

sneak 12y ago
I don't understand why they're using raid6 instead of file-level replication/integrity. They're already running an application over top of it - do the replication there and skip replacing disks...
- rythie 12y ago
  I don't think they are doing replication at all, don't see it mentioned. RAID6 is cheaper than replicating the files, which would mean twice as many servers.
  - aftbit 12y ago
    But you'll need some level of multi-data-center durability (or at least across racks), so you'll want to replicate user content anyway. Otherwise a dead server could prevent a restore.
    - rythie 12y ago
      I see no mention that they do that, clearly they trying to make this as cheap as possible. If the server is down people can wait for their recovery, many tape based backup systems require you to wait for 30 mins or more to get the backup which covers most outages. Even waiting a days for a recovery isn't the end of the world, especially given very few recoveries are being done and big recoveries require sending of a drive that takes days anyway. The worst case of course is that actually lose your data, probably as the result of a data center fire/explosion, though people should, in most cases, still have the primary copy of the data on their machines. However re-backing this up would take a long time.
  - vidarh 12y ago

atYevP 12y ago

Yev from Backblaze here -> If a disk goes bad we put the pod in a read-only mode, which limits the stress on the pod, and allows restores to continue. We then swap the drive, usually fairly quickly, but because we designed the system around failure, it's usually not critical, though we keep an eye on each one.

bluedino 12y ago

In their 2.0 blog post, they said they had a guy spending 1 day a week replacing approximately 10 drives. I would imagine that consists of identifying the drives and pods, issuing software commands to power them down, physically replacing the drive(s), then re-assembling and powering the unit back up and then bringing the storage software back online and issuing a rebuild command. They only had 9,000 drives back then.

atYevP 12y ago
Yev from Backblaze -> We've staffed up ;-)
- acmecorps 12y ago
  Hi there Yev, I'm a long time crashplan user and the reason why is because they have a family plan ($15 for 10machines). I have multiple machines that I need backed up, so that's why I go with this. I wished i can use backblaze because it looks very good and have native apps. If you have a family plan, I'll switch in a heartbeat. :)
  - brianwski 12y ago
    Brian from Backblaze here. CrashPlan is awesome, I have ALWAYS liked those guys. Backblaze focuses on being really simple, we already have 3 different pricing choices which is probably two more than we should have: 1) $5/month, 2) $50/year, or 3) $95/two years. All are unlimited, none have throttles or caps, we don't even price businesses differently. Simple. Our theory is (may not be correct) that if you offer customers too many complicated choices, they get stuck in the decision making process and don't buy any of the choices. If we offer a family plan that is MORE expensive for two computers but LESS expensive for three computers, blah blah, too much thought. It's $5/computer/month - done. You can get started IMMEDIATELY with one computer, and add them as you go, you don't have to decide to commit to 10 licenses to get the best price, etc, etc. We don't have any magic knowledge more than anybody, so internally we sometimes angst over stuff like a family plan, but for now we just want it to be simple and straight-forward.

keypusher 12y ago

No, you just replace the drive. I'm am sure they have automated means to notify them of exactly which drive has failed and this is a routine occurrence.