5 ms·

Hey Yev or Brian, First I think it is awesome that you guys come on HN to respond when backblaze makes the news. Thank you. If you bought a prebuilt storage p

by daveguy 11y ago

Hey Yev or Brian, First I think it is awesome that you guys come on HN to respond when backblaze makes the news. Thank you. If you bought a prebuilt storage pod (from somewhere like http://www.backuppods.com/ http://www.backuppods.com/) could you start out at less than capacity and add more as you go? Say you started with one drive -- would it need to go in a specific location? When you add more would there be a significant reconfiguration process? Is it possible to hot-expand? I guess location and config has a lot to do with software involved. Has there been a blog on the software side of managing that much storage? Can it be as simple as a spanning partition to treat all drives as a single drive? Also, are you tracking the cost of SSD drives? Do you expect cost parity soon? It seems that capacity parity is getting close, but cost parity (especially at that capacity) is farther off. Do you have an estimate of when it might be or is it "far off, check again in 5 years". I'm sure you all found the ssd endurance test at techreport interesting (http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead http://techreport.com/review/27909/the-ssd-endurance-experim...).

atYevP 11y ago
Yev here! It's an interesting question. We use the pods in a larger "Vault" configuration, so the way we use them is a bit different (basically a set of 20 pods are connected -> https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect...). But, if you're buying a single pod, you could configure the software however you want, though you'd likely need the first hard drive to go on to the backplane that gets read first. We use different software so ours expects there to be 60 drives, but there's nothing stopping you from configuring Debian or whatnot to read a specific drive first. We ARE tracking SSD costs but thus far it's not quite at the point where the Cost per GB is in our favor. We'd love to play with them once it gets close, it would neat to build an SSD pod and see how it stacks up!
brianwski 11y ago
Yes, and we do this internally in our QA and load testing lab. It's an expensive waste to put $200,000 worth of drives in a 20 pod vault when all you need is 1/60th of that! > are you tracking the cost of SSD drives? Do you expect cost parity soon? Yes we track SSDs, and this is a hotly contested topic inside Backblaze. Personally I'm predicting they "cross over" in cost effectiveness within 2 years. But I don't have any more insight than you do, and many inside Backblaze disagree with me. When I say "cost effective" that is total cost of ownership, which includes paying for less electricity to power the SSD, possibly the higher density of SSDs, and whatever failure rates of the SSDs forcing us to purchase more (might be better or worse than hard drives). So whatever our little spread sheet kicks out as cheaper -> that's the one we will purchase! One random nice thing about SSDs is there will be less case vibration, and the drives shouldn't be affected AT ALL by the small amount of fan vibration.
- DanielDent 11y ago
  Am I correct in assuming that the performance improvements of SSDs are worthless for your use case? It seems like the workload of Backblaze (like many "big data" workloads) is such that the extra IOPS don't actually help. With thousands of drive spindles, and a very small portion of the data on on the spindles being needed at any given moment, I don't imagine you are actually limited by the performance of spinning rust.
  - brianwski 11y ago
    > the performance improvements of SSDs are worthless No way! We are disk I/O limited right now. We think we can approximately double our performance simply by using SSDs instead of traditional 7200 RPM drives. I'm advocating for building an "SSD Pod" built with 1 or 2 TByte SSDs. > and a very small portion of the data on on the spindles being needed at any given moment With our original Personal Backup product that was true. Everybody was happy with waiting a few hours for their restores to complete. But with our new product line of B2 (competes with Amazon S3) then suddenly performance can matter much more, because programmers using B2 might be implementing all sorts of different access patterns. If those programmers are using B2 to implement online backup, then yes, still fine. But if the programmers are building hacker news or reddit or Dropbox the access patterns might require faster IOPS to be more responsive.
    - DanielDent 11y ago
      I guess the B2 product is growing quickly or is where you are looking to drive growth? Or are you using different storage pools for different products? It's surprising to hear this. I would have thought that your backup workloads would leave enough unused IOPS that you wouldn't be anywhere close to hitting a performance ceiling. What metrics do you optimize towards? Tail latency?
teraflop 11y ago
> Can it be as simple as a spanning partition to treat all drives as a single drive? Sure, but there are tradeoffs. One option is to just concatenate all the drives together, either with mdadm (grouping the drives into a RAID-0 array) or with LVM (adding them all to a single volume group). This lets you add drives very easily and efficiently, but has no redundancy. When (not if) one drive fails, your entire FS is toast. If you care about your data, then you'll want to use RAID-6 for redundancy. But adding a drive to a RAID-6 array is very time-consuming, because all of your data has to be re-shuffled onto a new drive. And you don't want to do this without a backup -- although the reshape process is designed to be safely interruptible, it's not bug-free. (I speak from experience, having nearly lost a few TB this way not too long ago.) You can compromise by grouping your drives into multiple RAID-6 arrays. For example, instead of using 58 data drives + 2 parity drives, you could use 6 arrays of (8 data + 2 parity). This gives you much less usable space (80% as opposed to 97%) but it allows you to add 10 drives at a time without having to re-shuffle all your data. And it means you can recover from a drive failure much more quickly. Of course, you can also just treat the drives as separate block devices, and provide a single redundant filesystem at a higher level, using something like Ceph. I would assume Backblaze is doing something along these lines.
- zzzcpan 11y ago
  > I would assume Backblaze is doing something along these lines. They are using a regular filesystem on each drive and implement redundancy on top of regular files. It's more like Swift [1] than Ceph. Which is something parent should consider too, if he's going to deal with that much data. [1] http://docs.openstack.org/developer/swift/overview_architecture.html http://docs.openstack.org/developer/swift/overview_architect...
  - bwilliams18 11y ago
    Has anyone built a Swift implementation in Swift?
    - AdamJacobMuller 11y ago
      Someone should if they haven't already. But seriously I do hate namespace collisions like this. Swift (the storage architecture) is something i've been a fan of for years, swift (the programming language) i've dabbled in a bit and its interesting for some use cases for sure (aside from iOS programming which I'm not interested in at all). The cognitive dissonance of switching back-and-forth is annoying. I do hope nobody ever writes or needs to use a swift API connector in swift, that would be painful.

by daveguy 11y ago

atYevP 11y ago

Yev here! It's an interesting question. We use the pods in a larger "Vault" configuration, so the way we use them is a bit different (basically a set of 20 pods are connected -> https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect...). But, if you're buying a single pod, you could configure the software however you want, though you'd likely need the first hard drive to go on to the backplane that gets read first. We use different software so ours expects there to be 60 drives, but there's nothing stopping you from configuring Debian or whatnot to read a specific drive first. We ARE tracking SSD costs but thus far it's not quite at the point where the Cost per GB is in our favor. We'd love to play with them once it gets close, it would neat to build an SSD pod and see how it stacks up!

brianwski 11y ago

Yes, and we do this internally in our QA and load testing lab. It's an expensive waste to put $200,000 worth of drives in a 20 pod vault when all you need is 1/60th of that! > are you tracking the cost of SSD drives? Do you expect cost parity soon? Yes we track SSDs, and this is a hotly contested topic inside Backblaze. Personally I'm predicting they "cross over" in cost effectiveness within 2 years. But I don't have any more insight than you do, and many inside Backblaze disagree with me. When I say "cost effective" that is total cost of ownership, which includes paying for less electricity to power the SSD, possibly the higher density of SSDs, and whatever failure rates of the SSDs forcing us to purchase more (might be better or worse than hard drives). So whatever our little spread sheet kicks out as cheaper -> that's the one we will purchase! One random nice thing about SSDs is there will be less case vibration, and the drives shouldn't be affected AT ALL by the small amount of fan vibration.

DanielDent 11y ago
Am I correct in assuming that the performance improvements of SSDs are worthless for your use case? It seems like the workload of Backblaze (like many "big data" workloads) is such that the extra IOPS don't actually help. With thousands of drive spindles, and a very small portion of the data on on the spindles being needed at any given moment, I don't imagine you are actually limited by the performance of spinning rust.
- brianwski 11y ago
  > the performance improvements of SSDs are worthless No way! We are disk I/O limited right now. We think we can approximately double our performance simply by using SSDs instead of traditional 7200 RPM drives. I'm advocating for building an "SSD Pod" built with 1 or 2 TByte SSDs. > and a very small portion of the data on on the spindles being needed at any given moment With our original Personal Backup product that was true. Everybody was happy with waiting a few hours for their restores to complete. But with our new product line of B2 (competes with Amazon S3) then suddenly performance can matter much more, because programmers using B2 might be implementing all sorts of different access patterns. If those programmers are using B2 to implement online backup, then yes, still fine. But if the programmers are building hacker news or reddit or Dropbox the access patterns might require faster IOPS to be more responsive.
  - DanielDent 11y ago
    I guess the B2 product is growing quickly or is where you are looking to drive growth? Or are you using different storage pools for different products? It's surprising to hear this. I would have thought that your backup workloads would leave enough unused IOPS that you wouldn't be anywhere close to hitting a performance ceiling. What metrics do you optimize towards? Tail latency?

teraflop 11y ago

> Can it be as simple as a spanning partition to treat all drives as a single drive? Sure, but there are tradeoffs. One option is to just concatenate all the drives together, either with mdadm (grouping the drives into a RAID-0 array) or with LVM (adding them all to a single volume group). This lets you add drives very easily and efficiently, but has no redundancy. When (not if) one drive fails, your entire FS is toast. If you care about your data, then you'll want to use RAID-6 for redundancy. But adding a drive to a RAID-6 array is very time-consuming, because all of your data has to be re-shuffled onto a new drive. And you don't want to do this without a backup -- although the reshape process is designed to be safely interruptible, it's not bug-free. (I speak from experience, having nearly lost a few TB this way not too long ago.) You can compromise by grouping your drives into multiple RAID-6 arrays. For example, instead of using 58 data drives + 2 parity drives, you could use 6 arrays of (8 data + 2 parity). This gives you much less usable space (80% as opposed to 97%) but it allows you to add 10 drives at a time without having to re-shuffle all your data. And it means you can recover from a drive failure much more quickly. Of course, you can also just treat the drives as separate block devices, and provide a single redundant filesystem at a higher level, using something like Ceph. I would assume Backblaze is doing something along these lines.

zzzcpan 11y ago
> I would assume Backblaze is doing something along these lines. They are using a regular filesystem on each drive and implement redundancy on top of regular files. It's more like Swift [1] than Ceph. Which is something parent should consider too, if he's going to deal with that much data. [1] http://docs.openstack.org/developer/swift/overview_architecture.html http://docs.openstack.org/developer/swift/overview_architect...
- bwilliams18 11y ago
  Has anyone built a Swift implementation in Swift?
  - AdamJacobMuller 11y ago
    Someone should if they haven't already. But seriously I do hate namespace collisions like this. Swift (the storage architecture) is something i've been a fan of for years, swift (the programming language) i've dabbled in a bit and its interesting for some use cases for sure (aside from iOS programming which I'm not interested in at all). The cognitive dissonance of switching back-and-forth is annoying. I do hope nobody ever writes or needs to use a swift API connector in swift, that would be painful.