8 ms·

Storage Pod 4.0: Direct Wire Drives – Faster, Simpler and Less Expensive

170 points by nuriaion 13y ago

kabdib 13y ago
Not sure I'm okay with the single power supply. I guess if you can do redundant storage across pods it'd be okay. But if that single PSU goes under, you're in for a fun couple of hours.
- mike-cardwell 13y ago
  This is better than their previous POD though, which had two PSUs, but no redundancy. I.e, if a single PSU failed, the entire machine would go down as it wouldn't have enough power to run all of the drives. Obviously, for them it is cheaper to live without PSU redundancy because it is a normal day for them to lose disks and PODs because they've got so many of them, and they designed for this scenario.
wilhil 13y ago
I'm so happy they give these plans for free. I have been trying to get involved with Open Compute for some time and just get no where - with these plans, anyone can get a head start. Amazing and I hope they continue for a long time!
pronoiac 13y ago
This is awesome! I like how they document everything. Tomorrow, I'm interviewing at a different place, which is trying to deal with their storage issues, and I was going to mention Storage Pod 3.0. So this is very timely for me!
- Xorlev 13y ago
  Just make sure they know they still need to manage data replication themselves. You plan on entire machines going dark, not just drives.
3327 13y ago
This is great, I might actually build one. I love to see a company emerge that doesn't take the status quo and hacks together a unique solution. This is innovation that I love to see.
kyrra 13y ago
Side story: A company I worked for built a 60 drive unit similar to this. On some of their early prototypes, they could hit power issues when all the drives spun up at once (someone forgot to do their math). So all the development on it had to be done with 3/4 the drives it actually supported. Story #2: A 4U enclosure fully loaded with 60 drives goes beyond the shipping weight limit for Fedex and UPS. You need to either freight ship it or not ship it loaded with drives.
- toomuchtodo 13y ago
  Did their SATA interface not support staggered spinups? Almost all do.
  - kyrra 13y ago
    I believe they did. It may have also done it when all drives were running at max speed... I honestly forget at this point (it's been 7 years or so). It was just the hardware guys screwing up on power distribution and calculated power draw from the drives. Luckily they were prototypes.
nulagrithom 13y ago
"When a 5-drive backplane went down, we had 5 drives go offline. Now if a connector goes down, only 1 drive will go offline." Wouldn't a backplane be more analogous to a SATA card than a single connector? In which case, 40 drives will go down on failure. Neat stuff though. Would be fun to build one.
- davis_m 13y ago
  The former model still had SATA cards that the backplanes connected to. I think the biggest win for them is the fact that the backplanes were one of the more error prone pieces in the box. On a side note, each of the new SATA cards can support up to 40 drives, but there are only 45 drives in the pod, so one failing would presumably take out 22-23 drives instead of 40. Still worse than losing 5 drives though.
  - justincormack 13y ago
    They still had 3 PCI cards before, as well as the multipliers, so card failure is not much different. But port multipliers are all all the same bus, so more error prone.
davis_m 13y ago
I'm sure that the move to direct wire was done purely because of the former backplanes being error prone, but the fact that they realize finding the parts is difficult for others and mentioning it in the blog post shows that the guys at Backblaze care about more than just making money.
- atYevP 13y ago
  Yev from Backblaze -> We do! In fact we make $0 off of the pod design and why we open sourced it. We want other folks to tinker, so we try to stay as open as possible about the design and building of the pods!
  - derekp7 13y ago
    Question: Has any of your design improvements come from the community as a result of being open? And do you end up getting better pricing from your suppliers because there is a bigger market for this design?
    - atYevP 13y ago
      Yes actually! One of the first things that the community came back with after Version 1 was that the hard drives vibrated a bit hard for most use-cases. They recommended we use "hard drive vibration dampening sleeves" (rubber bands) to make the drive fit a bit more snug. It worked like a charm! We don't really get much better pricing, though sometimes we get a small discount because we buy some pods/parts in bulk, though not in extreme quantities.
      - e12e 13y ago
        I really love these technical blog posts (and the designs, of course). Did you ever calculate something like mean-failure rate for pods? Such data might be helpful when evaluating if the architecture is viable for a certain use-case. I'm thinking both disk-loss (which you've blogged about before), but also aggregate failure rates (psu, controller cards etc)? I'm thinking that for small-medium deployments, (with archiving in mind) three pods would be a reasonable minimum, possibly starting them out sparse (single radi6 to each pod, say)? I'm aware there aren't any silver bullets :-)
jobu 13y ago
Looking at the parts list I see the largest cost is the 2x 40-port SATA cards (aside from the drives). I wonder why they didn't decide to reduce the number of drives to 40 so they could just use one SATA card and save $688?
- deleted 13y ago
  [deleted]
  - kgermino 13y ago
    It's worth noting they only put 45 drives in the box so that extra $700 only gets you a 12.5% storage gain. However I suspect that splitting the drives between the two increases read/write performance and reliability as well.
- atYevP 13y ago
  Yev from Backblaze here -> Density is key! Our biggest need is to fit as much data as possible on each pod for as little price. The cost difference between removing 1 SATA card or removing 5 drives is negligible, but the added drives at another 20TB to the pod.
  - dpe82 13y ago
    Why not use a smaller card (or the motherboard's controller) for the last 5 drives?
    - atYevP 13y ago
      We're constantly tweaking the design, that might pop up in the future!
  - darkarmani 13y ago
    If density is your biggest issue, why don't you boot from USB and fit a 46th drive in there using the boot drive? Does that put you over heat or power budgets?
theandrewbailey 13y ago
Reading this, I realized more than ever that while SATA is a good hardware standard, they messed up the naming. All throughout this post, mentions are made to "SATA 3 throughput", which made me think they were referring to the one with 300 MB/s transfers. Then further down, it was made clear that it referred to the third SATA version, AKA SATA 6.0 Gbit/s. At least we don't have to deal with master and slave jumpers anymore.
- mikeash 13y ago
  Could be worse. USB speeds go, in order, Low Speed, Full Speed, High Speed(!), SuperSpeed(??).
  - mkesper 13y ago
    Remember "Ultra High Frequency"? What will USB27.0 be called, "AbnormalSpeed"? ;)
    - radiowave 13y ago
      IIRC that one's going to be called "Ludicrous speed".
dpe82 13y ago
I love all the engineering Backblaze puts into this and their willingness to share their experience. I've noticed Supermicro offers a 45-drive 4u chassis* that costs more than the Storage Pod's raw parts cost, but less than if you buy one preassembled by 45Drives. Does anyone have any experience with Supermicro's solution? * Part: CSE-847E26-RJBOD1 (http://www.supermicro.com/products/chassis/4U/847/SC847E26-RJBOD1.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E26-R...)
- chx 13y ago
  Supermicro is a very popular "white box" server manufacturer. The chassis you linked is much more of a "real" server: hot swap drives, hot swap cooling and redundant PSU. Also, Supermicro even has a 90 (!) drive 4U chassis and for what it does, it's quite cheap.
- gnoway 13y ago
  This looks like a JBOD chassis. Am I missing something?
- budmang 13y ago
  Supermicro makes good systems that are very widely used and we love that companies are continuing to work toward more dense and less costly storage systems. When we started Backblaze in 2007, all the options were astoundingly expensive. One quick note on this particular Supermicro system - it's slightly apples/oranges as the Backblaze Storage Pod is a complete server and this Supermicro system is a JBOD, meaning it still needs to be plugged into a server to work. Gleb, Backblaze co-founder
  - dpe82 13y ago
    Thanks for the reply! Yeah, this is a JBOD of course. I was thinking of their 36 drive server cases, thought one was 45 and didn't look too closely when I found the link I thought I was looking for. :) Now that prices have come down so much on the 3rd party stuff, have you evaluated / considered using any of it?
  - nanch 13y ago
NKCSS 13y ago
Small error in Appendix a; they list 2 zippy psu's (they only use one) and the total also only count them once.
- budmang 13y ago
  Fixed. Thanks!
vicaya 13y ago
Always love the practical hardware designs from these guys. I'm also interested in some software stories: Linux kernel/distro used, issues encountered/solved etc.
andrewmunsell 13y ago
Backblaze is great-- the product is fast and works well. But, there's one big issue that prevents me from using it-- they delete your backups of your external hard drives if they aren't plugged in for 30 days. I have a drive that I may be away from (at least, my Macbook Air is) for weeks at a time, and I don't want to continually lose version history or have to reupload some of the larger files (home videos and my photos). CrashPlan doesn't delete your backups after 30 days (they keep everything as long as you're subscribed), but the upload speed is so horrendously slow I couldn't upload the entire photo library plus videos (~300 GB or so) in the month I was subscribed. I'm on a gigabit network over Ethernet, so it's not my connection that's the issue (Backblaze, on the other hand, uploads this data very quickly). I realize that they're trying to reduce costs by deleting data that doesn't seem to "be in use", but I'm sure a lot of people have "archival" data that they don't look at often, but really value. Instead, I'm currently using Arq 4 (which is fantastic, by the way) with Amazon Glacier for the photos. It may cost something to retrieve them if my drive dies or is physically destroyed, but I'm not planning for that to happen. I have Arq set to use S3 for my other documents and such so I can restore versions whenever I need to, so it's really the best of both worlds.
- stankal 13y ago
  I have to agree that this is one of the biggest downside of Backblaze as I’m right now dealing with recovering data from an external drive that has failed. According to their support staff there’s no way for them to extend that period which makes me super nervous about getting the data and verifying it’s all good before they delete it (I did not notice the drive has failed until I got an email from Backblaze telling me that they will delete the backup in 15 days) Their 30-day retention policy on external drives defeats the purpose of having the backup. The second problem with Backblaze is the recovery process: they decrypt the files on their servers and store unencrypted zip archive for 7 days or until you download it. I would much rather see them create a restore process where encrypted data is downloaded and then decrypted locally. I’m evaluating Arq, and so far it looks very good. Upload to S3/Glacier is super fast.
- ghshephard 13y ago
  Agreed that this was one of the issues I had with Backblaze - I constantly would lose all of my external drive backups when I went on a business trip, come back, and all my backups would be gone and I would have to start over again. I finally ended up just purchasing multiple hard drives and using superduper to clone/backup. Crashplan wasn't an option - would take 60 days to do a 1 Terabyte backup, and I don't leave my laptop connected to the externals that long. I'd be willing to pay more money to backblaze to have them retain my external hard drivers for a longer period of time.
DonGateley 13y ago
What an amazing degree of disclosure. Tim Nufire is an engineer's engineer. He gets me to thinking of how I might be able to justify the cost of one. :-)
ksec 13y ago
One of the thing i immediately noticed were the price /GB over the years. We had literally no decrease since the flood in 2011. So three years later we are still paying for about the same price per GB. But that is finally going to change. As Google lower their price, i see this as HAMR finally coming. With both Seagate and WD promised to get 20TB HDD by 2020, and with a 60TB version after. We have a clear roadmap of what is coming along. Which means Google could now price their storage by it.
km3k 13y ago
I love the look of everything Backblaze does, but I don't use them because I need Linux support. I currently use Crashplan because of their Linux support, but I find the service mediocre overall.

170 points by nuriaion 13y ago

Not sure I'm okay with the single power supply. I guess if you can do redundant storage across pods it'd be okay. But if that single PSU goes under, you're in for a fun couple of hours.

mike-cardwell 13y ago
This is better than their previous POD though, which had two PSUs, but no redundancy. I.e, if a single PSU failed, the entire machine would go down as it wouldn't have enough power to run all of the drives. Obviously, for them it is cheaper to live without PSU redundancy because it is a normal day for them to lose disks and PODs because they've got so many of them, and they designed for this scenario.

wilhil 13y ago

I'm so happy they give these plans for free. I have been trying to get involved with Open Compute for some time and just get no where - with these plans, anyone can get a head start. Amazing and I hope they continue for a long time!

pronoiac 13y ago

This is awesome! I like how they document everything. Tomorrow, I'm interviewing at a different place, which is trying to deal with their storage issues, and I was going to mention Storage Pod 3.0. So this is very timely for me!

Xorlev 13y ago
Just make sure they know they still need to manage data replication themselves. You plan on entire machines going dark, not just drives.

3327 13y ago

This is great, I might actually build one. I love to see a company emerge that doesn't take the status quo and hacks together a unique solution. This is innovation that I love to see.

kyrra 13y ago

Side story: A company I worked for built a 60 drive unit similar to this. On some of their early prototypes, they could hit power issues when all the drives spun up at once (someone forgot to do their math). So all the development on it had to be done with 3/4 the drives it actually supported. Story #2: A 4U enclosure fully loaded with 60 drives goes beyond the shipping weight limit for Fedex and UPS. You need to either freight ship it or not ship it loaded with drives.

toomuchtodo 13y ago
Did their SATA interface not support staggered spinups? Almost all do.
- kyrra 13y ago
  I believe they did. It may have also done it when all drives were running at max speed... I honestly forget at this point (it's been 7 years or so). It was just the hardware guys screwing up on power distribution and calculated power draw from the drives. Luckily they were prototypes.

nulagrithom 13y ago

"When a 5-drive backplane went down, we had 5 drives go offline. Now if a connector goes down, only 1 drive will go offline." Wouldn't a backplane be more analogous to a SATA card than a single connector? In which case, 40 drives will go down on failure. Neat stuff though. Would be fun to build one.

davis_m 13y ago
The former model still had SATA cards that the backplanes connected to. I think the biggest win for them is the fact that the backplanes were one of the more error prone pieces in the box. On a side note, each of the new SATA cards can support up to 40 drives, but there are only 45 drives in the pod, so one failing would presumably take out 22-23 drives instead of 40. Still worse than losing 5 drives though.
- justincormack 13y ago
  They still had 3 PCI cards before, as well as the multipliers, so card failure is not much different. But port multipliers are all all the same bus, so more error prone.

davis_m 13y ago

I'm sure that the move to direct wire was done purely because of the former backplanes being error prone, but the fact that they realize finding the parts is difficult for others and mentioning it in the blog post shows that the guys at Backblaze care about more than just making money.

atYevP 13y ago
Yev from Backblaze -> We do! In fact we make $0 off of the pod design and why we open sourced it. We want other folks to tinker, so we try to stay as open as possible about the design and building of the pods!
- derekp7 13y ago
  Question: Has any of your design improvements come from the community as a result of being open? And do you end up getting better pricing from your suppliers because there is a bigger market for this design?
  - atYevP 13y ago
    Yes actually! One of the first things that the community came back with after Version 1 was that the hard drives vibrated a bit hard for most use-cases. They recommended we use "hard drive vibration dampening sleeves" (rubber bands) to make the drive fit a bit more snug. It worked like a charm! We don't really get much better pricing, though sometimes we get a small discount because we buy some pods/parts in bulk, though not in extreme quantities.
    - e12e 13y ago
      I really love these technical blog posts (and the designs, of course). Did you ever calculate something like mean-failure rate for pods? Such data might be helpful when evaluating if the architecture is viable for a certain use-case. I'm thinking both disk-loss (which you've blogged about before), but also aggregate failure rates (psu, controller cards etc)? I'm thinking that for small-medium deployments, (with archiving in mind) three pods would be a reasonable minimum, possibly starting them out sparse (single radi6 to each pod, say)? I'm aware there aren't any silver bullets :-)

jobu 13y ago

Looking at the parts list I see the largest cost is the 2x 40-port SATA cards (aside from the drives). I wonder why they didn't decide to reduce the number of drives to 40 so they could just use one SATA card and save $688?

deleted 13y ago
[deleted]
- kgermino 13y ago
  It's worth noting they only put 45 drives in the box so that extra $700 only gets you a 12.5% storage gain. However I suspect that splitting the drives between the two increases read/write performance and reliability as well.
atYevP 13y ago
Yev from Backblaze here -> Density is key! Our biggest need is to fit as much data as possible on each pod for as little price. The cost difference between removing 1 SATA card or removing 5 drives is negligible, but the added drives at another 20TB to the pod.
- dpe82 13y ago
  Why not use a smaller card (or the motherboard's controller) for the last 5 drives?
  - atYevP 13y ago
    We're constantly tweaking the design, that might pop up in the future!
- darkarmani 13y ago
  If density is your biggest issue, why don't you boot from USB and fit a 46th drive in there using the boot drive? Does that put you over heat or power budgets?

theandrewbailey 13y ago

Reading this, I realized more than ever that while SATA is a good hardware standard, they messed up the naming. All throughout this post, mentions are made to "SATA 3 throughput", which made me think they were referring to the one with 300 MB/s transfers. Then further down, it was made clear that it referred to the third SATA version, AKA SATA 6.0 Gbit/s. At least we don't have to deal with master and slave jumpers anymore.

mikeash 13y ago
Could be worse. USB speeds go, in order, Low Speed, Full Speed, High Speed(!), SuperSpeed(??).
- mkesper 13y ago
  Remember "Ultra High Frequency"? What will USB27.0 be called, "AbnormalSpeed"? ;)
  - radiowave 13y ago
    IIRC that one's going to be called "Ludicrous speed".

dpe82 13y ago

I love all the engineering Backblaze puts into this and their willingness to share their experience. I've noticed Supermicro offers a 45-drive 4u chassis* that costs more than the Storage Pod's raw parts cost, but less than if you buy one preassembled by 45Drives. Does anyone have any experience with Supermicro's solution? * Part: CSE-847E26-RJBOD1 (http://www.supermicro.com/products/chassis/4U/847/SC847E26-RJBOD1.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E26-R...)

chx 13y ago
Supermicro is a very popular "white box" server manufacturer. The chassis you linked is much more of a "real" server: hot swap drives, hot swap cooling and redundant PSU. Also, Supermicro even has a 90 (!) drive 4U chassis and for what it does, it's quite cheap.
gnoway 13y ago
This looks like a JBOD chassis. Am I missing something?
budmang 13y ago
Supermicro makes good systems that are very widely used and we love that companies are continuing to work toward more dense and less costly storage systems. When we started Backblaze in 2007, all the options were astoundingly expensive. One quick note on this particular Supermicro system - it's slightly apples/oranges as the Backblaze Storage Pod is a complete server and this Supermicro system is a JBOD, meaning it still needs to be plugged into a server to work. Gleb, Backblaze co-founder
- dpe82 13y ago
  Thanks for the reply! Yeah, this is a JBOD of course. I was thinking of their 36 drive server cases, thought one was 45 and didn't look too closely when I found the link I thought I was looking for. :) Now that prices have come down so much on the 3rd party stuff, have you evaluated / considered using any of it?
- nanch 13y ago

NKCSS 13y ago

Small error in Appendix a; they list 2 zippy psu's (they only use one) and the total also only count them once.

budmang 13y ago
Fixed. Thanks!

vicaya 13y ago

Always love the practical hardware designs from these guys. I'm also interested in some software stories: Linux kernel/distro used, issues encountered/solved etc.

andrewmunsell 13y ago

Backblaze is great-- the product is fast and works well. But, there's one big issue that prevents me from using it-- they delete your backups of your external hard drives if they aren't plugged in for 30 days. I have a drive that I may be away from (at least, my Macbook Air is) for weeks at a time, and I don't want to continually lose version history or have to reupload some of the larger files (home videos and my photos). CrashPlan doesn't delete your backups after 30 days (they keep everything as long as you're subscribed), but the upload speed is so horrendously slow I couldn't upload the entire photo library plus videos (~300 GB or so) in the month I was subscribed. I'm on a gigabit network over Ethernet, so it's not my connection that's the issue (Backblaze, on the other hand, uploads this data very quickly). I realize that they're trying to reduce costs by deleting data that doesn't seem to "be in use", but I'm sure a lot of people have "archival" data that they don't look at often, but really value. Instead, I'm currently using Arq 4 (which is fantastic, by the way) with Amazon Glacier for the photos. It may cost something to retrieve them if my drive dies or is physically destroyed, but I'm not planning for that to happen. I have Arq set to use S3 for my other documents and such so I can restore versions whenever I need to, so it's really the best of both worlds.

stankal 13y ago
I have to agree that this is one of the biggest downside of Backblaze as I’m right now dealing with recovering data from an external drive that has failed. According to their support staff there’s no way for them to extend that period which makes me super nervous about getting the data and verifying it’s all good before they delete it (I did not notice the drive has failed until I got an email from Backblaze telling me that they will delete the backup in 15 days) Their 30-day retention policy on external drives defeats the purpose of having the backup. The second problem with Backblaze is the recovery process: they decrypt the files on their servers and store unencrypted zip archive for 7 days or until you download it. I would much rather see them create a restore process where encrypted data is downloaded and then decrypted locally. I’m evaluating Arq, and so far it looks very good. Upload to S3/Glacier is super fast.
ghshephard 13y ago
Agreed that this was one of the issues I had with Backblaze - I constantly would lose all of my external drive backups when I went on a business trip, come back, and all my backups would be gone and I would have to start over again. I finally ended up just purchasing multiple hard drives and using superduper to clone/backup. Crashplan wasn't an option - would take 60 days to do a 1 Terabyte backup, and I don't leave my laptop connected to the externals that long. I'd be willing to pay more money to backblaze to have them retain my external hard drivers for a longer period of time.

DonGateley 13y ago

What an amazing degree of disclosure. Tim Nufire is an engineer's engineer. He gets me to thinking of how I might be able to justify the cost of one. :-)

ksec 13y ago

One of the thing i immediately noticed were the price /GB over the years. We had literally no decrease since the flood in 2011. So three years later we are still paying for about the same price per GB. But that is finally going to change. As Google lower their price, i see this as HAMR finally coming. With both Seagate and WD promised to get 20TB HDD by 2020, and with a 60TB version after. We have a clear roadmap of what is coming along. Which means Google could now price their storage by it.

km3k 13y ago

I love the look of everything Backblaze does, but I don't use them because I need Linux support. I currently use Crashplan because of their Linux support, but I find the service mediocre overall.