10 ms·

Storage Pod 6.0: Building a 60 Drive 480TB Storage Server

181 points by geerlingguy 11y ago

lunixbochs 11y ago
One note - looks like you can get 8TB WD Helium drives for way cheaper ($250 vs $340) right now by shucking externals: http://www.amazon.com/Book-Desktop-External-Drive-WDBFJK0080HBK-NESN/dp/B01B6BN0Q2 http://www.amazon.com/Book-Desktop-External-Drive-WDBFJK0080...
- Scramblejams 11y ago
  Thanks for the tip. Though Backblaze has spoiled us with so much good reliability data, I now feel completely paralyzed in the face of buying a new drive if Backblaze hasn't told me how much I can trust it!
  - atYevP 11y ago
    Yev from Backblaze here -> You can trust it. As long as you're OK with it dying and have other backups in place ;-)
    - Scramblejams 11y ago
      Ha! Well, even if I have backups, I don't like buying replacements, so there's that...
- Keyframe 11y ago
  I think that's a WD Green inside, not Red.
  - lunixbochs 11y ago
    It's still a non-SMR 8TB drive, and pretty close to the Seagate archival SMR price.
daveguy 11y ago
Hey Yev or Brian, First I think it is awesome that you guys come on HN to respond when backblaze makes the news. Thank you. If you bought a prebuilt storage pod (from somewhere like http://www.backuppods.com/ http://www.backuppods.com/) could you start out at less than capacity and add more as you go? Say you started with one drive -- would it need to go in a specific location? When you add more would there be a significant reconfiguration process? Is it possible to hot-expand? I guess location and config has a lot to do with software involved. Has there been a blog on the software side of managing that much storage? Can it be as simple as a spanning partition to treat all drives as a single drive? Also, are you tracking the cost of SSD drives? Do you expect cost parity soon? It seems that capacity parity is getting close, but cost parity (especially at that capacity) is farther off. Do you have an estimate of when it might be or is it "far off, check again in 5 years". I'm sure you all found the ssd endurance test at techreport interesting (http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead http://techreport.com/review/27909/the-ssd-endurance-experim...).
- atYevP 11y ago
  Yev here! It's an interesting question. We use the pods in a larger "Vault" configuration, so the way we use them is a bit different (basically a set of 20 pods are connected -> https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect...). But, if you're buying a single pod, you could configure the software however you want, though you'd likely need the first hard drive to go on to the backplane that gets read first. We use different software so ours expects there to be 60 drives, but there's nothing stopping you from configuring Debian or whatnot to read a specific drive first. We ARE tracking SSD costs but thus far it's not quite at the point where the Cost per GB is in our favor. We'd love to play with them once it gets close, it would neat to build an SSD pod and see how it stacks up!
- brianwski 11y ago
  Yes, and we do this internally in our QA and load testing lab. It's an expensive waste to put $200,000 worth of drives in a 20 pod vault when all you need is 1/60th of that! > are you tracking the cost of SSD drives? Do you expect cost parity soon? Yes we track SSDs, and this is a hotly contested topic inside Backblaze. Personally I'm predicting they "cross over" in cost effectiveness within 2 years. But I don't have any more insight than you do, and many inside Backblaze disagree with me. When I say "cost effective" that is total cost of ownership, which includes paying for less electricity to power the SSD, possibly the higher density of SSDs, and whatever failure rates of the SSDs forcing us to purchase more (might be better or worse than hard drives). So whatever our little spread sheet kicks out as cheaper -> that's the one we will purchase! One random nice thing about SSDs is there will be less case vibration, and the drives shouldn't be affected AT ALL by the small amount of fan vibration.
scurvy 11y ago
Any reason on 60 drives instead of 70 or more? At that depth, there are several OEM designs that with that many drives (70). No 25/40gig networking? It's nice to point out the depth issue, as older racks are shallower than the newer versions. This blocks things like the 0U PDU channels and cable management channels. That said, the newer, deeper (and more popular) racks fit everything just fine. How tall are your racks? How many pods are you getting in there? 3ph 50A power?
- brianwski 11y ago
  > Any reason on 60 drives instead of 70 or more? One step at a time. :-) We think we can put a few more drives in the motherboard half of the POD without many other changes, and connect them to already existing SATA connectors on the motherboard. To get all the way to 75 drives we will need another row of port multipliers, which then means we need even more SATA cards, etc. > No 25/40gig networking? We're currently not able to saturate the 10 Gbit so for us it is wasted money to go faster. Remember that we run these in "vaults of 20" so we have 20 pods EACH with 10 Gbits so the vault hoovers in data at 200 Gbits/sec if you can thread the application. > How tall are your racks? Most of our racks are 45U tall where the PDU takes 4U, so we can fit (not necessarily power) 10 pods per rack and still fit a network switch in the 1U left at the top. In the past (with 45 drive pods) we ran 2 circuits of 30A 208V power (three phase). The two circuits are redundant - our power strips can flip over from one to the other if there is a loss of power on the currently used power. But I think they only put 8 or 9 of the 45 drive pods in a cabinet because it slightly exceeds what the datacenter likes. And now that we're moving up to all 60 drive pods, we have to rejigger the whole thing. It is also worth mentioning that when we bring a 20 pod vault online, we put each of the 20 pods in different racks in different parts of the datacenter. This helps keep the vault completely online if any one component fails. For example, if the power strip feeding the pods in 1 rack power dies, it only takes 1 pod offline out of several different vaults, so ALL the vaults stay online both reading and writing data. (I hope that made sense.)
  - scurvy 11y ago
    Makes sense. Thanks for the reply! As to spreading the vaults around the DC, it sounds like that's more to prepare for ToR switch failure -- its sounds like an ATS handles your PDU and power circuit failover. Yeah it sounds like you're mostly CPU bound calculating EC if you're going to rebalance and replace failed pods so 25 gbps ethernet probably isn't interesting (yet). That said, it's only marginally more expensive than 10gbps twinax. Interesting that you're running 10gig baseT. Interesting from a power and latency perspective, though I'd imagine you're not worried too much about latency.
Scarbutt 11y ago
Dropbox manage to do 1PB per server, why doesn't backblaze do it?
- bryanlarsen 11y ago
  I believe that DropBox is using 12TB drives, which aren't available for general sale, yet. Backblaze is using 4TB drives, which is the current $/TB leader.
  - wmf 11y ago
    It looks like 8TB SMR would be cheaper than 4TB. I don't mean to be negative, but Backblaze is starting to look like yesterday's technology today.
    - hackcasual 11y ago
      But if that 8TB drive has a higher failure rate, that needs to be taken into consideration as well.
    - rtkwe 11y ago
      It is but not by much. They actually provide the $/GB for a Seagate 8TB Archival SMR hdd in their SMR section. It comes to .034 $/GB vs .036 $/GB for the 4TB drives they're currently using. They've done the math it looks like and decided that the small gain doesn't make up for the time lost on overwrites.
- venomsnake 11y ago
  Backblaze are/were bootstrapped. They don't have the money to burn to move outside of the sweetspot $/GB - so their logic starts with which size drive is expected to be cheapest per GB in the next 12 months. I remember some of their old posts explaining why they made the jump from 2 to 3tb drives - they are slightly more expensive today, but we expect them to go down in price soon.
  - atYevP 11y ago
bluedino 11y ago
Any idea what software is used to make the Build Book? https://f001.backblaze.com/file/Backblaze_Blog/Storage-Pod-6/Build+Book+Backblaze+60+Drive.pdf https://f001.backblaze.com/file/Backblaze_Blog/Storage-Pod-6...
- jeffmcjunkin 11y ago
  exiftool says it's PowerPoint, surprisingly: $ exiftool Build+Book+Backblaze+60+Drive.pdf | tail -n5 | head -n3 Title : 7400-0283 Rev A OMS Assy Backblaze 60 Drive Producer : Mac OS X 10.11.3 Quartz PDFContext Creator : PowerPoint
- atYevP 11y ago
  Yev from Backblaze here -> You're not going to believe this, but the folks at Evolve used Powerpoint and a template...seriously.
  - bwilliams18 11y ago
    You would be astounded to learn how much use MS Office still gets today. Excel...that thing runs the world.
majke 11y ago
I've never heard about SMR before. I wonder - does it mean that a write to one area must be done in multiple disk rotations? Ie: if a sequential write to one file be aggregated into a single location on the disk or will it be spread around the whole plate. Just wondering how a software utilizing this should behave. http://www.storagereview.com/what_is_shingled_magnetic_recording_smr http://www.storagereview.com/what_is_shingled_magnetic_recor... https://en.wikipedia.org/wiki/Shingled_magnetic_recording https://en.wikipedia.org/wiki/Shingled_magnetic_recording
- codemac 11y ago
  Heads up - if you've bought a high density disk recently, it's probably SMR internally anyways. A write to one area can be done in a single disc rotation, as long as it's sequential. SMR disks added what are called ZBC, Zoned Block Commands to the SCSI spec. The disk itself exposes the ability to interact with zones several ways: - REPORT ZONES - What zones there are, how many, what type, and state of the zone. - OPEN / CLOSE / FINISH ZONE - zone write state management So essentially, it's like a drive exposes ~8 files, opened with O_APPEND, and you have to manage how you stripe your writes between those files to handle your workload. Your disk can also have zones that aren't shingled, but those seem to have been fairly unpopular.
geerlingguy 11y ago
I accidentally misspelled 'BackBlaze' in the title; can someone fix it please? Interesting decision by BackBlaze to stuff an extra row of drives in, lengthening the chassis to 33" instead of just under 29". It is their design, so they're free to do what they need for their own racks, but this does mean anyone trying to install a pod based on the 6.0 design couldn't use many closed-back racks or other kinds of racks.
- cthulhujr 11y ago
  Are they offering the 6.0 controllers, etc. with the standard length housing? I would think that's an option soon, if not now. Perhaps the 6.0 controllers have no benefit with fewer drives.
  - brianwski 11y ago
    Brian from Backblaze here. Just to be absolutely clear, you cannot "buy" one of these from Backblaze. We don't sell them. You are free to take these instructions and assemble one yourself though! And so yes, you could simply drop the 6.0 controllers into an older chassis and it will work fine. Also, the 6.0 housing will fit in most racks. If somebody can either measure their racks or include a link where the 33" chassis will no longer fit, we really want to hear about it!
    - toomuchtodo 11y ago
      Protocase, the vendor Backblaze sourced to fabricate the enclosures, has their own subsidiary that'll sell you turnkey pods: http://www.45drives.com/ http://www.45drives.com/
      - duskwuff 11y ago
        Surely they should be rebranding to 60drives now? ;)
  - gvb 11y ago
    Per the article, the electronics are unchanged, just 3 more backplanes and one SATA card: "When looking at what’s new in Storage Pod 6.0 it would easy to say it has 60 drives and stop there. After all, the Motherboard, CPU, memory, SATA cards, and backplanes we use didn’t change from 5.0." Later in the bullet list of changes: "Added 3 more backplanes and 1 more SATA card."
eloff 11y ago
Now at $0.036/GB. So basically for the cost of storing the data for a single month in the cloud - and cloud storage is one of the more price competitive options in the cloud.
- zzzcpan 11y ago
  Not sure what are you talking about. Their price for single month in the cloud is $0.005/GB.
  - brianwski 11y ago
    I think he was referring to Amazon S3 prices which are up around the numbers he mentioned. But Amazon also has "Glacier" for 3/4 of one penny per month, and Backblaze has an S3 competitor for even less at 1/2 of one penny per month.
    - toomuchtodo 11y ago
      Don't forget Google Nearline. $0.01/GB/month, same access speeds as S3.
      - amdavidson 11y ago
        Calling Nearline speeds the "same" is dependent on how much data you are storing per account, 4MBps per TB stored[0]. 0: https://cloud.google.com/storage/docs/nearline https://cloud.google.com/storage/docs/nearline
        boulos 11y ago
        If you needed to (as in a Disaster Recovery scenario), you'd just flip on "On-Demand I/O": https://cloud.google.com/storage/docs/on-demand-io https://cloud.google.com/storage/docs/on-demand-io . There will be a surcharge of $0.005 per GB of data transferred that goes over the 4 MB/s of throughput (https://cloud.google.com/storage/pricing#on-demand-pricing https://cloud.google.com/storage/pricing#on-demand-pricing). Disclosure: I work on Google Cloud (but not Nearline).
boulos 11y ago
So a hard drive weighs about 1.5 pounds, so 60 of them in a pod is about 100 pounds before you even include the pod itself. That doesn't sound particularly pleasant for the technician to install the first time... You put 10 of those in a rack and now we're up to 1000 pounds. At what point does this become challenging for the floor?
- atYevP 11y ago
  Yev from Backblaze here -> Good question! We actually took that in to consideration when designing them. In our datacenter the floors can handle it, but it's definitely something that should be considered if you're thinking about building these out. Check with your datacenter about what their "floor spec" is :)
  - nickpsecurity 11y ago
    Dropping in to say that from keeping the business financially in good shape to innovation level to opening up the hardware.... normally I'm the critic but all I can say is GREAT WORK. Good example to set and good tech. Keep it up. :)
    - atYevP 11y ago
      Thanks! We really do this so that others follow suite. It's sort of happened with Facebook's Open Compute, and Netflix's Open Connect releasing some of their data. Dropbox recently brought things in-house as well, so we're hoping they start talking about their goodies too! It makes things more interesting, and inexpensive storage opens up the possibilities for a lot of cool stuff. My favorite example is science labs. We've heard from a lot of them that they used to run experiments, collect the relevant data, and toss the rest. It was less expensive to run the experiment again than storing all of the data. That's changing now with storage being less expensive, and they're able to keep the entire experiment data set intact, and they love that because it means in the future they can take a look at all of the information surrounding their findings, and if the technology or science changes, they'll have the entire data set to play with instead of just a subset. Very cool stuff!
nocarrier 11y ago
It's always great to see storage hardware being opened up, thanks Backblaze. I have a question after reading the Vault overview here: https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect... I'm curious what the bandwidth demand is when an entire host fails and you have to drop a new replacement host in, since you would have 60 Tomes in the Vault that need to be rebuilt at once. You have at least two other parity copies when a host dies (assuming no other drives in the affected host's Tomes are dead on other hosts in the Vault set), so I'm guessing you can afford to wait the handful of days it would take to rebuild the host. I'm still curious to know what rebuild speed you plan for, since I'm guessing you'd be looking at 40Gbps NICs eventually. I was surprised to see only 2x10Gbps.
- scurvy 11y ago
  FYI, 25gbps ethernet is gaining in popularity much quicker than 40gbps ethernet. Mostly from cloud/scale-out stuff.
  - pavs 11y ago
    I always get confused with these, but are sfp/sfp+ ports also called ethernet ports? For whatever reason I associate ethernet with cat5/6 cable ports.
    - dsp1234 11y ago
      sfp/sfp+/qsfp are just the connector types, and can terminate cables that are made of copper or fiber. This is a separate issue from what protocol (ethernet/infiniband/etc) is running over that cable. For an example, take a look at Mellanox's cable page which lists a large number of combinations of connectors and cable types. So in general, to call a random sfp cable/port an ethernet port would be confusing unless both parties already knew it was passing ethernet traffic. http://www.mellanox.com/page/cables http://www.mellanox.com/page/cables
      - pavs 11y ago
m4dc4pXXX 11y ago
I also found an older post describing how they use Scrum to design the hardware to be pretty interesting: https://www.backblaze.com/blog/designing-the-next-storage-pod https://www.backblaze.com/blog/designing-the-next-storage-po.... I wonder what limitations the 6-month production run requirement put on design - it would seem you would have to be a lot more conservative in your "definition of done" (a bad decision can be expensive when you buy a half-year worht of gear).
Already__Taken 11y ago
Where or how do you find places that make chassis? I'd love to buy 100 or so completely basic cases to our design.
asdz 11y ago
At the rate of 0.036/GB, I'm just paying Backblaze USD50/year. Meaning if I store more than 115.74GB per month, Backblaze is losing money? And this is the cost only for storage, it could be 80GB to break even the cost and revenue. How do you guys make sure you can sustain in the long run? (I don't want to face another 'Copy case' T.T)
- atYevP 11y ago
  Yev from Backblaze here -> Accurate. We live off the averages. More people store less data than store more data. A lot of folks that have multiple terabytes backed up often tell their friends with less data and eventually it all averages out. We're bootstrapped so we have to be cash-flow positive, and at our $5/month price-point we are. We've been at this for 9 years too and so far its worked out OK! No reason to charge a lot for a service if you don't have to. We might need to move stuff around or change things in the future, but so far we've never had to raise prices and instead just keep adding features. Tell your friends? :)
nwmcsween 11y ago
Whoa wait they don't use ecc ram?
ksec 11y ago
Whenever i see a post on Storagepod, i always fantasize how much it will cost Apple to provide free iCloud Backup to its users. If you have 600M active users and 30GB per backup @ 0.036/Gb. That is 648M, so even triple redundancy will only cost less then 2Billion. If Apple could afford to give free OSX update, free iWork and ilife, Why not free iOS Backup? Or at least offer a Time Capsule that i can backup my iOS devices without going to the cloud.
- atYevP 11y ago
  Yea. Right?

181 points by geerlingguy 11y ago

One note - looks like you can get 8TB WD Helium drives for way cheaper ($250 vs $340) right now by shucking externals: http://www.amazon.com/Book-Desktop-External-Drive-WDBFJK0080HBK-NESN/dp/B01B6BN0Q2 http://www.amazon.com/Book-Desktop-External-Drive-WDBFJK0080...

Scramblejams 11y ago
Thanks for the tip. Though Backblaze has spoiled us with so much good reliability data, I now feel completely paralyzed in the face of buying a new drive if Backblaze hasn't told me how much I can trust it!
- atYevP 11y ago
  Yev from Backblaze here -> You can trust it. As long as you're OK with it dying and have other backups in place ;-)
  - Scramblejams 11y ago
    Ha! Well, even if I have backups, I don't like buying replacements, so there's that...
Keyframe 11y ago
I think that's a WD Green inside, not Red.
- lunixbochs 11y ago
  It's still a non-SMR 8TB drive, and pretty close to the Seagate archival SMR price.

daveguy 11y ago

Hey Yev or Brian, First I think it is awesome that you guys come on HN to respond when backblaze makes the news. Thank you. If you bought a prebuilt storage pod (from somewhere like http://www.backuppods.com/ http://www.backuppods.com/) could you start out at less than capacity and add more as you go? Say you started with one drive -- would it need to go in a specific location? When you add more would there be a significant reconfiguration process? Is it possible to hot-expand? I guess location and config has a lot to do with software involved. Has there been a blog on the software side of managing that much storage? Can it be as simple as a spanning partition to treat all drives as a single drive? Also, are you tracking the cost of SSD drives? Do you expect cost parity soon? It seems that capacity parity is getting close, but cost parity (especially at that capacity) is farther off. Do you have an estimate of when it might be or is it "far off, check again in 5 years". I'm sure you all found the ssd endurance test at techreport interesting (http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead http://techreport.com/review/27909/the-ssd-endurance-experim...).

atYevP 11y ago
Yev here! It's an interesting question. We use the pods in a larger "Vault" configuration, so the way we use them is a bit different (basically a set of 20 pods are connected -> https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect...). But, if you're buying a single pod, you could configure the software however you want, though you'd likely need the first hard drive to go on to the backplane that gets read first. We use different software so ours expects there to be 60 drives, but there's nothing stopping you from configuring Debian or whatnot to read a specific drive first. We ARE tracking SSD costs but thus far it's not quite at the point where the Cost per GB is in our favor. We'd love to play with them once it gets close, it would neat to build an SSD pod and see how it stacks up!
brianwski 11y ago
Yes, and we do this internally in our QA and load testing lab. It's an expensive waste to put $200,000 worth of drives in a 20 pod vault when all you need is 1/60th of that! > are you tracking the cost of SSD drives? Do you expect cost parity soon? Yes we track SSDs, and this is a hotly contested topic inside Backblaze. Personally I'm predicting they "cross over" in cost effectiveness within 2 years. But I don't have any more insight than you do, and many inside Backblaze disagree with me. When I say "cost effective" that is total cost of ownership, which includes paying for less electricity to power the SSD, possibly the higher density of SSDs, and whatever failure rates of the SSDs forcing us to purchase more (might be better or worse than hard drives). So whatever our little spread sheet kicks out as cheaper -> that's the one we will purchase! One random nice thing about SSDs is there will be less case vibration, and the drives shouldn't be affected AT ALL by the small amount of fan vibration.

scurvy 11y ago

Any reason on 60 drives instead of 70 or more? At that depth, there are several OEM designs that with that many drives (70). No 25/40gig networking? It's nice to point out the depth issue, as older racks are shallower than the newer versions. This blocks things like the 0U PDU channels and cable management channels. That said, the newer, deeper (and more popular) racks fit everything just fine. How tall are your racks? How many pods are you getting in there? 3ph 50A power?

brianwski 11y ago
> Any reason on 60 drives instead of 70 or more? One step at a time. :-) We think we can put a few more drives in the motherboard half of the POD without many other changes, and connect them to already existing SATA connectors on the motherboard. To get all the way to 75 drives we will need another row of port multipliers, which then means we need even more SATA cards, etc. > No 25/40gig networking? We're currently not able to saturate the 10 Gbit so for us it is wasted money to go faster. Remember that we run these in "vaults of 20" so we have 20 pods EACH with 10 Gbits so the vault hoovers in data at 200 Gbits/sec if you can thread the application. > How tall are your racks? Most of our racks are 45U tall where the PDU takes 4U, so we can fit (not necessarily power) 10 pods per rack and still fit a network switch in the 1U left at the top. In the past (with 45 drive pods) we ran 2 circuits of 30A 208V power (three phase). The two circuits are redundant - our power strips can flip over from one to the other if there is a loss of power on the currently used power. But I think they only put 8 or 9 of the 45 drive pods in a cabinet because it slightly exceeds what the datacenter likes. And now that we're moving up to all 60 drive pods, we have to rejigger the whole thing. It is also worth mentioning that when we bring a 20 pod vault online, we put each of the 20 pods in different racks in different parts of the datacenter. This helps keep the vault completely online if any one component fails. For example, if the power strip feeding the pods in 1 rack power dies, it only takes 1 pod offline out of several different vaults, so ALL the vaults stay online both reading and writing data. (I hope that made sense.)
- scurvy 11y ago
  Makes sense. Thanks for the reply! As to spreading the vaults around the DC, it sounds like that's more to prepare for ToR switch failure -- its sounds like an ATS handles your PDU and power circuit failover. Yeah it sounds like you're mostly CPU bound calculating EC if you're going to rebalance and replace failed pods so 25 gbps ethernet probably isn't interesting (yet). That said, it's only marginally more expensive than 10gbps twinax. Interesting that you're running 10gig baseT. Interesting from a power and latency perspective, though I'd imagine you're not worried too much about latency.

Scarbutt 11y ago

Dropbox manage to do 1PB per server, why doesn't backblaze do it?

bryanlarsen 11y ago
I believe that DropBox is using 12TB drives, which aren't available for general sale, yet. Backblaze is using 4TB drives, which is the current $/TB leader.
- wmf 11y ago
  It looks like 8TB SMR would be cheaper than 4TB. I don't mean to be negative, but Backblaze is starting to look like yesterday's technology today.
  - hackcasual 11y ago
    But if that 8TB drive has a higher failure rate, that needs to be taken into consideration as well.
  - rtkwe 11y ago
    It is but not by much. They actually provide the $/GB for a Seagate 8TB Archival SMR hdd in their SMR section. It comes to .034 $/GB vs .036 $/GB for the 4TB drives they're currently using. They've done the math it looks like and decided that the small gain doesn't make up for the time lost on overwrites.
venomsnake 11y ago
Backblaze are/were bootstrapped. They don't have the money to burn to move outside of the sweetspot $/GB - so their logic starts with which size drive is expected to be cheapest per GB in the next 12 months. I remember some of their old posts explaining why they made the jump from 2 to 3tb drives - they are slightly more expensive today, but we expect them to go down in price soon.
- atYevP 11y ago

bluedino 11y ago

Any idea what software is used to make the Build Book? https://f001.backblaze.com/file/Backblaze_Blog/Storage-Pod-6/Build+Book+Backblaze+60+Drive.pdf https://f001.backblaze.com/file/Backblaze_Blog/Storage-Pod-6...

jeffmcjunkin 11y ago
exiftool says it's PowerPoint, surprisingly: $ exiftool Build+Book+Backblaze+60+Drive.pdf | tail -n5 | head -n3 Title : 7400-0283 Rev A OMS Assy Backblaze 60 Drive Producer : Mac OS X 10.11.3 Quartz PDFContext Creator : PowerPoint
atYevP 11y ago
Yev from Backblaze here -> You're not going to believe this, but the folks at Evolve used Powerpoint and a template...seriously.
- bwilliams18 11y ago
  You would be astounded to learn how much use MS Office still gets today. Excel...that thing runs the world.

majke 11y ago

I've never heard about SMR before. I wonder - does it mean that a write to one area must be done in multiple disk rotations? Ie: if a sequential write to one file be aggregated into a single location on the disk or will it be spread around the whole plate. Just wondering how a software utilizing this should behave. http://www.storagereview.com/what_is_shingled_magnetic_recording_smr http://www.storagereview.com/what_is_shingled_magnetic_recor... https://en.wikipedia.org/wiki/Shingled_magnetic_recording https://en.wikipedia.org/wiki/Shingled_magnetic_recording

codemac 11y ago
Heads up - if you've bought a high density disk recently, it's probably SMR internally anyways. A write to one area can be done in a single disc rotation, as long as it's sequential. SMR disks added what are called ZBC, Zoned Block Commands to the SCSI spec. The disk itself exposes the ability to interact with zones several ways: - REPORT ZONES - What zones there are, how many, what type, and state of the zone. - OPEN / CLOSE / FINISH ZONE - zone write state management So essentially, it's like a drive exposes ~8 files, opened with O_APPEND, and you have to manage how you stripe your writes between those files to handle your workload. Your disk can also have zones that aren't shingled, but those seem to have been fairly unpopular.

geerlingguy 11y ago

I accidentally misspelled 'BackBlaze' in the title; can someone fix it please? Interesting decision by BackBlaze to stuff an extra row of drives in, lengthening the chassis to 33" instead of just under 29". It is their design, so they're free to do what they need for their own racks, but this does mean anyone trying to install a pod based on the 6.0 design couldn't use many closed-back racks or other kinds of racks.

cthulhujr 11y ago
Are they offering the 6.0 controllers, etc. with the standard length housing? I would think that's an option soon, if not now. Perhaps the 6.0 controllers have no benefit with fewer drives.
- brianwski 11y ago
  Brian from Backblaze here. Just to be absolutely clear, you cannot "buy" one of these from Backblaze. We don't sell them. You are free to take these instructions and assemble one yourself though! And so yes, you could simply drop the 6.0 controllers into an older chassis and it will work fine. Also, the 6.0 housing will fit in most racks. If somebody can either measure their racks or include a link where the 33" chassis will no longer fit, we really want to hear about it!
  - toomuchtodo 11y ago
    Protocase, the vendor Backblaze sourced to fabricate the enclosures, has their own subsidiary that'll sell you turnkey pods: http://www.45drives.com/ http://www.45drives.com/
    - duskwuff 11y ago
      Surely they should be rebranding to 60drives now? ;)
- gvb 11y ago
  Per the article, the electronics are unchanged, just 3 more backplanes and one SATA card: "When looking at what’s new in Storage Pod 6.0 it would easy to say it has 60 drives and stop there. After all, the Motherboard, CPU, memory, SATA cards, and backplanes we use didn’t change from 5.0." Later in the bullet list of changes: "Added 3 more backplanes and 1 more SATA card."

eloff 11y ago

Now at $0.036/GB. So basically for the cost of storing the data for a single month in the cloud - and cloud storage is one of the more price competitive options in the cloud.

zzzcpan 11y ago
Not sure what are you talking about. Their price for single month in the cloud is $0.005/GB.
- brianwski 11y ago
  I think he was referring to Amazon S3 prices which are up around the numbers he mentioned. But Amazon also has "Glacier" for 3/4 of one penny per month, and Backblaze has an S3 competitor for even less at 1/2 of one penny per month.
  - toomuchtodo 11y ago
    Don't forget Google Nearline. $0.01/GB/month, same access speeds as S3.
    - amdavidson 11y ago
      Calling Nearline speeds the "same" is dependent on how much data you are storing per account, 4MBps per TB stored[0]. 0: https://cloud.google.com/storage/docs/nearline https://cloud.google.com/storage/docs/nearline
      - boulos 11y ago
        If you needed to (as in a Disaster Recovery scenario), you'd just flip on "On-Demand I/O": https://cloud.google.com/storage/docs/on-demand-io https://cloud.google.com/storage/docs/on-demand-io . There will be a surcharge of $0.005 per GB of data transferred that goes over the 4 MB/s of throughput (https://cloud.google.com/storage/pricing#on-demand-pricing https://cloud.google.com/storage/pricing#on-demand-pricing). Disclosure: I work on Google Cloud (but not Nearline).

boulos 11y ago

So a hard drive weighs about 1.5 pounds, so 60 of them in a pod is about 100 pounds before you even include the pod itself. That doesn't sound particularly pleasant for the technician to install the first time... You put 10 of those in a rack and now we're up to 1000 pounds. At what point does this become challenging for the floor?

atYevP 11y ago
Yev from Backblaze here -> Good question! We actually took that in to consideration when designing them. In our datacenter the floors can handle it, but it's definitely something that should be considered if you're thinking about building these out. Check with your datacenter about what their "floor spec" is :)
- nickpsecurity 11y ago
  Dropping in to say that from keeping the business financially in good shape to innovation level to opening up the hardware.... normally I'm the critic but all I can say is GREAT WORK. Good example to set and good tech. Keep it up. :)
  - atYevP 11y ago
    Thanks! We really do this so that others follow suite. It's sort of happened with Facebook's Open Compute, and Netflix's Open Connect releasing some of their data. Dropbox recently brought things in-house as well, so we're hoping they start talking about their goodies too! It makes things more interesting, and inexpensive storage opens up the possibilities for a lot of cool stuff. My favorite example is science labs. We've heard from a lot of them that they used to run experiments, collect the relevant data, and toss the rest. It was less expensive to run the experiment again than storing all of the data. That's changing now with storage being less expensive, and they're able to keep the entire experiment data set intact, and they love that because it means in the future they can take a look at all of the information surrounding their findings, and if the technology or science changes, they'll have the entire data set to play with instead of just a subset. Very cool stuff!

nocarrier 11y ago

It's always great to see storage hardware being opened up, thanks Backblaze. I have a question after reading the Vault overview here: https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect... I'm curious what the bandwidth demand is when an entire host fails and you have to drop a new replacement host in, since you would have 60 Tomes in the Vault that need to be rebuilt at once. You have at least two other parity copies when a host dies (assuming no other drives in the affected host's Tomes are dead on other hosts in the Vault set), so I'm guessing you can afford to wait the handful of days it would take to rebuild the host. I'm still curious to know what rebuild speed you plan for, since I'm guessing you'd be looking at 40Gbps NICs eventually. I was surprised to see only 2x10Gbps.

scurvy 11y ago
FYI, 25gbps ethernet is gaining in popularity much quicker than 40gbps ethernet. Mostly from cloud/scale-out stuff.
- pavs 11y ago
  I always get confused with these, but are sfp/sfp+ ports also called ethernet ports? For whatever reason I associate ethernet with cat5/6 cable ports.
  - dsp1234 11y ago
    sfp/sfp+/qsfp are just the connector types, and can terminate cables that are made of copper or fiber. This is a separate issue from what protocol (ethernet/infiniband/etc) is running over that cable. For an example, take a look at Mellanox's cable page which lists a large number of combinations of connectors and cable types. So in general, to call a random sfp cable/port an ethernet port would be confusing unless both parties already knew it was passing ethernet traffic. http://www.mellanox.com/page/cables http://www.mellanox.com/page/cables
    - pavs 11y ago

m4dc4pXXX 11y ago

I also found an older post describing how they use Scrum to design the hardware to be pretty interesting: https://www.backblaze.com/blog/designing-the-next-storage-pod https://www.backblaze.com/blog/designing-the-next-storage-po.... I wonder what limitations the 6-month production run requirement put on design - it would seem you would have to be a lot more conservative in your "definition of done" (a bad decision can be expensive when you buy a half-year worht of gear).

Already__Taken 11y ago

Where or how do you find places that make chassis? I'd love to buy 100 or so completely basic cases to our design.

asdz 11y ago

At the rate of 0.036/GB, I'm just paying Backblaze USD50/year. Meaning if I store more than 115.74GB per month, Backblaze is losing money? And this is the cost only for storage, it could be 80GB to break even the cost and revenue. How do you guys make sure you can sustain in the long run? (I don't want to face another 'Copy case' T.T)

atYevP 11y ago
Yev from Backblaze here -> Accurate. We live off the averages. More people store less data than store more data. A lot of folks that have multiple terabytes backed up often tell their friends with less data and eventually it all averages out. We're bootstrapped so we have to be cash-flow positive, and at our $5/month price-point we are. We've been at this for 9 years too and so far its worked out OK! No reason to charge a lot for a service if you don't have to. We might need to move stuff around or change things in the future, but so far we've never had to raise prices and instead just keep adding features. Tell your friends? :)

nwmcsween 11y ago

Whoa wait they don't use ecc ram?

ksec 11y ago

Whenever i see a post on Storagepod, i always fantasize how much it will cost Apple to provide free iCloud Backup to its users. If you have 600M active users and 30GB per backup @ 0.036/Gb. That is 648M, so even triple redundancy will only cost less then 2Billion. If Apple could afford to give free OSX update, free iWork and ilife, Why not free iOS Backup? Or at least offer a Time Capsule that i can backup my iOS devices without going to the cloud.

atYevP 11y ago
Yea. Right?