8 ms·

135 Terabytes for $12,000

158 points by gourneau 15y ago

rbanffy 15y ago
Well... You can make all 46 drives into a zpool with raid-z and boot FreeBSD or *Solaris from it. BTW, I can't see the vibration sleeves around the disks on the pictures. Vibration must be a problem when you pack so much rotating media in such a limited space.
- cbs 15y ago
  Backblaze's blog talks about the vibration problem and their solution.
  - moe 15y ago
    Just a datapoint, yes, vibration is a real issue: http://www.youtube.com/watch?v=tDacjrSCeq4 http://www.youtube.com/watch?v=tDacjrSCeq4
- deleted 15y ago
  [deleted]
dmk23 15y ago
The real lesson here is that if your service relies on "Big Data" processing it pays off to build your own storage. If you were to buy 135Tb on Amazon EC2/EBS the same capacity would cost you ~$19,000 per month, not counting charges for I/O and bandwidth.
- beagle3 15y ago
  Well, space, air conditioning electricity and redundancy comparable to what Amazon offers would add a few thousands per month (I guess $2000-$5000, depending on where you are). Still, way cheaper than Amazon, but there are nontrivial running costs beyond the purchase and build price.
  - moe 15y ago
    You should be able to rent half a rack with sufficient power to house this for around $1000/mo in just about any civilized datacenter.
    - lsc 15y ago
      I'd love to sell you a half rack and a 20a circuit for a grand a month. That's high even here in the sf bay area.
- DanBC 15y ago
  No. If you rely on a lot of storage it pays to have really smart people building the hardware and software, and maintaining it, with service agreements in place. If 100 TB was as easy as buying a bunch of discs and slapping NASlite or FreeNAS on them then everyone would be doing it.
  - DanBC 15y ago
    Not sure why that's controversial?
    - lsc 15y ago
      because nineteen grand a month buys you a boatload of hardware /and/ a really smart person. A really smart person that can also be used on other things. S3, economically speaking, is an incredible deal if you are only using a little. But if you are using a lot? it makes NetApp or EMC start to look good, which is to say, only a good deal if money doesn't matter. I mean, yeah, if you are in a situation where money doesn't matter, or rather a situation where the amount of money you are spending on the storage is trivial compared to what you are doing with the storage, sure, NetApp or EMC will deliver a far superior product. S3 may have it's uses too. But if you are doing something where the storage needs to be cheap? yeah, neither one of those players, nor S3 will get you where you want to go.
tomkinstinch 15y ago
I am mildly surprised they can dissipate enough heat from the pods--those drives are packed so close together. I can't tell from the photos: Is there any airflow between them? (For that matter, do higher temperatures decrease MTTF for hard drives?)
- brianwski 15y ago
  Backblaze employee here. Yes, the 6 fans move a lot of air through the drives. We monitor every drive in every pod in our datacenter with the smartmontools (/usr/sbin/smartctl) which includes the temperature of the drives. The drives stay well within their recommended operating temperatures as long as the fans are running. In the past, we have detected that several fans failed in one single pod when alarms went off saying some drives were running hot, so the fans are working (normally) and the air is flowing. Side note: personally I think it's really awesome that consumer grade drives come with such detailed internal monitoring for free. I also think it is disappointing that desktop consumer OS's don't monitor these things BY DEFAULT and popup dialogs explaining to you when your drive is exhibiting high temperatures or high data loss!
- mturmon 15y ago
  Temp does not matter as much as has been thought. See: http://labs.google.com/papers/disk_failures.html http://labs.google.com/papers/disk_failures.html IIRC, up to about 40 C, failure rates are pretty flat as a function of temperature. The paper has some nice graphs. There is one other large-scale study in circulation that showed the same thing, but I don't recall the source.
- drv 15y ago
  A Google study about hard drive failure indicated that they couldn't find any conclusive relationship between temperature and failure rate. However, they did mention it had some effect at the extreme high end of the temperature spectrum (based on the graph, above 45 °C or so). http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/disk_failures.pdf http://static.googleusercontent.com/external_content/untrust...
  - 15y ago
rorrr 15y ago
Seems expensive. 2TB drives are now around $70 (Maybe even cheaper if you buy in bulk). 135TB = 68 drives. 68 * $70 = $4,760 Is that rack so expensive?
- m0shen 15y ago
  They're using 3TB drives
- ceejayoz 15y ago
  The prices are entirely laid out in part 3, titled "Real World Backblaze Costs". http://bioteam.net/2011/08/real-world-backblaze-costs/ http://bioteam.net/2011/08/real-world-backblaze-costs/
- wazoox 15y ago
  The box has 48 slots, so they must use 3 TB drives. They may be not completely crazy and use professional, not desktop drives. A 3TB pro drive costs about $200.
  - duskwuff 15y ago
    If you're sufficiently crazy to trust your data to a Backblaze pod, you're probably going to go full retard and use desktop disks anyway. If you cared about your data you probably wouldn't be building one of these things in the first place. :) Also: depends what you're considering "pro drives", but enterprise SATA 3TB disks are in the $250 - $300 range -- about 2x the price of a similar consumer-grade disk.
    - serverascode 15y ago
      AFAIK 3TB SAS drives, of which there's only Hitachi and Seagate I believe, cost about $320 CDN. ;) I use that number to figure out how much enterprise storage chassis actually cost by subtracting the number of disks * $320. :)
    - imaginator 15y ago
      Has anyone done any research to find a difference between "desktop" drives and "enterprise/pro" drives? I'm inclined to believe it's marketing speak and targeted at the same people that buy electron-aligned speaker cable.
ww520 15y ago
What would the electric bill and cooling cost be? Power bill is quite expensive in data center.
- mrb 15y ago
  Very low. I estimate the power consumption at the wall to be ~400W. That's $29/month at $0.10/kWh. Host this in a datacenter with a non-remarkable PUE of 1.5, and that's $43.5/month including cooling. * 45 x Hitachi HDS5C3030ALA630 drives, 4.8W each (average when idle), a tiny bit more when seeking so let's round to 5W. Source: http://www.hitachigst.com/tech/techlib.nsf/techdocs/02D9197756A273D0862577D50024EC1D/$file/DS5K3000_ds.pdf http://www.hitachigst.com/tech/techlib.nsf/techdocs/02D91977... and my measurements that HDDs seeking only consume a tiny bit more current than when idle http://blog.zorinaq.com/?e=50 http://blog.zorinaq.com/?e=50 * 100 Watt for the mobo, CPU (73W TDP), SATA controllers, and all the fans. Source: my own clamp-meter measurements on dozens of PCs http://blog.zorinaq.com/?e=42 http://blog.zorinaq.com/?e=42 * 85% average efficiency of the Zippy PSM-5760 power supply. Source: its 80PLUS certification report http://www.plugloadsolutions.com/psu_reports/SP105_EMACS_HU2-5760V_760W_Report.pdf http://www.plugloadsolutions.com/psu_reports/SP105_EMACS_HU2... * Hence: (5.0*45+100)/.85 = 382 Watt
mrb 15y ago
"The backblaze 2.0 pod has exceeded expectations when it comes to data movement and throughput. We get near wire-speed performance across a single Gigabit Ethernet link." - http://bioteam.net/2011/08/backblaze-performance/ http://bioteam.net/2011/08/backblaze-performance/ Oh my. These guys' expectations were very, very wrong. Many single-drive configurations can sustain 120MB/s+ at the beginning of the platter. A single drive! It should be no surprise that a 45-drive beast can saturate a GbE link!
- icefox 15y ago
  It is too bad they didn't bond the two ethernet links. Reading through I didn't see a mention of what the other ethernet link was used for. That should have given them a jump in speed.
  - mmetzger 15y ago
    We're considering building one of these but I used to work for a clustered storage vendor. The import thing to notice here is this is single client performance - aggregate performance is often considerably better. ie, you might be able to push 1GB/s with 10 clients, but only 100MB/s with one...
blhack 15y ago
Okay, so maybe I'm missing the point here, but, uh... No shit? Is there anybody who seriously is tossing 135Tb at a single JBOD? I'm not a storage guy, not even close, and even I know that that is a frighteningly ridiculous, get-you-fired idea. These are meant as individual parts in a larger, highly redundant setup. That's the entire point. Is the next post going to be: "Why you won't be able to download music at 1Gbps even with a GIGABIT SWITCH! Cisco lies!"?
- tommi 15y ago
  Read the post. In particular "Why the folks at Backblaze don’t care about the “risks”" section.
  - blhack 15y ago
    I know, my point is that anybody with enough knowledge about this topic to be interested in this article would already know.
- smackfu 15y ago
  It seems like an interesting one-off kind of scenario. They have a lot of downloaded data, none of which is irreplaceable, but which would take a significant amount of time to redownload. This pod is a backup but a not very rigorous one, since the goal is to never have to download all the data, not to never have to download any of the data.
- serverascode 15y ago
  I'm looking at enclosures that can hold 60 disks. So 3TB SAS disks would be 180TB. Most enclosures of that kind have dual everything. Some SAS drives even have dual connections I believe (should look that up again). At some point it's really a function of the size of the drives that makes the volume large. I think 60 drives is going to pretty much be the physical limit in 4 rack units. You can buy the NL-108 from Isilon which has 108TB (36 3TB disks) if you want "enterprise." ;) Eventually they'll use 4TB drives in those.
- soult 15y ago
  I take it you only read the first part of the article. They used RAID6 over 15 drives each, combining those 15 drives via LVM, allowing the loss of 2 drives from the same group without data loss. This is the same configuration Backblaze uses as well, because even though their fancy custom software mirrors all data to different storage pods, it is easier to just replace a drive and rebuild the raid than rebuilding the whole JBOD from the data on other pods.
johngalt 15y ago
There may be a reason to build 20 backblaze pods, but I doubt there is a reason to build 1. The whole point of a backblaze build is to move the redundancy out of the individual box.
- deleted 15y ago
  [deleted]
- asto 15y ago
  If you're handling large amounts of data on your sever(s) and want a single backup just in case? I think that's what they wanted.
  - johngalt 15y ago
    By itself this device is poorly suited to backup. Specially for their use case. A huge pile of desktop Hitachi disks isn't something you can stick in a rack and forget about, and people like to forget about backups. Best case: you have someone who's constantly checking/maintaining this box and replacing the drives as they fail. Worst case: an expectation of a backup, but every time you refer to it, it doesn't work. Most likely case: A mix of both of the above. Where the box takes a lot of attention, and occasionally works as intended. The only redeeming factor is that they proclaim all the reasons why this is a poor choice. So it seems someone is at least thinking about it.
serverascode 15y ago
I like the backblaze pod just in terms of using it as a comparison, where it is the absolute cheapest possible storage really. (Sure you could prob go cheaper.) I think they've done something that makes sense for them. It doesn't make sense for me to use it for storage where I work, but it works for them in this instance. And it seems to work for Backblaze.
driverdan 15y ago
I've been following the online storage space for years. Every time something like this comes out I redo my back-of-the-napkin math on potential profitability, get excited, then find excuses to not move into the space. Prices have gotten so low that there's no excuse for lack of quality online storage providers. There are many, but there is only a handful I would consider good and their pricing isn't following the market. I see this leaving a huge opening for competition. If anyone else is seriously interested in online storage contact me (see profile).
- lsc 15y ago
  eh, I'm working on it right now; I'll be bringing the prgmr.com values, which is to say, the focus is on cheap, standard, and transparent, uptime goals will be... realistic. I'm doing it with my co-author, and I have at least one investor who is interested, though I'm not sure we'll need him... even at our target of a penny per month per replication, it's going to be ridiculously profitable, probably enough so that we likely won't need money. What are you bringing to the table?
- gourneau 15y ago
  I think there would be a big market to provide $100,000ish per Petabyte collocation, when the hardware gets cheap enough for that.
  - lsc 15y ago
    Am I missing a zero? that's about $0.10 per gigabyte, no? which is dang close to what amazon charges. Unless you mean one hundred grand up front rather than one hundred grand a month.
    - gourneau 15y ago
      I mean a upfront cost, and then some monthly hosting fee maybe. Might be a while for hardware to be that cheap.
      - 15y ago
wmf 15y ago
Given that an empty pod costs $5,395, Supermicro might actually be cheaper (although a little less dense) and it's a real product instead of a prototype.
- lsc 15y ago
  I'm looking at http://www.supermicro.com/products/chassis/4U/847/SC847E26-R1400U.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E26-R... (36 drives, and room for a motherboard) or http://www.supermicro.com/products/chassis/4U/847/SC847E16-RJBOD1.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E16-R... (45 drives, das. I have had serious outages caused by DAS back in the SCSI-3 VHDCI days, so I'm likely to start with the 36 bay system.) for my mass storage project. they are both well under $1500 with power supplies, backplanes and expanders taken care of. For me, the big thing is that with my storage model, I'm going to be replacing disks as they fail and rebuilding the RAID, so having easily accessible and easily swapped disks is worth paying a premium. (I am planning to have some cross-chassis redundancy by using zfs snapshots, but I'd rather just keep the nodes going as is.) Also, rack density? doesn't save you that much money. Most of what you are paying for in a data center is power. At the cheapest co-lo I'm in, here is the cost breakdown: 1 full cabinet (44u) with two twenty-amp 120v circuits: $875 1 full cabinet (44u) with one twenty-amp 120v circuit: $530 So, if I can double my density, I save $185 a month; and even at the disk density for my compute nodes (close to 100 disks in a rack) I get one disk failure maybe every two months per rack; so if I have to slide out the whole goddam computer, causing some chance of the power getting disconnected and downtime? yeah, with my model? it's probably worth paying the premium. I'm just saying; if you are small enough that paying five grand for a backblaze pod. Sure, at-scale, it's best to design your systems so that you can get zero downtime even when hardware fails. But, that's really difficult to do without introducing new failure modes; even amazon has trouble with it. My strategy is to accept that hardware failures mean a truck roll and downtime for customers on the hardware in question. As long as you don't have any one server go down more than once a year, (and a particular server failing once a year is pretty pathetic) and as long as you don't have a system where any one server brings down everyone, you are going to see pretty good reliability using this strategy.
  - 15y ago
charlesap 15y ago
Grown defects. Graaah.
gourneau 15y ago
What do you guys think about using this with Swift from OpenStack? Could I buy 100 of these pods, drop OpenStack on them and get close to 135,000 Terabytes with some level of redundancy?
- notmyname 15y ago
  Some of the hardware choices that Backblaze has made in their pod design are interesting. Most of the concerns I have (like the difficulty of getting to drives to replace them) can be addressed by operational practices (update the swift ring rather than immediately replace the drive). Other concerns (like 2gbe for 135TB) are more nuanced. Lack of redundancy in the Backblaze pods is addressed by swift itself--swift will ensure that no two copies are on the same pod. I would love to see someone run swift on some Backblaze pods. If you'd like to talk further, my contact info is in my profile (or drop by #openstack on freenode).
iamelgringo 15y ago
Wow. The cost of 1 Petabyte of spinny disk storage is $100k. A friend of mine who raised money in 1999 said that to get their MVP off the ground, it took 20 engineers, and $7M in venture funding. I doubt $100k would have paid for the Oracle licenses back then.
mckoss 15y ago
The application here is write once, read rarely. Would it be possible to unpower drives once they are full (until needed)?

158 points by gourneau 15y ago

Well... You can make all 46 drives into a zpool with raid-z and boot FreeBSD or *Solaris from it. BTW, I can't see the vibration sleeves around the disks on the pictures. Vibration must be a problem when you pack so much rotating media in such a limited space.

cbs 15y ago
Backblaze's blog talks about the vibration problem and their solution.
- moe 15y ago
  Just a datapoint, yes, vibration is a real issue: http://www.youtube.com/watch?v=tDacjrSCeq4 http://www.youtube.com/watch?v=tDacjrSCeq4
deleted 15y ago
[deleted]

dmk23 15y ago

The real lesson here is that if your service relies on "Big Data" processing it pays off to build your own storage. If you were to buy 135Tb on Amazon EC2/EBS the same capacity would cost you ~$19,000 per month, not counting charges for I/O and bandwidth.

beagle3 15y ago
Well, space, air conditioning electricity and redundancy comparable to what Amazon offers would add a few thousands per month (I guess $2000-$5000, depending on where you are). Still, way cheaper than Amazon, but there are nontrivial running costs beyond the purchase and build price.
- moe 15y ago
  You should be able to rent half a rack with sufficient power to house this for around $1000/mo in just about any civilized datacenter.
  - lsc 15y ago
    I'd love to sell you a half rack and a 20a circuit for a grand a month. That's high even here in the sf bay area.
DanBC 15y ago
No. If you rely on a lot of storage it pays to have really smart people building the hardware and software, and maintaining it, with service agreements in place. If 100 TB was as easy as buying a bunch of discs and slapping NASlite or FreeNAS on them then everyone would be doing it.
- DanBC 15y ago
  Not sure why that's controversial?
  - lsc 15y ago
    because nineteen grand a month buys you a boatload of hardware /and/ a really smart person. A really smart person that can also be used on other things. S3, economically speaking, is an incredible deal if you are only using a little. But if you are using a lot? it makes NetApp or EMC start to look good, which is to say, only a good deal if money doesn't matter. I mean, yeah, if you are in a situation where money doesn't matter, or rather a situation where the amount of money you are spending on the storage is trivial compared to what you are doing with the storage, sure, NetApp or EMC will deliver a far superior product. S3 may have it's uses too. But if you are doing something where the storage needs to be cheap? yeah, neither one of those players, nor S3 will get you where you want to go.

tomkinstinch 15y ago

I am mildly surprised they can dissipate enough heat from the pods--those drives are packed so close together. I can't tell from the photos: Is there any airflow between them? (For that matter, do higher temperatures decrease MTTF for hard drives?)

brianwski 15y ago
Backblaze employee here. Yes, the 6 fans move a lot of air through the drives. We monitor every drive in every pod in our datacenter with the smartmontools (/usr/sbin/smartctl) which includes the temperature of the drives. The drives stay well within their recommended operating temperatures as long as the fans are running. In the past, we have detected that several fans failed in one single pod when alarms went off saying some drives were running hot, so the fans are working (normally) and the air is flowing. Side note: personally I think it's really awesome that consumer grade drives come with such detailed internal monitoring for free. I also think it is disappointing that desktop consumer OS's don't monitor these things BY DEFAULT and popup dialogs explaining to you when your drive is exhibiting high temperatures or high data loss!
mturmon 15y ago
Temp does not matter as much as has been thought. See: http://labs.google.com/papers/disk_failures.html http://labs.google.com/papers/disk_failures.html IIRC, up to about 40 C, failure rates are pretty flat as a function of temperature. The paper has some nice graphs. There is one other large-scale study in circulation that showed the same thing, but I don't recall the source.
drv 15y ago
A Google study about hard drive failure indicated that they couldn't find any conclusive relationship between temperature and failure rate. However, they did mention it had some effect at the extreme high end of the temperature spectrum (based on the graph, above 45 °C or so). http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/disk_failures.pdf http://static.googleusercontent.com/external_content/untrust...
- 15y ago

rorrr 15y ago

Seems expensive. 2TB drives are now around $70 (Maybe even cheaper if you buy in bulk). 135TB = 68 drives. 68 * $70 = $4,760 Is that rack so expensive?

m0shen 15y ago
They're using 3TB drives
ceejayoz 15y ago
The prices are entirely laid out in part 3, titled "Real World Backblaze Costs". http://bioteam.net/2011/08/real-world-backblaze-costs/ http://bioteam.net/2011/08/real-world-backblaze-costs/
wazoox 15y ago
The box has 48 slots, so they must use 3 TB drives. They may be not completely crazy and use professional, not desktop drives. A 3TB pro drive costs about $200.
- duskwuff 15y ago
  If you're sufficiently crazy to trust your data to a Backblaze pod, you're probably going to go full retard and use desktop disks anyway. If you cared about your data you probably wouldn't be building one of these things in the first place. :) Also: depends what you're considering "pro drives", but enterprise SATA 3TB disks are in the $250 - $300 range -- about 2x the price of a similar consumer-grade disk.
  - serverascode 15y ago
    AFAIK 3TB SAS drives, of which there's only Hitachi and Seagate I believe, cost about $320 CDN. ;) I use that number to figure out how much enterprise storage chassis actually cost by subtracting the number of disks * $320. :)
  - imaginator 15y ago
    Has anyone done any research to find a difference between "desktop" drives and "enterprise/pro" drives? I'm inclined to believe it's marketing speak and targeted at the same people that buy electron-aligned speaker cable.

ww520 15y ago

What would the electric bill and cooling cost be? Power bill is quite expensive in data center.

mrb 15y ago
Very low. I estimate the power consumption at the wall to be ~400W. That's $29/month at $0.10/kWh. Host this in a datacenter with a non-remarkable PUE of 1.5, and that's $43.5/month including cooling. * 45 x Hitachi HDS5C3030ALA630 drives, 4.8W each (average when idle), a tiny bit more when seeking so let's round to 5W. Source: http://www.hitachigst.com/tech/techlib.nsf/techdocs/02D9197756A273D0862577D50024EC1D/$file/DS5K3000_ds.pdf http://www.hitachigst.com/tech/techlib.nsf/techdocs/02D91977... and my measurements that HDDs seeking only consume a tiny bit more current than when idle http://blog.zorinaq.com/?e=50 http://blog.zorinaq.com/?e=50 * 100 Watt for the mobo, CPU (73W TDP), SATA controllers, and all the fans. Source: my own clamp-meter measurements on dozens of PCs http://blog.zorinaq.com/?e=42 http://blog.zorinaq.com/?e=42 * 85% average efficiency of the Zippy PSM-5760 power supply. Source: its 80PLUS certification report http://www.plugloadsolutions.com/psu_reports/SP105_EMACS_HU2-5760V_760W_Report.pdf http://www.plugloadsolutions.com/psu_reports/SP105_EMACS_HU2... * Hence: (5.0*45+100)/.85 = 382 Watt

mrb 15y ago

"The backblaze 2.0 pod has exceeded expectations when it comes to data movement and throughput. We get near wire-speed performance across a single Gigabit Ethernet link." - http://bioteam.net/2011/08/backblaze-performance/ http://bioteam.net/2011/08/backblaze-performance/ Oh my. These guys' expectations were very, very wrong. Many single-drive configurations can sustain 120MB/s+ at the beginning of the platter. A single drive! It should be no surprise that a 45-drive beast can saturate a GbE link!

icefox 15y ago
It is too bad they didn't bond the two ethernet links. Reading through I didn't see a mention of what the other ethernet link was used for. That should have given them a jump in speed.
- mmetzger 15y ago
  We're considering building one of these but I used to work for a clustered storage vendor. The import thing to notice here is this is single client performance - aggregate performance is often considerably better. ie, you might be able to push 1GB/s with 10 clients, but only 100MB/s with one...

blhack 15y ago

Okay, so maybe I'm missing the point here, but, uh... No shit? Is there anybody who seriously is tossing 135Tb at a single JBOD? I'm not a storage guy, not even close, and even I know that that is a frighteningly ridiculous, get-you-fired idea. These are meant as individual parts in a larger, highly redundant setup. That's the entire point. Is the next post going to be: "Why you won't be able to download music at 1Gbps even with a GIGABIT SWITCH! Cisco lies!"?

tommi 15y ago
Read the post. In particular "Why the folks at Backblaze don’t care about the “risks”" section.
- blhack 15y ago
  I know, my point is that anybody with enough knowledge about this topic to be interested in this article would already know.
smackfu 15y ago
It seems like an interesting one-off kind of scenario. They have a lot of downloaded data, none of which is irreplaceable, but which would take a significant amount of time to redownload. This pod is a backup but a not very rigorous one, since the goal is to never have to download all the data, not to never have to download any of the data.
serverascode 15y ago
I'm looking at enclosures that can hold 60 disks. So 3TB SAS disks would be 180TB. Most enclosures of that kind have dual everything. Some SAS drives even have dual connections I believe (should look that up again). At some point it's really a function of the size of the drives that makes the volume large. I think 60 drives is going to pretty much be the physical limit in 4 rack units. You can buy the NL-108 from Isilon which has 108TB (36 3TB disks) if you want "enterprise." ;) Eventually they'll use 4TB drives in those.
soult 15y ago
I take it you only read the first part of the article. They used RAID6 over 15 drives each, combining those 15 drives via LVM, allowing the loss of 2 drives from the same group without data loss. This is the same configuration Backblaze uses as well, because even though their fancy custom software mirrors all data to different storage pods, it is easier to just replace a drive and rebuild the raid than rebuilding the whole JBOD from the data on other pods.

johngalt 15y ago

There may be a reason to build 20 backblaze pods, but I doubt there is a reason to build 1. The whole point of a backblaze build is to move the redundancy out of the individual box.

deleted 15y ago
[deleted]
asto 15y ago
If you're handling large amounts of data on your sever(s) and want a single backup just in case? I think that's what they wanted.
- johngalt 15y ago
  By itself this device is poorly suited to backup. Specially for their use case. A huge pile of desktop Hitachi disks isn't something you can stick in a rack and forget about, and people like to forget about backups. Best case: you have someone who's constantly checking/maintaining this box and replacing the drives as they fail. Worst case: an expectation of a backup, but every time you refer to it, it doesn't work. Most likely case: A mix of both of the above. Where the box takes a lot of attention, and occasionally works as intended. The only redeeming factor is that they proclaim all the reasons why this is a poor choice. So it seems someone is at least thinking about it.

serverascode 15y ago

I like the backblaze pod just in terms of using it as a comparison, where it is the absolute cheapest possible storage really. (Sure you could prob go cheaper.) I think they've done something that makes sense for them. It doesn't make sense for me to use it for storage where I work, but it works for them in this instance. And it seems to work for Backblaze.

driverdan 15y ago

I've been following the online storage space for years. Every time something like this comes out I redo my back-of-the-napkin math on potential profitability, get excited, then find excuses to not move into the space. Prices have gotten so low that there's no excuse for lack of quality online storage providers. There are many, but there is only a handful I would consider good and their pricing isn't following the market. I see this leaving a huge opening for competition. If anyone else is seriously interested in online storage contact me (see profile).

lsc 15y ago
eh, I'm working on it right now; I'll be bringing the prgmr.com values, which is to say, the focus is on cheap, standard, and transparent, uptime goals will be... realistic. I'm doing it with my co-author, and I have at least one investor who is interested, though I'm not sure we'll need him... even at our target of a penny per month per replication, it's going to be ridiculously profitable, probably enough so that we likely won't need money. What are you bringing to the table?
gourneau 15y ago
I think there would be a big market to provide $100,000ish per Petabyte collocation, when the hardware gets cheap enough for that.
- lsc 15y ago
  Am I missing a zero? that's about $0.10 per gigabyte, no? which is dang close to what amazon charges. Unless you mean one hundred grand up front rather than one hundred grand a month.
  - gourneau 15y ago
    I mean a upfront cost, and then some monthly hosting fee maybe. Might be a while for hardware to be that cheap.
    - 15y ago

wmf 15y ago

Given that an empty pod costs $5,395, Supermicro might actually be cheaper (although a little less dense) and it's a real product instead of a prototype.

lsc 15y ago
I'm looking at http://www.supermicro.com/products/chassis/4U/847/SC847E26-R1400U.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E26-R... (36 drives, and room for a motherboard) or http://www.supermicro.com/products/chassis/4U/847/SC847E16-RJBOD1.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E16-R... (45 drives, das. I have had serious outages caused by DAS back in the SCSI-3 VHDCI days, so I'm likely to start with the 36 bay system.) for my mass storage project. they are both well under $1500 with power supplies, backplanes and expanders taken care of. For me, the big thing is that with my storage model, I'm going to be replacing disks as they fail and rebuilding the RAID, so having easily accessible and easily swapped disks is worth paying a premium. (I am planning to have some cross-chassis redundancy by using zfs snapshots, but I'd rather just keep the nodes going as is.) Also, rack density? doesn't save you that much money. Most of what you are paying for in a data center is power. At the cheapest co-lo I'm in, here is the cost breakdown: 1 full cabinet (44u) with two twenty-amp 120v circuits: $875 1 full cabinet (44u) with one twenty-amp 120v circuit: $530 So, if I can double my density, I save $185 a month; and even at the disk density for my compute nodes (close to 100 disks in a rack) I get one disk failure maybe every two months per rack; so if I have to slide out the whole goddam computer, causing some chance of the power getting disconnected and downtime? yeah, with my model? it's probably worth paying the premium. I'm just saying; if you are small enough that paying five grand for a backblaze pod. Sure, at-scale, it's best to design your systems so that you can get zero downtime even when hardware fails. But, that's really difficult to do without introducing new failure modes; even amazon has trouble with it. My strategy is to accept that hardware failures mean a truck roll and downtime for customers on the hardware in question. As long as you don't have any one server go down more than once a year, (and a particular server failing once a year is pretty pathetic) and as long as you don't have a system where any one server brings down everyone, you are going to see pretty good reliability using this strategy.
- 15y ago

charlesap 15y ago

Grown defects. Graaah.

gourneau 15y ago

What do you guys think about using this with Swift from OpenStack? Could I buy 100 of these pods, drop OpenStack on them and get close to 135,000 Terabytes with some level of redundancy?

notmyname 15y ago
Some of the hardware choices that Backblaze has made in their pod design are interesting. Most of the concerns I have (like the difficulty of getting to drives to replace them) can be addressed by operational practices (update the swift ring rather than immediately replace the drive). Other concerns (like 2gbe for 135TB) are more nuanced. Lack of redundancy in the Backblaze pods is addressed by swift itself--swift will ensure that no two copies are on the same pod. I would love to see someone run swift on some Backblaze pods. If you'd like to talk further, my contact info is in my profile (or drop by #openstack on freenode).

iamelgringo 15y ago

Wow. The cost of 1 Petabyte of spinny disk storage is $100k. A friend of mine who raised money in 1999 said that to get their MVP off the ground, it took 20 engineers, and $7M in venture funding. I doubt $100k would have paid for the Oracle licenses back then.

mckoss 15y ago

The application here is write once, read rarely. Would it be possible to unpower drives once they are full (until needed)?