9 ms·

Petabytes on a budget: How to build cheap cloud storage

401 points by bensummers 17y ago

dmillar 17y ago
This is unbelievably awesome. Can you imagine what we could achieve if every company had this level of transparency?
- PStamatiou 17y ago
  Thinking the exact same thing! While technology isn't more than any typical hacker can slap together in their apartment (minus maybe the case fab), I think the application is ingenious and the full blog post about it, like dmillar, pointed out is rather awe-inspiring. And I was all happy because my new Core i7 box has 2TB of space and 6GB of ram... sigh That being said - did seagate ever fix their firmware issue on the 1.5TB drives that would cause random corruption? (heard about it maybe 6 months ago)
  - barrkel 17y ago
    I should think so - I have 4 of them in my ZFS pool, and scrubs are turning up no errors.
  - sophacles 17y ago
    I have the 500GB in the same series (with the corrupted firmware) and the fix did very nice things for it. Also, I have never had problems with the 7200.12 (the newer version of the same drive) in a 1TB model.
  - jbellis 17y ago
    > While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.
    - PStamatiou 17y ago
      I would have some smart-ass comment here but I just read your HN profile. Good day, Sir.
      - tzury 17y ago
        smart move ;-)
    - Periodic 17y ago
      This is what always gets in my way. It takes a lot of work and a lot of expertise to put together a home-grown system that works as well as one from the major vendors. If you're only going to be using one or two, you're much better off going to one of those major vendors because a large part of the price is their expertise and testing that went into it. For a large setup like Backblaze they can spread the cost of design over many systems, but for smaller companies it just isn't feasible. We hacker types love to think that we could do the same thing in no time with little budget, and I'm sure we could get a first approximation. But the devil is in the details. Debugging the complex interaction of 20 different hardware components is not my idea of fun. Hats off to them, particularly for sharing.
      - rantfoil 17y ago
        Secondary revenue streams anyone? I know we'd be interested in becoming a customer if Backblaze at some point decided to monetize this part of their knowledge too. This seems like an opportunity to disrupt from the low end of the market for some scrappy enterprising engineers. If they don't do it, someone should.
      - uhgygghhj 17y ago
        I did this as an inhouse backup for a data warehousing app. Just slapping 4 ide cards into a case and putting 16x250gb IDE drives on them resulted in a system that would copy about 1 disk worth before hanging with some fault or suddenly dropping to 1% speed. Just because you can in theory hook 40 drives to n cards doesn't mean it will work - well done to them
      - 17y ago
  - bestes 17y ago
    Yes, they did a firmware update. Here is one article that talks about it: http://www.tomshardware.com/news/seagate-barracuda-1.5TB-freezing,6582.html http://www.tomshardware.com/news/seagate-barracuda-1.5TB-fre...
    - herf 17y ago
      Seagate took about 4 months to fully resolve the issue (their first fix had a bug!) I had a drive that bricked when applying the fix several months after the initial firmware updates. Apparently a buffer overflow in the error log that locked the drive on reboot. Seems dangerous to source all your hardware from one vendor over a small timeframe. Would be nice to have some redundancy over manufacturers too.
  - lsc 17y ago
    as for the firmware 'bug' - yeah, they fixed that one, However, consumer sata disks are still not something you want in most raid systems. See, a consumer disk, when it encounters an unrecoverable error, retries over and over, hanging your raid until you fail the bad drive by hand. (tested on a 3ware hardware raid and with MD. I am given to understand that zfs handles this in a sane manner.) Western digital allows you to change the behavior, search for WDTLER, but in general. I'm sure the backblaze folks have built something that handles that sort of thing into the system, but just saying; it's something you have to think about before you slap consumer drives into a raid that needs to keep working when one drive fails.
    - Devilboy 17y ago
      The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive since the drives make up most of the hardware cost.
      - lsc 17y ago
        But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.
        Devilboy 17y ago
        In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build large arrays to beat triple JBOD for storage efficiency - a 12 x 1TB array yields 5TB usable space while triple JBOD gives you 4TB. BUT you still have a single point of failure - your RAID card can give out and your array is unavailable. Using triple JBOD you'll be able to have your data on different physical machines or even geographically separate data centers if you wish. The advantages outweigh the reduction of storage density when you're dealing with petabytes of storage - why else would Google et al be doing it?
edw519 17y ago
I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81,000 Backblaze $117,000 Dell $826,000 Sun $1,000,000 NetApp $1,714,000 Amazon $2,806,000 EMC $2,860,000 I had no idea. Kinda makes one rethink what business they want to be in.
- spydez 17y ago
  Well, to be fair, Backblaze is using consumer-grade hardware here. No redundant PSUs. No mirrored OS drives. Cheaper components. Etc. They're going Google's route, which is fine, it's just... a whole different direction than a Sun Thumper. I'd be interested to see how their costs per petabyte stack against Google's.
  - blasdel 17y ago
    No, they aren't going Google's route! Google's operation has at most half the disks-per-rack density of Backblaze, but with 4 or 8 times the server density. Google does almost all their computation on the same nodes where the data is stored. Google is also storing each tablet a minimum of 3 times within each cluster, with most systems having multiple replica clusters. Backblaze's system is going to have several orders of magnitude less bandwidth: way too many drives in one server (some of them on the PCI bus!) use of port multipliers (causing 5 drives to share one SATA cable's 300MB/s) RAID6 with too many drives per array (15-way XOR is no fun) why use JFS? only access is via HTTPS, not clear if SSL is done on the pod
    - cheriot 17y ago
      I'm guessing that backups are accessed rather infrequently and that someone recovering their data won't mind a few seconds in lag time. This isn't my area of expertise, but their trade off seems sensible. It's just a different scenario than what google is dealing with.
- cperciva 17y ago
  Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids, etc. -- and any system which stores all of its data in a single datacenter lacks any protection against that. Store each file on two different servers in each of two different datecenters, and suddenly Backblaze's $134k turns into $536k. The price for Amazon, in contrast, is based on the assumption that their prices remain fixed for the next 3 years -- which seems a rather radical assumption. Is backblaze's solution cheaper than S3? Absolutely. But they're also twisting the numbers a bit.
modoc 17y ago
I love how open they are about their solution, and I also want one for Christmas:)
psranga 17y ago
Great stuff. Would anybody care to clarify why access is through HTTPS and not HTTP? I presume all accesses to these pods are from within their data center? Or do they directly expose these boxes to clients (whoa!)?
- phsr 17y ago
  BackBlaze runs online, off-site backups (like Mozy and Carbonite) at $5/month for unlimited storage. HTTPS is used to keep client data protected, I assume
tsuraan 17y ago
67TB of storage, with 4GB of cache. I'd really love to see some performance numbers versus the way-too-expensive competition. If the systems are being used as tape drive replacements, I could see this working well, but as an actual NAS-like device, I can't imaging how it could perform acceptably. Of course, if those Intel motherboards have the dual 1Gb/s NICs that Intel boards generally do, it will probably take a while to fill the drives anyhow.
- notaddicted 17y ago
  http://www.wolframalpha.com/input/?i=(67+TB)+%2F+(2*(1+Gb%2Fs)+ http://www.wolframalpha.com/input/?i=(67+TB)+%2F+(2*(1+Gb%2F...) It'll take 3 days at the theoretical max of the networking equipment to read/write the 67TB. The overhead of HTTPS constrains the networking so this is too low. I'd expect that their internet connection (i.e. in/out of the data center) is the real bottleneck. I believe that the system is being used as a tape drive replacement.
  - lsc 17y ago
    If the load is relatively sequential, then yeah. But if the load is mostly random, then there is an argument for having more cache.
- lsc 17y ago
  Actually, that's the problem with most NetAPP/EMC foo, too. they are way light on the cache. These guys only use 4GiB ram because they are saving money on the motherboard, I bet. Personally, if I were building it, I'd increase the cost by another grand or so and use dual low power opterons with 32GiB ram. (of course, that would also increase the space taken by the motherboard, so that would require some case redesigns. Still, opterons and registered ecc ddr2 are both incredibly cheap right now.)
- pmorici 17y ago
  What if they used AndrewFS with SSD drives for cache on front end nodes.
gstar 17y ago
They might have missed a trick here. To address vibration, acoustics and gyroscopic effect, what I've seen done in highly dense enclosures is to rotate every second drive around 180 degrees in a bit of a shotgun approach to balancing stuff. Still, awesome.
- sh1mmer 17y ago
  This is the trick, though. They want to improve their system. By sharing with you they may have solicited some really helpful advice.
- ciupicri 17y ago
  This has sounded ok for me too, but when I've thought a bit about it I've realized that the drives must be synchronized so that each one would "cancel" each others vibrations. The vibrations generated by the drives might have indeed the same amplitude and (spatial and temporal) frequency, but I doubt that they'll have the same phase offset. For this to happen they would need to be started at _exactly_ the same time which I don't think that happens in practice. Do you have some pictures of those enclosures?
  - gstar 17y ago
    I don't have pictures. I can describe, though. Two disks were mounted in a frame linearly, both screwed to the frame, with the power/sata connectors toward the middle of the frame, and one drive upside-down. These cassettes were removable as a unit for hot-swap, and were inserted linearly into a half-deep 19" rack enclosure. That the two drives were physically connected to the same frame, and removed and replaced as a unit would make it seem as if they would be started in phase. Now I'm not physicist, but I'm not 100% sure that's so important - if you have two gyroscopes contra-rotating and firmly connected, running at the same speed, surely they resist movement by sheer gyroscopic effect?
- Andys 17y ago
  Unfortunately the SATA backplanes are only aligned one way, so they could only rotate a set of 5 drives at once.
swombat 17y ago
Wow. Do they offer S3-like storage? They should. If they can offer something like S3, but at one third of a penny per gigabyte per month (heck, let's splash out - a whole penny per gigabyte per month) I know quite a few people who'll be interested in talking to them... (including myself)
- jacquesm 17y ago
  Apparently they use HTTPS for input/output using tomcat and some custom application. That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ? quote from the article: "A Backblaze Storage Pod isn’t a complete building block until it boots and is on the network. The pods boot 64-bit Debian 4 Linux and the JFS file system, and they are self-contained appliances, where all access to and from the pods is through HTTPS. Below is a layer cake diagram. Starting at the bottom, there are 45 hard drives exposed through the SATA controllers. We then use the fdisk tool on Linux to create one partition per drive. On top of that, we cluster 15 hard drives into a single RAID6 volume with two parity drives (out of the 15). The RAID6 is created with the mdadm utility. On top of that is the JFS file system, and the only access we then allow to this totally self-contained storage building block is through HTTPS running custom Backblaze application layer logic in Apache Tomcat 5.5." That's an odd choice for a storage server protocol stack.
  - sophacles 17y ago
    One thought I had regarding that -- their client talks to a "main server" management system, which talks to each pod on using the tomcat layer. The "main server" then replies to the client with an IP or similar to send data to directly. Since this goes over the web, https is used for security. Since https is already being used, it is not so difficult to use it all the way for authentication etc.
  - antonovka 17y ago
    That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ? To help prevent a network compromise from resulting in storage management compromise? Just because something is internal doesn't mean it's safe. Once a host/network segment is compromised, you don't want it to be easy to jump to the next. Otherwise, you've built M&M security. Hard candy shell on the outside, soft gooey chocolate insides. Mmm.
tricky 17y ago
effing love this. I wonder what the reliability stats on this setup is, though. Is it really cheaper to jam all those drives in one unit without redundant PSU's, MB's, or a boot drive? I'd guess you'd have to build at least 2 of these units and mirror them to get any sort of reliability. And, at that point, how long does it take to copy 58 TB across https? data is hard.
- skolor 17y ago
  I would assume they don't care about any of the hardware, just the data. If you look at the setup, and how the drives are raided, there 45 drives are sub-divided into 15 raid arrays, of which it would take 3 to die before they lost data. Essentially, they would need 20% of their drives to die simultaneously for them to lose data. Now, for the rest of the hardware, its not that important to if it fails. If one of the other components die, you're only looking at some down time (and a possible dead hard drive or two from a PSU dying, which I assume they monitor regularly). As long as the data is secure and in one piece, it doesn't really matter whether the pod is up or down, until someone needs the data. Just send out your repair guy to replace it and reboot it, and its fine.
- Andys 17y ago
  In sysadmin terms, they have aimed for data integrity over availability. If a system goes down and they have to replace a disk or a power supply or motherboard, the data is still safe, and if by chance there's any $5/month users who need to restore data held on that one unit, well they can wait a few hours :-)
byoung2 17y ago
Very interesting insight! I may try building a scaled-down version just for fun. The cost comparison between raw drives, their custom solution, and Amazon S3, etc was a little skewed. S3 is designed for pay as you go storage so you're not paying for capacity you don't need. If you just need a few dozen gigabytes, it's a much better deal. If you need terabytes or a petabyte, a dedicated storage solution is more economical. It's the same argument as vacation house vs timeshare. If you lived in a timeshare all year, it would cost more than buying the house.
dabeeeenster 17y ago
What happens when one of the PSU's fails catastrophically, pushing a big electrical surge through the hard disks, frying half of them? I didn't see anything in here that discusses that eventuality? And when you have that many servers, it IS going to happen at some point...
- notaddicted 17y ago
  It is called a fuse.
- papercrane 17y ago
  It's addressed in the section "A Backblaze Storage Pod is a Building Block" From the article: When you run a datacenter with thousands of hard drives, CPUs, motherboards, and power supplies, you are going to have hardware failures it’s irrefutable. Backblaze Storage Pods are building blocks upon which a larger system can be organized that doesn’t allow for a single point of failure.
  - dabeeeenster 17y ago
    Sounds to me like they haven't implemented this yet. I wouldn't want my data on that sort of solution. Backed up data would surely need a level of geographic redundancy.
    - papercrane 17y ago
      In the next section they say they have implemented the machine level redundancy. As for geographic redundancy I think the idea is that you use them for that. As in you backup your data locally and then send it to them as a redundant copy.
- ars 17y ago
  To get all the various logos and certifications on them, they are required not to do that. Every line is voltage limited.
sschueller 17y ago
How do you replace a drive? With so many drives it seems like a lot of effort to pull the entire server out to just access one drive.
- papercrane 17y ago
  They have three raid-6 volumes per machine, so they probably leave it in the rack until two of the volumes are unusable and then refurbish the machine.
  - rarrrrrr 17y ago
    I doubt it very much. When drives start failing out of a RAID6 array, you lose it's ability to automatically detect and correct single bit errors on disk. With that much data, single bit errors will happen predictably. If their usage patterns are anything like ours, disk failures are well under normal rates. Most backup data is stored and ignored. The drives just don't get much stress. Downing a machine for maintenance is probably acceptable to their usage pattern.
ShabbyDoo 17y ago
What terrified me was the number of low-level issues they had to address. SATA protocol problems, custom-designed SATA cards? Wow. I'd have no idea how to begin with that stuff. As other posters have noted, this company's business is storage. How many PB must one host for such specialization to make sense?
Bjoern 17y ago
What a pity, I'd like to have a way to store just a encrypted backup directly (via ssh) instead of installing a app which crawls my system. Oh did I mention I use Linux? sign Does somebody know any other company which provides this solution? I don't trust other people to crawl my system and to "encrypt" it for me. EDIT: Added Question
- californiaguy2 17y ago
  Yeah, rsync.net
- bestes 17y ago
  Did you look at tarsnap (http://www.tarsnap.com/ http://www.tarsnap.com/)? I don't use it, but read about it here quite a bit and it sounds really great.
- blhack 17y ago
  I'm not entirely sure if this is a troll or not, but... rsync does exactly what you're looking for. You can also pipe it over SSH encyption on the wire. type "man rsync" at a shell for a description of how.
- soult 17y ago
  How about http://allmydata.com/ http://allmydata.com/? They offer a JSON-based, restful API, an FTP interface or you can use their open source software which includes encryption.
preview 17y ago
I wonder about the single point of failure posed by the power supplies. One failed box is not a big deal (since I assume the data is replicated over several). But, what if they get a bad batch of supplies and see a relatively high failure rate? I wonder how high a power supply failure rate they can handle. The need to stagger the power on of the two supplies poses a problem. What if power to a data center is lost? When power is restored, all box will try to start, blowing all fuses. Granted, this is a catastrophic event, so its frequency should be very low. But, this also seems like an area that could be automated.
- lsc 17y ago
  You should only use 75% of the rated capacity of a circuit, which means you have enough power to turn them all on at once. some of the more expensive managed power supplies also support a staggered power on after power fail. But I don't worry about it; only using 75% of the power circuit solves that problem for me.
  - preview 17y ago
    Unfortunately, using 75% of the rated capacity may not be enough to handle the inrush. The article discusses this point, "...if you power up both PSUs at the same time, it can draw a large (14 amp) spike of 120V power from the socket." That would mean one pod per 20A circuit. Ugh. In normal operating conditions, a 5.6A max load would allow three pods per circuit. Addressing this would require a little bit of design, but the problem is relatively simple. If they wanted to get fancy, they could add a chaining feature--pods on the same circuit would be connected together so that they'd power on serially. This would get away from their goal of using off-the-shelf parts. It is, as with many things, an engineering trade-off.
    - lsc 17y ago
      The PDUs that support 'staggered power-on' are 'off the shelf' - if that is not an option (really, we're talking maybe $500 for each 20a circuit, retail) the next thing I'd do is set 'power on after power fail' to off, then have some remotely accessible way to trip the power button. (I'm working on a solution to that particular problem, but that's not 'off the shelf' - yeah, everything is on PDUs I can trip remotely, but there are reasons why it is much better to ungracefully reboot with the 'reset' jumper than to ungracefully reboot via cutting off the power.) from there it would be easy enough to have an automated process turn on servers one at a time.
thaumaturgy 17y ago
I have been searching high and low for something like this for months -- I have a number of clients that have been begging me for this, as well as something for my own needs. I'm excitedly posting a link to this on my personal site, and today I have a lot of phone calls to make to clients. Why is it that the best products and services are also the hardest to find when you're looking for them?
durana 17y ago
The price comparison with other solutions isn't really fair. The cost per PB of these storage building blocks is directly compared to complete storage solutions from companies like NetApp and EMC. In the end the cost of the complete solution they assemble these building blocks into may well be cheaper than solutions from other companies and that's the number they should be using for comparison.
datums 17y ago
Service Idea: S3 offsite backups. Run http://www.eucalyptus.com/open/ http://www.eucalyptus.com/open/ on this kind of hardware.
sh1mmer 17y ago
I'd love to hear what they are doing to monitor their system. They didn't mention that. 3 out of 15 drives failures to loose a volume seems quite likely at that scale. I'd like to know what levels of warnings and alarms they use with which system, e.g. nagios, etc.
sh1mmer 17y ago
One of the smart guys in our (Y!) cloud team just point out something to me that hadn't occurred to me. This system is definitely optimized for backup. That totally make sense for Backblaze. However it's important to not compare this like for like with something like S3 which is optimized for much better read/write performance. At the basic level the cooling on this system seems minimal. Those tightly packed drives would sure get hot if they were all spinning a lot. More than that since they are using commodity consumer hardware, and they already used up their PCIe slots for the SATA controllers there isn't any place to add anything more than the gitabit (I assume) ethernet jack on the mobo. That means there throughput is limited. Again, this is a great system for backup. Most of the data will just sit happily sipping little power. However, if you are thinking of this as equivalent to a filer, that's an unfair comparison.
- hyuen 17y ago
  I was thinking about the IOPS that you can sustain with this type of device, and as mentioned here, this is intended for backup, which may or may not be very good acting as a filer. Probably you can get some high IOPS with all these drives in parallel, but probably not as much as the other more expensive ones (EMC, Netapp). But I have to agree this is an awesome setup optimized for a specific application. Way to go
- Andys 17y ago
  Fast 120mm fans can move a pretty decent amount of air - up to 120 ft^3/min each, and they used three in parallel. It would take about a dozen 80mm normal-speed computer fans to reach this.
pmorici 17y ago
Seems along the same lines as Capricorn Tech. http://www.capricorn-tech.com/products.php http://www.capricorn-tech.com/products.php
- ggruschow 17y ago
  Not really. BackBlaze's post is about how to get on-line storage up for the lowest cost per bit. They appear to have done a great job at quadrupling the density of standard 1U setups for a similar price. Capricorn appears to be in the business of selling standard 4 drive per 1U setups and support. The fact you have to contact their sales department to even get close to a price seems to indicate they're not competing on price.
ALee 17y ago
These guys are very very good at what they do. Btw folks, no funding and they're already profitable. Gleb and crew rock!
jwilliams 17y ago
Awesome. Brings up a few questions for me (sure there are answers, just curious really). Why a Core 2? An Atom mobo would be lower power and cheaper. Why 4Gb? Seems like an overkill. They are using a HD to boot. Couldn't they boot of a USB key?
- yellowbkpk 17y ago
  I don't know for sure, but there might not be an Atom-based board that has as many PCI/PCIe slots.
  - jwilliams 17y ago
    Good point. The one I've got only has one PCI in fact.
silverscreen 17y ago
Great read! Why a seperated boot disk? Why can't that be on the storage array?
eleitl 17y ago
This made the Beowulf list yesterday, and below is what I wrote in response: "Seagate ST31500341AS 1.5TB Barracuda 7200.11 SATA 3Gb/s 3.5″ Aargh! Should be definitely substituted by 2 TByte WD RE4 drive. Today I've built a 32 TByte raw storage Supermicro box with X8DDAi (dual-socket Nehalem, 24 GByte RAM, IPMI), two LSI SAS3081E-R and OpenSolaris sees all (WD2002FYPS) drives so far (the board refuses to boot from DVD when more than 12 drives are in though probably to some BIOS brain damage, so you have to manually build a raidz-2 with all 16 drives in it once Solaris has booted up). The drives are about 3170 EUR sans VAT total for all 16, the box itself around 3000 EUR sans VAT. I presume Linux with RAID 6 would work (haven't checked yet), too, and if you need more you can use a cluster FS. Maybe not as cheap as a Backblaze, but off-shelf (BTO) and you get what you pay for.

401 points by bensummers 17y ago

dmillar 17y ago

This is unbelievably awesome. Can you imagine what we could achieve if every company had this level of transparency?

PStamatiou 17y ago
Thinking the exact same thing! While technology isn't more than any typical hacker can slap together in their apartment (minus maybe the case fab), I think the application is ingenious and the full blog post about it, like dmillar, pointed out is rather awe-inspiring. And I was all happy because my new Core i7 box has 2TB of space and 6GB of ram... sigh That being said - did seagate ever fix their firmware issue on the 1.5TB drives that would cause random corruption? (heard about it maybe 6 months ago)
- barrkel 17y ago
  I should think so - I have 4 of them in my ZFS pool, and scrubs are turning up no errors.
- sophacles 17y ago
  I have the 500GB in the same series (with the corrupted firmware) and the fix did very nice things for it. Also, I have never had problems with the 7200.12 (the newer version of the same drive) in a 1TB model.
- jbellis 17y ago
  > While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.
  - PStamatiou 17y ago
    I would have some smart-ass comment here but I just read your HN profile. Good day, Sir.
    - tzury 17y ago
      smart move ;-)
  - Periodic 17y ago
    This is what always gets in my way. It takes a lot of work and a lot of expertise to put together a home-grown system that works as well as one from the major vendors. If you're only going to be using one or two, you're much better off going to one of those major vendors because a large part of the price is their expertise and testing that went into it. For a large setup like Backblaze they can spread the cost of design over many systems, but for smaller companies it just isn't feasible. We hacker types love to think that we could do the same thing in no time with little budget, and I'm sure we could get a first approximation. But the devil is in the details. Debugging the complex interaction of 20 different hardware components is not my idea of fun. Hats off to them, particularly for sharing.
    - rantfoil 17y ago
      Secondary revenue streams anyone? I know we'd be interested in becoming a customer if Backblaze at some point decided to monetize this part of their knowledge too. This seems like an opportunity to disrupt from the low end of the market for some scrappy enterprising engineers. If they don't do it, someone should.
    - uhgygghhj 17y ago
      I did this as an inhouse backup for a data warehousing app. Just slapping 4 ide cards into a case and putting 16x250gb IDE drives on them resulted in a system that would copy about 1 disk worth before hanging with some fault or suddenly dropping to 1% speed. Just because you can in theory hook 40 drives to n cards doesn't mean it will work - well done to them
    - 17y ago
- bestes 17y ago
  Yes, they did a firmware update. Here is one article that talks about it: http://www.tomshardware.com/news/seagate-barracuda-1.5TB-freezing,6582.html http://www.tomshardware.com/news/seagate-barracuda-1.5TB-fre...
  - herf 17y ago
    Seagate took about 4 months to fully resolve the issue (their first fix had a bug!) I had a drive that bricked when applying the fix several months after the initial firmware updates. Apparently a buffer overflow in the error log that locked the drive on reboot. Seems dangerous to source all your hardware from one vendor over a small timeframe. Would be nice to have some redundancy over manufacturers too.
- lsc 17y ago
  as for the firmware 'bug' - yeah, they fixed that one, However, consumer sata disks are still not something you want in most raid systems. See, a consumer disk, when it encounters an unrecoverable error, retries over and over, hanging your raid until you fail the bad drive by hand. (tested on a 3ware hardware raid and with MD. I am given to understand that zfs handles this in a sane manner.) Western digital allows you to change the behavior, search for WDTLER, but in general. I'm sure the backblaze folks have built something that handles that sort of thing into the system, but just saying; it's something you have to think about before you slap consumer drives into a raid that needs to keep working when one drive fails.
  - Devilboy 17y ago
    The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive since the drives make up most of the hardware cost.
    - lsc 17y ago
      But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.
      - Devilboy 17y ago
        In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build large arrays to beat triple JBOD for storage efficiency - a 12 x 1TB array yields 5TB usable space while triple JBOD gives you 4TB. BUT you still have a single point of failure - your RAID card can give out and your array is unavailable. Using triple JBOD you'll be able to have your data on different physical machines or even geographically separate data centers if you wish. The advantages outweigh the reduction of storage density when you're dealing with petabytes of storage - why else would Google et al be doing it?

edw519 17y ago

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81,000 Backblaze $117,000 Dell $826,000 Sun $1,000,000 NetApp $1,714,000 Amazon $2,806,000 EMC $2,860,000 I had no idea. Kinda makes one rethink what business they want to be in.

spydez 17y ago
Well, to be fair, Backblaze is using consumer-grade hardware here. No redundant PSUs. No mirrored OS drives. Cheaper components. Etc. They're going Google's route, which is fine, it's just... a whole different direction than a Sun Thumper. I'd be interested to see how their costs per petabyte stack against Google's.
- blasdel 17y ago
  No, they aren't going Google's route! Google's operation has at most half the disks-per-rack density of Backblaze, but with 4 or 8 times the server density. Google does almost all their computation on the same nodes where the data is stored. Google is also storing each tablet a minimum of 3 times within each cluster, with most systems having multiple replica clusters. Backblaze's system is going to have several orders of magnitude less bandwidth: way too many drives in one server (some of them on the PCI bus!) use of port multipliers (causing 5 drives to share one SATA cable's 300MB/s) RAID6 with too many drives per array (15-way XOR is no fun) why use JFS? only access is via HTTPS, not clear if SSL is done on the pod
  - cheriot 17y ago
    I'm guessing that backups are accessed rather infrequently and that someone recovering their data won't mind a few seconds in lag time. This isn't my area of expertise, but their trade off seems sensible. It's just a different scenario than what google is dealing with.
cperciva 17y ago
Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids, etc. -- and any system which stores all of its data in a single datacenter lacks any protection against that. Store each file on two different servers in each of two different datecenters, and suddenly Backblaze's $134k turns into $536k. The price for Amazon, in contrast, is based on the assumption that their prices remain fixed for the next 3 years -- which seems a rather radical assumption. Is backblaze's solution cheaper than S3? Absolutely. But they're also twisting the numbers a bit.

modoc 17y ago

I love how open they are about their solution, and I also want one for Christmas:)

psranga 17y ago

Great stuff. Would anybody care to clarify why access is through HTTPS and not HTTP? I presume all accesses to these pods are from within their data center? Or do they directly expose these boxes to clients (whoa!)?

phsr 17y ago
BackBlaze runs online, off-site backups (like Mozy and Carbonite) at $5/month for unlimited storage. HTTPS is used to keep client data protected, I assume

tsuraan 17y ago

67TB of storage, with 4GB of cache. I'd really love to see some performance numbers versus the way-too-expensive competition. If the systems are being used as tape drive replacements, I could see this working well, but as an actual NAS-like device, I can't imaging how it could perform acceptably. Of course, if those Intel motherboards have the dual 1Gb/s NICs that Intel boards generally do, it will probably take a while to fill the drives anyhow.

notaddicted 17y ago
http://www.wolframalpha.com/input/?i=(67+TB)+%2F+(2*(1+Gb%2Fs)+ http://www.wolframalpha.com/input/?i=(67+TB)+%2F+(2*(1+Gb%2F...) It'll take 3 days at the theoretical max of the networking equipment to read/write the 67TB. The overhead of HTTPS constrains the networking so this is too low. I'd expect that their internet connection (i.e. in/out of the data center) is the real bottleneck. I believe that the system is being used as a tape drive replacement.
- lsc 17y ago
  If the load is relatively sequential, then yeah. But if the load is mostly random, then there is an argument for having more cache.
lsc 17y ago
Actually, that's the problem with most NetAPP/EMC foo, too. they are way light on the cache. These guys only use 4GiB ram because they are saving money on the motherboard, I bet. Personally, if I were building it, I'd increase the cost by another grand or so and use dual low power opterons with 32GiB ram. (of course, that would also increase the space taken by the motherboard, so that would require some case redesigns. Still, opterons and registered ecc ddr2 are both incredibly cheap right now.)
pmorici 17y ago
What if they used AndrewFS with SSD drives for cache on front end nodes.

gstar 17y ago

They might have missed a trick here. To address vibration, acoustics and gyroscopic effect, what I've seen done in highly dense enclosures is to rotate every second drive around 180 degrees in a bit of a shotgun approach to balancing stuff. Still, awesome.

sh1mmer 17y ago
This is the trick, though. They want to improve their system. By sharing with you they may have solicited some really helpful advice.
ciupicri 17y ago
This has sounded ok for me too, but when I've thought a bit about it I've realized that the drives must be synchronized so that each one would "cancel" each others vibrations. The vibrations generated by the drives might have indeed the same amplitude and (spatial and temporal) frequency, but I doubt that they'll have the same phase offset. For this to happen they would need to be started at _exactly_ the same time which I don't think that happens in practice. Do you have some pictures of those enclosures?
- gstar 17y ago
  I don't have pictures. I can describe, though. Two disks were mounted in a frame linearly, both screwed to the frame, with the power/sata connectors toward the middle of the frame, and one drive upside-down. These cassettes were removable as a unit for hot-swap, and were inserted linearly into a half-deep 19" rack enclosure. That the two drives were physically connected to the same frame, and removed and replaced as a unit would make it seem as if they would be started in phase. Now I'm not physicist, but I'm not 100% sure that's so important - if you have two gyroscopes contra-rotating and firmly connected, running at the same speed, surely they resist movement by sheer gyroscopic effect?
Andys 17y ago
Unfortunately the SATA backplanes are only aligned one way, so they could only rotate a set of 5 drives at once.

swombat 17y ago

Wow. Do they offer S3-like storage? They should. If they can offer something like S3, but at one third of a penny per gigabyte per month (heck, let's splash out - a whole penny per gigabyte per month) I know quite a few people who'll be interested in talking to them... (including myself)

jacquesm 17y ago
Apparently they use HTTPS for input/output using tomcat and some custom application. That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ? quote from the article: "A Backblaze Storage Pod isn’t a complete building block until it boots and is on the network. The pods boot 64-bit Debian 4 Linux and the JFS file system, and they are self-contained appliances, where all access to and from the pods is through HTTPS. Below is a layer cake diagram. Starting at the bottom, there are 45 hard drives exposed through the SATA controllers. We then use the fdisk tool on Linux to create one partition per drive. On top of that, we cluster 15 hard drives into a single RAID6 volume with two parity drives (out of the 15). The RAID6 is created with the mdadm utility. On top of that is the JFS file system, and the only access we then allow to this totally self-contained storage building block is through HTTPS running custom Backblaze application layer logic in Apache Tomcat 5.5." That's an odd choice for a storage server protocol stack.
- sophacles 17y ago
  One thought I had regarding that -- their client talks to a "main server" management system, which talks to each pod on using the tomcat layer. The "main server" then replies to the client with an IP or similar to send data to directly. Since this goes over the web, https is used for security. Since https is already being used, it is not so difficult to use it all the way for authentication etc.
- antonovka 17y ago
  That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ? To help prevent a network compromise from resulting in storage management compromise? Just because something is internal doesn't mean it's safe. Once a host/network segment is compromised, you don't want it to be easy to jump to the next. Otherwise, you've built M&M security. Hard candy shell on the outside, soft gooey chocolate insides. Mmm.

tricky 17y ago

effing love this. I wonder what the reliability stats on this setup is, though. Is it really cheaper to jam all those drives in one unit without redundant PSU's, MB's, or a boot drive? I'd guess you'd have to build at least 2 of these units and mirror them to get any sort of reliability. And, at that point, how long does it take to copy 58 TB across https? data is hard.

skolor 17y ago
I would assume they don't care about any of the hardware, just the data. If you look at the setup, and how the drives are raided, there 45 drives are sub-divided into 15 raid arrays, of which it would take 3 to die before they lost data. Essentially, they would need 20% of their drives to die simultaneously for them to lose data. Now, for the rest of the hardware, its not that important to if it fails. If one of the other components die, you're only looking at some down time (and a possible dead hard drive or two from a PSU dying, which I assume they monitor regularly). As long as the data is secure and in one piece, it doesn't really matter whether the pod is up or down, until someone needs the data. Just send out your repair guy to replace it and reboot it, and its fine.
Andys 17y ago
In sysadmin terms, they have aimed for data integrity over availability. If a system goes down and they have to replace a disk or a power supply or motherboard, the data is still safe, and if by chance there's any $5/month users who need to restore data held on that one unit, well they can wait a few hours :-)

byoung2 17y ago

Very interesting insight! I may try building a scaled-down version just for fun. The cost comparison between raw drives, their custom solution, and Amazon S3, etc was a little skewed. S3 is designed for pay as you go storage so you're not paying for capacity you don't need. If you just need a few dozen gigabytes, it's a much better deal. If you need terabytes or a petabyte, a dedicated storage solution is more economical. It's the same argument as vacation house vs timeshare. If you lived in a timeshare all year, it would cost more than buying the house.

dabeeeenster 17y ago

What happens when one of the PSU's fails catastrophically, pushing a big electrical surge through the hard disks, frying half of them? I didn't see anything in here that discusses that eventuality? And when you have that many servers, it IS going to happen at some point...

notaddicted 17y ago
It is called a fuse.
papercrane 17y ago
It's addressed in the section "A Backblaze Storage Pod is a Building Block" From the article: When you run a datacenter with thousands of hard drives, CPUs, motherboards, and power supplies, you are going to have hardware failures it’s irrefutable. Backblaze Storage Pods are building blocks upon which a larger system can be organized that doesn’t allow for a single point of failure.
- dabeeeenster 17y ago
  Sounds to me like they haven't implemented this yet. I wouldn't want my data on that sort of solution. Backed up data would surely need a level of geographic redundancy.
  - papercrane 17y ago
    In the next section they say they have implemented the machine level redundancy. As for geographic redundancy I think the idea is that you use them for that. As in you backup your data locally and then send it to them as a redundant copy.
ars 17y ago
To get all the various logos and certifications on them, they are required not to do that. Every line is voltage limited.

sschueller 17y ago

How do you replace a drive? With so many drives it seems like a lot of effort to pull the entire server out to just access one drive.

papercrane 17y ago
They have three raid-6 volumes per machine, so they probably leave it in the rack until two of the volumes are unusable and then refurbish the machine.
- rarrrrrr 17y ago
  I doubt it very much. When drives start failing out of a RAID6 array, you lose it's ability to automatically detect and correct single bit errors on disk. With that much data, single bit errors will happen predictably. If their usage patterns are anything like ours, disk failures are well under normal rates. Most backup data is stored and ignored. The drives just don't get much stress. Downing a machine for maintenance is probably acceptable to their usage pattern.

ShabbyDoo 17y ago

What terrified me was the number of low-level issues they had to address. SATA protocol problems, custom-designed SATA cards? Wow. I'd have no idea how to begin with that stuff. As other posters have noted, this company's business is storage. How many PB must one host for such specialization to make sense?

Bjoern 17y ago

What a pity, I'd like to have a way to store just a encrypted backup directly (via ssh) instead of installing a app which crawls my system. Oh did I mention I use Linux? sign Does somebody know any other company which provides this solution? I don't trust other people to crawl my system and to "encrypt" it for me. EDIT: Added Question

californiaguy2 17y ago
Yeah, rsync.net
bestes 17y ago
Did you look at tarsnap (http://www.tarsnap.com/ http://www.tarsnap.com/)? I don't use it, but read about it here quite a bit and it sounds really great.
blhack 17y ago
I'm not entirely sure if this is a troll or not, but... rsync does exactly what you're looking for. You can also pipe it over SSH encyption on the wire. type "man rsync" at a shell for a description of how.
soult 17y ago
How about http://allmydata.com/ http://allmydata.com/? They offer a JSON-based, restful API, an FTP interface or you can use their open source software which includes encryption.

preview 17y ago

I wonder about the single point of failure posed by the power supplies. One failed box is not a big deal (since I assume the data is replicated over several). But, what if they get a bad batch of supplies and see a relatively high failure rate? I wonder how high a power supply failure rate they can handle. The need to stagger the power on of the two supplies poses a problem. What if power to a data center is lost? When power is restored, all box will try to start, blowing all fuses. Granted, this is a catastrophic event, so its frequency should be very low. But, this also seems like an area that could be automated.

lsc 17y ago
You should only use 75% of the rated capacity of a circuit, which means you have enough power to turn them all on at once. some of the more expensive managed power supplies also support a staggered power on after power fail. But I don't worry about it; only using 75% of the power circuit solves that problem for me.
- preview 17y ago
  Unfortunately, using 75% of the rated capacity may not be enough to handle the inrush. The article discusses this point, "...if you power up both PSUs at the same time, it can draw a large (14 amp) spike of 120V power from the socket." That would mean one pod per 20A circuit. Ugh. In normal operating conditions, a 5.6A max load would allow three pods per circuit. Addressing this would require a little bit of design, but the problem is relatively simple. If they wanted to get fancy, they could add a chaining feature--pods on the same circuit would be connected together so that they'd power on serially. This would get away from their goal of using off-the-shelf parts. It is, as with many things, an engineering trade-off.
  - lsc 17y ago
    The PDUs that support 'staggered power-on' are 'off the shelf' - if that is not an option (really, we're talking maybe $500 for each 20a circuit, retail) the next thing I'd do is set 'power on after power fail' to off, then have some remotely accessible way to trip the power button. (I'm working on a solution to that particular problem, but that's not 'off the shelf' - yeah, everything is on PDUs I can trip remotely, but there are reasons why it is much better to ungracefully reboot with the 'reset' jumper than to ungracefully reboot via cutting off the power.) from there it would be easy enough to have an automated process turn on servers one at a time.

thaumaturgy 17y ago

I have been searching high and low for something like this for months -- I have a number of clients that have been begging me for this, as well as something for my own needs. I'm excitedly posting a link to this on my personal site, and today I have a lot of phone calls to make to clients. Why is it that the best products and services are also the hardest to find when you're looking for them?

durana 17y ago

The price comparison with other solutions isn't really fair. The cost per PB of these storage building blocks is directly compared to complete storage solutions from companies like NetApp and EMC. In the end the cost of the complete solution they assemble these building blocks into may well be cheaper than solutions from other companies and that's the number they should be using for comparison.

datums 17y ago

Service Idea: S3 offsite backups. Run http://www.eucalyptus.com/open/ http://www.eucalyptus.com/open/ on this kind of hardware.

sh1mmer 17y ago

I'd love to hear what they are doing to monitor their system. They didn't mention that. 3 out of 15 drives failures to loose a volume seems quite likely at that scale. I'd like to know what levels of warnings and alarms they use with which system, e.g. nagios, etc.

sh1mmer 17y ago

One of the smart guys in our (Y!) cloud team just point out something to me that hadn't occurred to me. This system is definitely optimized for backup. That totally make sense for Backblaze. However it's important to not compare this like for like with something like S3 which is optimized for much better read/write performance. At the basic level the cooling on this system seems minimal. Those tightly packed drives would sure get hot if they were all spinning a lot. More than that since they are using commodity consumer hardware, and they already used up their PCIe slots for the SATA controllers there isn't any place to add anything more than the gitabit (I assume) ethernet jack on the mobo. That means there throughput is limited. Again, this is a great system for backup. Most of the data will just sit happily sipping little power. However, if you are thinking of this as equivalent to a filer, that's an unfair comparison.

hyuen 17y ago
I was thinking about the IOPS that you can sustain with this type of device, and as mentioned here, this is intended for backup, which may or may not be very good acting as a filer. Probably you can get some high IOPS with all these drives in parallel, but probably not as much as the other more expensive ones (EMC, Netapp). But I have to agree this is an awesome setup optimized for a specific application. Way to go
Andys 17y ago
Fast 120mm fans can move a pretty decent amount of air - up to 120 ft^3/min each, and they used three in parallel. It would take about a dozen 80mm normal-speed computer fans to reach this.

pmorici 17y ago

Seems along the same lines as Capricorn Tech. http://www.capricorn-tech.com/products.php http://www.capricorn-tech.com/products.php

ggruschow 17y ago
Not really. BackBlaze's post is about how to get on-line storage up for the lowest cost per bit. They appear to have done a great job at quadrupling the density of standard 1U setups for a similar price. Capricorn appears to be in the business of selling standard 4 drive per 1U setups and support. The fact you have to contact their sales department to even get close to a price seems to indicate they're not competing on price.

ALee 17y ago

These guys are very very good at what they do. Btw folks, no funding and they're already profitable. Gleb and crew rock!

jwilliams 17y ago

Awesome. Brings up a few questions for me (sure there are answers, just curious really). Why a Core 2? An Atom mobo would be lower power and cheaper. Why 4Gb? Seems like an overkill. They are using a HD to boot. Couldn't they boot of a USB key?

yellowbkpk 17y ago
I don't know for sure, but there might not be an Atom-based board that has as many PCI/PCIe slots.
- jwilliams 17y ago
  Good point. The one I've got only has one PCI in fact.

silverscreen 17y ago

Great read! Why a seperated boot disk? Why can't that be on the storage array?

eleitl 17y ago

This made the Beowulf list yesterday, and below is what I wrote in response: "Seagate ST31500341AS 1.5TB Barracuda 7200.11 SATA 3Gb/s 3.5″ Aargh! Should be definitely substituted by 2 TByte WD RE4 drive. Today I've built a 32 TByte raw storage Supermicro box with X8DDAi (dual-socket Nehalem, 24 GByte RAM, IPMI), two LSI SAS3081E-R and OpenSolaris sees all (WD2002FYPS) drives so far (the board refuses to boot from DVD when more than 12 drives are in though probably to some BIOS brain damage, so you have to manually build a raidz-2 with all 16 drives in it once Solaris has booted up). The drives are about 3170 EUR sans VAT total for all 16, the box itself around 3000 EUR sans VAT. I presume Linux with RAID 6 would work (haven't checked yet), too, and if you need more you can use a cluster FS. Maybe not as cheap as a Backblaze, but off-shelf (BTO) and you get what you pay for.