9 ms·
Under the hood: Facebook’s cold storage system
- emocakes 11y agothe amount of engineering at this scale is just insane. kudos to facebook engineering.
- carb 11y agoResurrecting a dead comment: "the amount of engineering at this scale is just insane. kudos to facebook engineering." - emocakes
- akurilin 11y agoYeah, it's pretty unbelievable. Is the idea here that they're keeping these storages as general-purpose as possible to serve many different use cases and survive the test of time and change? Or do they specialize these data centers for ONE specific feature? I wouldn't want to be in the shoes of the people designing the full integration of the system in the latter scenario, imagine your use-case changes... that's a whole new level of refactoring, especially once you go into special-purpose hardware.
- brettproctor 11y agoYes, the idea is to be as general purpose as possible. That was one of the motivations for avoiding tape.
- jtchang 11y agoMy guess is this is what Google and Amazon are also doing for Nearline and Glacier. When will SSD drives catch up in storage capacity and price to traditional spinning platters? A SSD consumes very little power so this type of setup will not be as necessary. Also what about the life cycle of drives that get powered on and off repeatedly? On one hand you have backblaze running consumer off the shelf drives 24/7. On the other you have FB/Google/Amazon powering on and off these drives.
- wmf 11y agoWhen will SSD drives catch up in storage capacity and price to traditional spinning platters? Never. Someone did calculate that if you can get flash that lasts 15 years it would be cheaper than disk, though (because the disk has to be replaced). I can't find the links on those. Also what about the life cycle of drives that get powered on and off repeatedly? We wrote a paper on that topic where we treated start-stop cycles as a resource to be rationed with a token bucket filter: http://storage-conference.org/2011/Papers/Research/10.Felter.pdf http://storage-conference.org/2011/Papers/Research/10.Felter...
- JunkDNA 11y ago"Never" is a pretty log time. Curious what reasons you have for making that statement?
- rasz_pl 11y agophysics
- dexterdog 11y agoBut a flash disk that is healthy after 14 years is taking up probably 4x or more the DC space and power at a lower speed of a replacement disk at the end of its life span. The power part is probably not that significant, but the space and port availability is. 15 years is an eternity in hardware.
- stox 11y agoIn this environment, the power cycles are going to be less stressful than typical, as you won't see the degree of thermal cycling you do in a hot chassis. I'd love to see what the data says a year out.
- narsil 11y ago> My guess is this is what Google and Amazon are also doing for Nearline and Glacier. Pretty likely. One of the post's authors, Kestutis Patiejunas, was an architect working on Glacier a few years ago. (src: https://www.linkedin.com/in/kestutisp https://www.linkedin.com/in/kestutisp )
- RyJones 11y agoIt looks like every four days Facebook gets a Flickr worth of photos.
- dummyfellow 11y agoseems FB will be bigger then google, once they launch the video service.
- cloudwalking 11y agoGoogle's a lot bigger than you imagine.
- thrownaway2424 11y agoDoubtful. This article describes only 2EB of storage, while Google was estimated to have 15EB of storage two years ago. Another comparison with Google would be regarding the erasure coding technique described in this blog post. Google offhandedly mentioned using it in a 2010 presentation. Is this really Facebook's first deployment of erasure coding?
- brettproctor 11y agoNo. We published an article of doing raid in hdfs in june 2014: https://code.facebook.com/posts/536638663113101/saving-capacity-with-hdfs-raid/ https://code.facebook.com/posts/536638663113101/saving-capac...
- sidcool 11y agoThat's true, it's insane amount of photos. Two billion daily!!!
- curiousDog 11y agoWould be interesting to see stats on how often they had to resort to cold storage? Also, how do they go about thoroughly testing such a service? This is a main advantage of having 3 replicas, you can test HA in your statement in a straight-forward way. How does that work with the Reed-Solomon system? Also, kudos to Facebook for publishing articles to this to the public. A lot of companies keep this stuff secret.
- brettproctor 11y agoThis quote from the article talks about data verification. Although we all know that a backup isn't really a backup until you've tested restoring from it :-) """ To tackle this, we built a background “anti-entropy” process that detects data aberrations by periodically scanning all data on all the drives and reporting any detected corruptions. Given the inexpensive drives we would be using, we calculated that we should complete a full scan of all drives every 30 days or so to ensure we would be able to re-create any lost data successfully. Once an error was found and reported, another process would take over to read enough data to reconstruct the missing pieces and write them to new drives elsewhere. This separates the detection and root-cause analysis of the failure from reconstructing and protecting the data at hand. As a result of doing repairs in this distributed fashion, we were able to reduce reconstruction from hours to minutes. """
- meesterdude 11y agoI remember par2 from usenet, seems like similar/same concept http://www.quickpar.org.uk/ http://www.quickpar.org.uk/ and something i've been wondering about for a while for storage; glad to see it being implemented at such a scale! makes good sense. Though I would be curious how corruption/rebuild plays out over time. Also impressive how they shaved off all that extra power requirements and built-in expected unreliability. And really, utility is pretty darn reliable anyway, mixed with non-customer facing storage. Good move.
- whyoh 11y agoQuickPar hasn't been updated in a long time. Check out MultiPar for an actively developed alternative. (And of course there are other programs that handle PAR2 files as well.)
- acqq 11y agoPar, Par2, Par3, MultiPar, QuickPar and formats explained at: http://en.wikipedia.org/wiki/Parchive http://en.wikipedia.org/wiki/Parchive
- fla 11y agoReed–Solomon error correction is one of these things that still feels like black magic to me.
- qrmn 11y agoThey're not even the most arcane. Consider reading up on rapid tornado (Raptor) codes, other fountain codes, Goppa codes… there are whole families of error-correcting codes with different sets of tradeoffs. And the decoding can be enlightening, too: the Viterbi algorithm and its progeny can be surprisingly useful. Then blow your mind even further by looking up the principles of the McEliece cryptosystem (but don't use it as originally specified - modern developments have weakened and refined it a lot). I warn you: this is one of those down-the-rabbit-hole subjects that has a deliciously large amount of literature to soak up, you could find yourselves with bookshelves full of it. (An alarming number of patents, too, for the unwary.)
- tmdp 11y agoWhat a waste of electricity
- thrownaway122 11y agoAt some point humanity will have to face the question. "Is it worth storing so many cat pictures and selfies?"
- pbhjpbhj 11y agoI wonder - looking at the data they have now - how much of the data stored has never been used by ordinary users. There's got to be a lot of cruft that could be trimmed. In the past people would keep letters they felt were important. Now, with the network it's hard to tell what's important, a user might prune items that other users want to keep or might prune items that later become more important. Is there really enough value in keeping everything. I'm a consumate hoarder, I've got 2 decades of emails saved (but not all emails!) ... even I think perhaps we've gone over the top. Perhaps it's good for social history. Or maybe in the future for a Black Mirror like reconstruction of people's personalities in an AI. Doesn't there need to be limits on what we keep? [Meta: the parents is a valid remark and one that I feel adds to the complexion of the scenario under consideration; perhaps it could have been made better, fleshed out, but still. Wish HN - as a population - would value those that add to the conversation in this way.]
- bkeroack 11y ago"...since we were using low-end commodity storage that was by no means enterprise-quality." If Facebook (and I believe it's similar with other big players like Google and Amazon) are not using "enterprise grade" hardware (because it doesn't make economic sense at scale), who is? And more importantly, why? Why do "enterprise grade" products even exist if the largest and most deep-pocketed corporations don't find them to be good value propositions?
- outside1234 11y agoBecause Facebook, Google, and Amazon have the engineering talent to build their own. No "ordinary" enterprise (basically everyone else) can just say "oh, mounting and unmounting is an issue? let's use a raw disk instead!" -- that is just well beyond their capabilities. Also, only Facebook / Google / Microsoft / Amazon have the scale that designing your own cold storage makes sense.
- fragmede 11y agoNot really. Skipping the filesystem layer and using the raw disk has been an optimization for databases (ms-sql, mysql, oracle db) has been a practice for quite a while.
- xyzzyz 11y agoMicrosoft and Oracle aren't ordinary enterprises either.
- raverbashing 11y agoThere's a joke that the O_DIRECT flag on Open(2) in Linux should be changed to O_RACLE
- pkz 11y agoThis is very true. I work in the public sector and while we often need to store petabytes of data we have a hard time finding and keeping engineering talent to do stuff like this. Instead we buy expensive enterprise stuff. I find it frustrating.
- sengork 11y agoFacebook went quiet on their Blu-ray library cold storage method.
- krig 11y agoThere was an interesting presentation about the cold storage system used at Alibaba on openSUSE conf 15. Unfortunately the video isn't online yet, but they have a system that allows them to turn off power to the cold storage servers and only wake the disks up on a monthly schedule. They also use erasure coding and have calculated that they won't have to replace any disks for 5 years if the MTBF numbers from the vendors are correct: https://events.opensuse.org/conference/osc15/proposal/556 https://events.opensuse.org/conference/osc15/proposal/556
- philipw 11y agohttp://events.linuxfoundation.org/sites/events/files/slides/LFVault2015_Alibaba.pdf http://events.linuxfoundation.org/sites/events/files/slides/... The slides from Vault 2015 talk about Sheepdog on low power hardware and SMR drives at Alibaba.
- amelius 11y agoSo in what respect are generic solutions such as glusterfs lacking? Only the cold storage? And would that be easy to fix?
- devonkim 11y agoEven the companies that have $100M+ allocated for storage won't necessarily be able nor even want to hire the kind of developers that could create their own data storage system in-house - they've already invested so much of their IT staff into buying into their SAN infrastructure. With most companies outside tech and $100M+ available for IT, they'll just send it to paying for better developers on things that offer better growth potential like big data analytics or whatever because IT is a cost center if your revenue does not directly depend upon reliability of your information systems, and with limited engineering talent available at non-tech companies you'd rather have them work on things that directly contribute to raising the top line. IT is about the bottom line until a business decides IT is not a cost center for them.
- louwrentius 11y ago"For example, one of our test production runs hit a complete standstill when we realized that the data center personnel simply could not move the racks. Since these racks were a modification of the OpenVault system, we used the same rack castors that allowed us to easily roll the racks into place. But the inclusion of 480 4 TB drives drove the weight to over 1,100 kg, effectively crushing the rubber wheels. This was the first time we'd ever deployed a rack that heavy, and it wasn't something we'd originally baked into the development plan!" That's just fun. In some way I'm amazed that nobody of the smart people at FB tough of the weight of this stuff. But then again, I probably made sillier mistakes myself on a much smaller scale.
- mbubb 11y agoFascinating stuff. My guess is that they were definitely aware of the weight tolerance of the floor and the tiles but looks like the weak spot was the canister wheels on the bottom of the rack. I wonder how they eventually got it out. They probably used one of those rack jacks to get a stronger dolly underneath... Or removed the HDDs and moved it. This made me wonder about something else - the power density. They say the layout: "can support up to one exabyte (1,000 PB) per data hall. Since storage density generally increases as technology advances, this was the baseline from which we started. In other words, there's plenty of room to grow." I wonder how they account for increased energy consumption as they put in larger powersupplies to deal with the increase in power to run and cooling for the larger HDDs. I imagine in this layout everything is in proportion. CPU / RAM / Disk Space. It must be hard to accurately target growth with in the constraints of a limited power footprint.
- pcl 11y agoIt sounds like they actually reduced the overall power requirements significantly, by only delivering power to a subset of the drives at any one time.
- andyidsinga 11y agoafaik - as density increases per disk, ex going from 1TB to 4TB, power efficiency per TB should be better too as long as they are also not also increasing the total number of spindles by adding new storage nodes. One thing I'm not sure of is if _total load_ for the hall increases as increased accesses may occur because of having more density. The hard disk power management part is really fascinating in terms of keeping the storage cluster software alive and well to be able to respond to requests while keep the hard disks off. A low power SSD cache tier on the front may be part of the solution.
- revelation 11y agoYou know, Facebook, you can just buy these things from companies that spend all of their R&D exclusively on making them. Not invented here..
- philipw 11y agoYou say this but there's a huge difference of the typical enterprise storage solution and what the internet giants require. Only recently are those vendors realising this...
- cmurf 11y agoThose are arguably antiquated models. Even if you don't agree with that, those models are predicated on companies that don't actually know what they're doing when it comes to storage, which is why they defer to others and pay the markup that comes with it. Google, Amazon, Microsoft, Apple, Facebook, many others (and growing), this is their business now, to know how to reliably store things, do it in-house in a way that works for their use case. I don't use Facebook at all, but I appreciate the many investments they've made in open hardware and open source software. They don't give away everything. But they definitely contribute in significant ways.