13 ms·

Scaling the Facebook data warehouse to 300 PB

128 points by ot 13y ago

andrewchoi 13y ago
When they introduced ORCFile, I was kind of hoping that the next iteration would be the URUKFile.
- zerd 13y ago
  Or HUMANFile.
wehadfun 13y ago
The things facebook does to keep people's shower thoughts readily available.
- ihsw 13y ago
  > The things facebook does to keep people's shower thoughts readily available to advertisers. FTFY.
- Theodores 13y ago
  ...not to mention our uber-lords at the NSA... (yawn) Actually all tech can be criticised for banality. Think of the telephone - 'they put in all those cables just so she can talk to mother...'. Or television - 'they dug up teh street just so they could lay those fibre optic cables so your gran could watch the wresting...'. Or even the trains - the train stopping at my home town does seem a waste of time, I can't believe they bother when you look at who gets on.
  - tedks 13y ago
    It's a fallacy that all technology can be criticized for banality. Technologies are different. Some are more criticizable than others. Telephone was the first infrastructure to provide real-time voice communication. It enables families staying in contact, but it also enables economic growth, and a more effective society writ large. Television is now a mindless wastefield of race-to-the-bottom drivel, but there are newer networks have haven't yet succumbed to drive, mostly on digital cable. I only have the respect for science and technology that I do because I grew up watching The Magic School Bus and Bill Nye the Science Guy. My parents watched the moon landing on television. Facebook does not do anything novel, nor has it ever been used for anything terrifically insightful. It provides some social value and exists for that reason, but it is clearly not equivalent to all other technologies.
    - nemothekid 13y ago
      I don't quite get it. Being able to transmit images is "novel" but being able to query more information that has ever been available before in the history of the human race is not? You should really separate the application of the technology from the technology itself. Your last sentence can be said the exact same way about the telephone "It provides some social value and exists for that reason" but that is clearly a ridiculous statement to make about the telephone.
  - wehadfun 13y ago
    My comment was more of a dig at things like banks which barely will let you view your financial transactions more than a year or two out. Yet I can pull up someones stupid rant about a basketball game from 7+ years ago on facebook.
Skywing 13y ago
Based on my current connections, and experience, I can't fully see myself ever getting the chance to do so, but I'd love to have the opportunity to work on similar problems - at Facebook or anywhere. I've always loved working on lower level problems such as this. I learned to program by writing computer game hacks, reverse engineering games and coding in C / ASM. Currently, I find myself writing C# every day for a small company at which nobody understands a single word I say about programming.
- rdl 13y ago
  Yeah, it's interesting how Facebook does something which on its face seems trivial and unimportant, but due to scale, has some really amazing engineering and infrastructure challenges -- and they solve them by building tech to make things scale from commodity systems (like Google, etc have done), vs. buying third-party big iron solutions (which is what eBay did when they had similar challenges, and what most big companies do).
  - salemh 13y ago
    Do you think its a cost/control to not buy big iron (outsource to a degree) vs build/scale themselves? I'd like to hear some HN thoughts if some don't mind sharing.
    - applecore 13y ago
      If you're a pure technology company, like Google and Facebook, you're not going to outsource your core competency. It's not an issue of costs; buying "big iron", even if they could, would be akin to style drift for an investment manager.
      - rdl 13y ago
        You could definitely still do a lot less in-house than FB does, and be successful. FB seems to delight in building tools and infrastructure. Bloomberg is probably a better example of a company which builds "optional" technology in house, just to be awesome, though -- they're not at the scale of FB (where "traditional" solutions break down), but from what I've seen, they do a lot of interesting work in-house because their staff want to do it, and because it lets them have really top-quality staff in a highly competitive market.
oregon_engineer 13y ago
Where's the power coming from to scale like that? If Facebook is relying on power from the Columbia River for their data centers, they better talk to the tribes and the salmon. Those dams are failing and many parties in the Northwest want to see them removed. A better bet might be for them to scale up in Utah and harvest methane from waste to generate power.
- taftster 13y ago
  Because Utah has a lot of methane? I'm not quite following. I'd think methane production would be greater in a cow state like Iowa or Ohio than in a desert state like Utah? joke Utah's power grid is primarily coal fired, but they use a fair bit of natural (methane) gas as well. [1] Utah has a couple of major hydroelectric dams (glen canyon, flaming gorge), and the use of solar and wind are on the rise. NSA is building a major data warehouse in Utah; one of the considerations would have to have been cheap power. I'm guessing the Columbia river produces cheaper electricity than pretty much anywhere else, but Utah has very diverse (and affordable) power production overall. [1] http://www.deseretnews.com/article/700051087/Coal-mostly-powers-electricity-hungry-Utah.html?pg=all http://www.deseretnews.com/article/700051087/Coal-mostly-pow... [edit] failed on humor the first time, trying again.
pokstad 13y ago
A lot of fancy tech for only 805MB per user (based on estimate of 500 million users). What if we had a pure p2p web app using webrtc & local storage to replace Facebook? Think everyone could spare 805MB? Calculation: https://www.google.com/search?q=(300PB)%2F(400000000) https://www.google.com/search?q=(300PB)%2F(400000000)
- tsax 13y ago
  What about persistence?
  - MasterScrat 13y ago
    There was a trend a few years ago where users would deactivate their FB accounts when logging out, so that people couldn't see/interact with them while they were away. http://edition.cnn.com/2010/TECH/social.media/11/12/facebook.superlogoff/ http://edition.cnn.com/2010/TECH/social.media/11/12/facebook... So maybe that's not as stupid as it sounds. You could also have some level of caching, eg I visited your pictures yesterday so people can get it from me for some time if you're away. I definitely see potential in this form of "distributed but controlled" storage mechanisms. Similarly, I like the idea of having a local backup drive, and another copy at a friend's place (possibly encrypted).
- kosievdmerwe 13y ago
  You're a bit behind, it's 1.23 billion monthly active users or if you feel that's a bit disingenuous: 757 million daily active users. Source: http://newsroom.fb.com/company-info/ http://newsroom.fb.com/company-info/ PS. you calculated using 400 million.
  - e12e 13y ago
    300 pb / 10^9 ~ 300 10^15B/10^9 ~ 300 10^6 B ~ 300MB/user. Considering my latest fb backup was ~18MB (unzipped), out of which 14MB was pictures, this doesn't sound too unreasonable to me. If anything, it sounds very conservative. If I was actively using fb for photos, I'd easily have at least 100 times as many, maybe 200 times as many. Not to mention that the single (short) video I've uploaded is 1.4 MB.
siculars 13y ago
And the moat keeps getting larger. Good for them. Yes, they are kind enough to open source this, but how advanced is the tech that they don't? And how much time have they enjoyed as the first to benefit from this impressive tech? Again, good for them.
ForHackernews 13y ago
Maybe they should just delete something once in a while. How about actually deleting data for users that close their account?
- deleted 13y ago
  [deleted]
NicoJuicy 13y ago
I'm actually wondering, are their any ways to play with data like this (eg. downloading data from StackExchange http://blog.stackexchange.com/category/cc-wiki-dump/ http://blog.stackexchange.com/category/cc-wiki-dump/ ) Any other ways? I don't believe there is a VM for this sort of "experimentation"
- zackangelo 13y ago
  http://aws.amazon.com/datasets http://aws.amazon.com/datasets
kakoni 13y ago
I wonder, how does vertica link to this. (http://www.vertica.com/2013/12/12/welcoming-facebook-to-the-growing-family-of-hp-vertica-customers http://www.vertica.com/2013/12/12/welcoming-facebook-to-the-...)

128 points by ot 13y ago

andrewchoi 13y ago

When they introduced ORCFile, I was kind of hoping that the next iteration would be the URUKFile.

zerd 13y ago
Or HUMANFile.

wehadfun 13y ago

The things facebook does to keep people's shower thoughts readily available.

ihsw 13y ago
> The things facebook does to keep people's shower thoughts readily available to advertisers. FTFY.
Theodores 13y ago
...not to mention our uber-lords at the NSA... (yawn) Actually all tech can be criticised for banality. Think of the telephone - 'they put in all those cables just so she can talk to mother...'. Or television - 'they dug up teh street just so they could lay those fibre optic cables so your gran could watch the wresting...'. Or even the trains - the train stopping at my home town does seem a waste of time, I can't believe they bother when you look at who gets on.
- tedks 13y ago
  It's a fallacy that all technology can be criticized for banality. Technologies are different. Some are more criticizable than others. Telephone was the first infrastructure to provide real-time voice communication. It enables families staying in contact, but it also enables economic growth, and a more effective society writ large. Television is now a mindless wastefield of race-to-the-bottom drivel, but there are newer networks have haven't yet succumbed to drive, mostly on digital cable. I only have the respect for science and technology that I do because I grew up watching The Magic School Bus and Bill Nye the Science Guy. My parents watched the moon landing on television. Facebook does not do anything novel, nor has it ever been used for anything terrifically insightful. It provides some social value and exists for that reason, but it is clearly not equivalent to all other technologies.
  - nemothekid 13y ago
    I don't quite get it. Being able to transmit images is "novel" but being able to query more information that has ever been available before in the history of the human race is not? You should really separate the application of the technology from the technology itself. Your last sentence can be said the exact same way about the telephone "It provides some social value and exists for that reason" but that is clearly a ridiculous statement to make about the telephone.
- wehadfun 13y ago
  My comment was more of a dig at things like banks which barely will let you view your financial transactions more than a year or two out. Yet I can pull up someones stupid rant about a basketball game from 7+ years ago on facebook.

Skywing 13y ago

Based on my current connections, and experience, I can't fully see myself ever getting the chance to do so, but I'd love to have the opportunity to work on similar problems - at Facebook or anywhere. I've always loved working on lower level problems such as this. I learned to program by writing computer game hacks, reverse engineering games and coding in C / ASM. Currently, I find myself writing C# every day for a small company at which nobody understands a single word I say about programming.

rdl 13y ago
Yeah, it's interesting how Facebook does something which on its face seems trivial and unimportant, but due to scale, has some really amazing engineering and infrastructure challenges -- and they solve them by building tech to make things scale from commodity systems (like Google, etc have done), vs. buying third-party big iron solutions (which is what eBay did when they had similar challenges, and what most big companies do).
- salemh 13y ago
  Do you think its a cost/control to not buy big iron (outsource to a degree) vs build/scale themselves? I'd like to hear some HN thoughts if some don't mind sharing.
  - applecore 13y ago
    If you're a pure technology company, like Google and Facebook, you're not going to outsource your core competency. It's not an issue of costs; buying "big iron", even if they could, would be akin to style drift for an investment manager.
    - rdl 13y ago
      You could definitely still do a lot less in-house than FB does, and be successful. FB seems to delight in building tools and infrastructure. Bloomberg is probably a better example of a company which builds "optional" technology in house, just to be awesome, though -- they're not at the scale of FB (where "traditional" solutions break down), but from what I've seen, they do a lot of interesting work in-house because their staff want to do it, and because it lets them have really top-quality staff in a highly competitive market.

oregon_engineer 13y ago

Where's the power coming from to scale like that? If Facebook is relying on power from the Columbia River for their data centers, they better talk to the tribes and the salmon. Those dams are failing and many parties in the Northwest want to see them removed. A better bet might be for them to scale up in Utah and harvest methane from waste to generate power.

taftster 13y ago
Because Utah has a lot of methane? I'm not quite following. I'd think methane production would be greater in a cow state like Iowa or Ohio than in a desert state like Utah? joke Utah's power grid is primarily coal fired, but they use a fair bit of natural (methane) gas as well. [1] Utah has a couple of major hydroelectric dams (glen canyon, flaming gorge), and the use of solar and wind are on the rise. NSA is building a major data warehouse in Utah; one of the considerations would have to have been cheap power. I'm guessing the Columbia river produces cheaper electricity than pretty much anywhere else, but Utah has very diverse (and affordable) power production overall. [1] http://www.deseretnews.com/article/700051087/Coal-mostly-powers-electricity-hungry-Utah.html?pg=all http://www.deseretnews.com/article/700051087/Coal-mostly-pow... [edit] failed on humor the first time, trying again.

pokstad 13y ago

A lot of fancy tech for only 805MB per user (based on estimate of 500 million users). What if we had a pure p2p web app using webrtc & local storage to replace Facebook? Think everyone could spare 805MB? Calculation: https://www.google.com/search?q=(300PB)%2F(400000000) https://www.google.com/search?q=(300PB)%2F(400000000)

tsax 13y ago
What about persistence?
- MasterScrat 13y ago
  There was a trend a few years ago where users would deactivate their FB accounts when logging out, so that people couldn't see/interact with them while they were away. http://edition.cnn.com/2010/TECH/social.media/11/12/facebook.superlogoff/ http://edition.cnn.com/2010/TECH/social.media/11/12/facebook... So maybe that's not as stupid as it sounds. You could also have some level of caching, eg I visited your pictures yesterday so people can get it from me for some time if you're away. I definitely see potential in this form of "distributed but controlled" storage mechanisms. Similarly, I like the idea of having a local backup drive, and another copy at a friend's place (possibly encrypted).
kosievdmerwe 13y ago
You're a bit behind, it's 1.23 billion monthly active users or if you feel that's a bit disingenuous: 757 million daily active users. Source: http://newsroom.fb.com/company-info/ http://newsroom.fb.com/company-info/ PS. you calculated using 400 million.
- e12e 13y ago
  300 pb / 10^9 ~ 300 10^15B/10^9 ~ 300 10^6 B ~ 300MB/user. Considering my latest fb backup was ~18MB (unzipped), out of which 14MB was pictures, this doesn't sound too unreasonable to me. If anything, it sounds very conservative. If I was actively using fb for photos, I'd easily have at least 100 times as many, maybe 200 times as many. Not to mention that the single (short) video I've uploaded is 1.4 MB.

siculars 13y ago

And the moat keeps getting larger. Good for them. Yes, they are kind enough to open source this, but how advanced is the tech that they don't? And how much time have they enjoyed as the first to benefit from this impressive tech? Again, good for them.

ForHackernews 13y ago

Maybe they should just delete something once in a while. How about actually deleting data for users that close their account?

deleted 13y ago
[deleted]

NicoJuicy 13y ago

I'm actually wondering, are their any ways to play with data like this (eg. downloading data from StackExchange http://blog.stackexchange.com/category/cc-wiki-dump/ http://blog.stackexchange.com/category/cc-wiki-dump/ ) Any other ways? I don't believe there is a VM for this sort of "experimentation"

zackangelo 13y ago
http://aws.amazon.com/datasets http://aws.amazon.com/datasets

kakoni 13y ago

I wonder, how does vertica link to this. (http://www.vertica.com/2013/12/12/welcoming-facebook-to-the-growing-family-of-hp-vertica-customers http://www.vertica.com/2013/12/12/welcoming-facebook-to-the-...)