6 ms·

How Facebook Copes with Scale: Three Approaches

66 points by Katydid 11y ago

ckluis 11y ago
40 PB a day for photos...
- mikegioia 11y ago
  This is the part that I have trouble wrapping my head around. If you're adding 40 PB of photos per day, and if you can somehow get say $0.01/GB [1] then that's adding $400,000 per day in storage costs. Facebook doesn't/can't/won't delete photos so how on earth is this sustainable? After 10-20 years wouldn't hosting costs just be too high to function? [1] Aaverage cost per GB in 2014 was $0.03: http://www.statisticbrain.com/average-cost-of-hard-drive-storage/ http://www.statisticbrain.com/average-cost-of-hard-drive-sto...
  - daigoba66 11y ago
    At scale it must get cheaper. For example AWS Glacier is only $0.007 per GB ignoring bandwidth.
    - reitzensteinm 11y ago
      Per month.
    - ra1n85 11y ago
      Glacier wouldn't fit this model, as it's based on access frequency and not storage volume.
  - CamatHN 11y ago
    Thats 400k*365 = 146 million per year extra in storage costs. Considering FB's market cap/market worth is 265 Billion with $13B in annual revenue and photos is part of its core service, that amount of extra cost in negligible to their business. Not hard to increase those revenues by at least 3% annually which would bring another $400m a year, easily covering any extra storage costs from photos.
    - mikegioia 11y ago
      Yea I agree, but this is also just photos.
- aqwwe 11y ago
  it is probably actually much less then that since Facebook doesn't appear to store full resolutions... for example, they serve a very low quality version of the original picture...
  - rhizome 11y ago
    Yeah, I don't think I see anything larger than (guessing) 600x600px, which they probably compress to ~100k max.
    - sp332 11y ago
      You can download much higher res versions with the Options->Download or Download Full Resolution (depending on platform).
ape4 11y ago
Just a now on facebook I clicked on a friend's name (a link) and after waiting about a minute I gave up. But I am not going to complain (to them) since its free.
- brozak 11y ago
  "free"
adamnemecek 11y ago
Good thing that those graphs have the y-axis labeled.
merb 11y ago
I always wondered how much power could be saved if those platforms would be implemented nearer to the metal. I mean I don't want to flame PHP, etc.. but I think running at the scale of Facebook it will definitely matter which language you use, especially in the matter of computing power, I mean Facebook could throw enough people at anything so the choice of language shouldn't be a problem. That's why I don't get it, why they wrote Hack/etc. When I would be Facebook I would've tried to write something like JPHP where PHP could run on top of the JVM, since that introduces way more languages like Scala, Groovy, Java itself. However they gone from PHP to their self hosted language where it will even be hard to find developers when Hack won't grow.
- gt565k 11y ago
  Check out whats app's architecture "Our results have demonstrated the fantastic scalability of Erlang, and in this talk we will share some of the discoveries and modifications we have made along the path to supporting millions of connected users per server. " https://vimeo.com/44312354 https://vimeo.com/44312354 Some of the speculations about why facebook bought them is their architecture build on top of Erlang
  - simonw 11y ago
    Facebook Chat was originally written in Erlang: https://www.facebook.com/notes/facebook-engineering/chat-stability-and-scalability/51412338919 https://www.facebook.com/notes/facebook-engineering/chat-sta... - but they ended up switching it from Erlang to C++: https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-for-Facebook-Chat https://www.quora.com/When-did-Facebook-switch-away-from-usi...
- segmondy 11y ago
  You must be living under a rock if you think facebook is doing everything in PHP. Do you really believe the neural network and video encoding mentioned in the article was written in PHP?
  - frik 11y ago

66 points by Katydid 11y ago

ckluis 11y ago

40 PB a day for photos...

mikegioia 11y ago
This is the part that I have trouble wrapping my head around. If you're adding 40 PB of photos per day, and if you can somehow get say $0.01/GB [1] then that's adding $400,000 per day in storage costs. Facebook doesn't/can't/won't delete photos so how on earth is this sustainable? After 10-20 years wouldn't hosting costs just be too high to function? [1] Aaverage cost per GB in 2014 was $0.03: http://www.statisticbrain.com/average-cost-of-hard-drive-storage/ http://www.statisticbrain.com/average-cost-of-hard-drive-sto...
- daigoba66 11y ago
  At scale it must get cheaper. For example AWS Glacier is only $0.007 per GB ignoring bandwidth.
  - reitzensteinm 11y ago
    Per month.
  - ra1n85 11y ago
    Glacier wouldn't fit this model, as it's based on access frequency and not storage volume.
- CamatHN 11y ago
  Thats 400k*365 = 146 million per year extra in storage costs. Considering FB's market cap/market worth is 265 Billion with $13B in annual revenue and photos is part of its core service, that amount of extra cost in negligible to their business. Not hard to increase those revenues by at least 3% annually which would bring another $400m a year, easily covering any extra storage costs from photos.
  - mikegioia 11y ago
    Yea I agree, but this is also just photos.
aqwwe 11y ago
it is probably actually much less then that since Facebook doesn't appear to store full resolutions... for example, they serve a very low quality version of the original picture...
- rhizome 11y ago
  Yeah, I don't think I see anything larger than (guessing) 600x600px, which they probably compress to ~100k max.
  - sp332 11y ago
    You can download much higher res versions with the Options->Download or Download Full Resolution (depending on platform).

ape4 11y ago

Just a now on facebook I clicked on a friend's name (a link) and after waiting about a minute I gave up. But I am not going to complain (to them) since its free.

brozak 11y ago
"free"

adamnemecek 11y ago

Good thing that those graphs have the y-axis labeled.

merb 11y ago

I always wondered how much power could be saved if those platforms would be implemented nearer to the metal. I mean I don't want to flame PHP, etc.. but I think running at the scale of Facebook it will definitely matter which language you use, especially in the matter of computing power, I mean Facebook could throw enough people at anything so the choice of language shouldn't be a problem. That's why I don't get it, why they wrote Hack/etc. When I would be Facebook I would've tried to write something like JPHP where PHP could run on top of the JVM, since that introduces way more languages like Scala, Groovy, Java itself. However they gone from PHP to their self hosted language where it will even be hard to find developers when Hack won't grow.

gt565k 11y ago
Check out whats app's architecture "Our results have demonstrated the fantastic scalability of Erlang, and in this talk we will share some of the discoveries and modifications we have made along the path to supporting millions of connected users per server. " https://vimeo.com/44312354 https://vimeo.com/44312354 Some of the speculations about why facebook bought them is their architecture build on top of Erlang
- simonw 11y ago
  Facebook Chat was originally written in Erlang: https://www.facebook.com/notes/facebook-engineering/chat-stability-and-scalability/51412338919 https://www.facebook.com/notes/facebook-engineering/chat-sta... - but they ended up switching it from Erlang to C++: https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-for-Facebook-Chat https://www.quora.com/When-did-Facebook-switch-away-from-usi...
segmondy 11y ago
You must be living under a rock if you think facebook is doing everything in PHP. Do you really believe the neural network and video encoding mentioned in the article was written in PHP?
- frik 11y ago