14 ms·

Is it possible to host Facebook on AWS?

198 points by vinnyglennon 9y ago

OJFord 9y ago
> If we take the 2012 estimate to be true, Facebook’s server capacity outstrips Moore’s Law. As well as inflation, and all manner of other totally irrelevant concepts.
- jxramos 9y ago
  is it really irrelevant? It's a well understood benchmark for tech growth rate is it not?
  - positr0n 9y ago
    It's a benchmark for transistor density, nothing else. Of course a single company could grow faster than transistor density can scale. The only possible insight, if you can call it that, is that facebook owns more square centimeters of transistors now than they did in 2012.
    - jxramos 9y ago
      That's the original inspiration, but I thought people adopted the exponential rate from the original context as a bench mark for all sorts of stuff. https://en.wikipedia.org/wiki/Moore%27s_law#Other_formulations_and_similar_observations https://en.wikipedia.org/wiki/Moore%27s_law#Other_formulatio.... At best I can figure the problem with the auhtor's usage is that it's not relative to a per unit performance but rather a per enterprise performance, ie not a faster server but more of them over a unit of time. I'm not entirely convinced the restriction to a per unit basis is valid, but maybe that's a more meaningful definition than just arbitrary exponential growth.
      - jxramos 9y ago
        I'm a bit mystified by all the downvotes, am I mistaken in understanding that Moore's law can be understood in a generalized context with respect to the rate not the quantity the rate was originally measured from? Basically he observed a particular rate constant or exponential growth profile. The quoted wiki page states: "Several measures of digital technology are improving at exponential rates related to Moore's law, including the size, cost, density, and speed of components. Moore wrote only about the density of components, "a component being a transistor, resistor, diode or capacitor,"[102] at minimum cost."". So essentially when comparing the growth of one quantity vs another it's not the quantities that are being compared but the rates.
- tyingq 9y ago
  It could be relevant if there is some fixed refresh cycle of servers. That is, newer servers can serve more end users because of the faster cpu, drives, etc. The author seems to have skipped the concept though.
  - puzzle 9y ago
    Yeah, I looked through the article, but never does he mention that one rack of today's server vintage might have an extra order of magnitude more RAM, if not cores, than one from 5 years ago. Ten years ago, four-core Barcelona Opterons were so hot and in high demand that, if you wanted to be on the latest servers, you had to show a performance boost worth the premium. Sometime this year you'll be able to buy dual processor Naples systems with 64 cores (128 threads).
    - jjeaff 9y ago
      I assume that their assumption is that the facebook application has grown in complexity to keep pace with better servers. There is certainly a lot more going on as they had several years ago.
curiousgal 9y ago
Tech's own Fermi problems.
jzelinskie 9y ago
I think a lot of the smaller details this article glosses over contribute to FB's ability to run on so few machines. For example, comparing my anecdotes of FB's internal network (they probably have the most organized/mature IPv6 network on the planet) with those of my AWS public cloud experience, I'd reckon that a lot of FB applications would require architectural redesigns to operate in that kind of environment. The real question isn't "can FB be hosted on AWS?", it's "why isn't FB competing with AWS?" because what they've already got is much better for the range of applications that they deploy.
- ryandvm 9y ago
  I suppose the reason Facebook doesn't resell their infrastructure is because their endgame is to _be_ the internet. From their point of view, any site you're on that isn't Facebook is their competition.
  - bedros 9y ago
    That's exactly why only a fool will use reactjs in their web design, because FB will put you out of business and if you sue them for infringing your patents, your reactjs license is revoked
    - dgfgfdagasdfgfa 9y ago
      That's exactly why only a fool would build their software product around patent protection.
    - zepolen 9y ago
      I didn't know this, this needs to be more well known. How does this affect libraries like preact?
      - madeofpalk 9y ago
        It is rather well known but it's mostly FUD.
      - 9y ago
pier25 9y ago
> Its application code is still developed using PHP This is surprising since 1) Facebook is most likely using Hack and 2) it could save a lot of money by moving to Go. > Facebook’s entire site runs on HHVM (desktop, API and mobile) Isn't Facebook running mostly on React? I'm guessing HHVM only really powers the API, business logic, etc.
- sgslo 9y ago
  HHVM is a VM for executing Hack and PHP, used extensively on the backend at Facebook. The majority (all?) of the front end and mobile apps are powered by React and React Native, respectively.
  - pier25 9y ago
    Yes, that was why I mentioned it. The article said PHP was powering their web apps, but in fact it's only powering the APIs since their frontends are SPAs.
- pryelluw 9y ago
  Explain point #2, please.
  - pier25 9y ago
    Let's imagine Facebook spends 2 billion on servers every year. If it could reduce the number of servers by 20% because of the better Go performance it would save 200 million every year. Go outperforms HHVM by a wider margin than 20%. I've seen benchmarks as high as 50%. Sure, benchmarks aren't real world applications, but it's safe to say that generally speaking Go is significantly faster than HHVM.
    - dboreham 9y ago
      It would need a much higher return than your estimate if 20% to be worthwhile given the downsides (everything breaks, legions of engineers spend years figuring out how to do the migration). Also most systems end up being I/O or memory pressure bound, so the 20% number could easily be 0% or even negative.
oneplane 9y ago
The answer is no. This is because you can't compare commodity hosting like AWS with specialized hosting like Facebook's data centers and infrastructure software. This question of possibility has nothing to do with 'number of servers' or 'amount of money'. It's architecturally incompatible.
- accountyaccount 9y ago
  it's still possible, you just have to change the architecture
  - oneplane 9y ago
    With that concept, everything is always possible. "Can you do X with Y and Z?" -> "Yes but change Y and Z and then X is possible".
drawkbox 9y ago
Reddit and Netflix are hosted on AWS so yes you could. You wouldn't run it the same as if you had your own infrastructure though, and probably you would run it more horizontal and cached but it could be done much cheaper than this estimate. On your own environments you can use larger servers, dbs and sharding or clustering is closer. But the cloud is different and you'd be able to run it but differently. The cloud definitely influences architecture.
- jsmthrowaway 9y ago
  > Reddit and Netflix are hosted on AWS so yes you could. Those are read-heavy properties (yes, even Reddit, a miracle of aggressive caching). Don't underestimate Facebook's write load, which is the bloody difficult thing to scale.
  - sk1pper 9y ago
    Not sure if I buy that Reddit is read-heavy compared to Facebook, got any data on that?
    - jsmthrowaway 9y ago
      No comparison -- I wouldn't buy that either. Just saying they're an eyeball property, with a somewhat moderate write load all things considered. Many billions of page views versus their ~millions DAUs is my gut feeling there, since there are probably far more people reading threads off of Akamai than interacting. Little bit (somewhat outdated) here: http://highscalability.com/blog/2013/8/26/reddit-lessons-learned-from-mistakes-made-scaling-to-1-billi.html http://highscalability.com/blog/2013/8/26/reddit-lessons-lea...
    - wolco 9y ago
      Facebook is recording screen movement and a variety of other logging that reddit wouldn't be recording. User actions like upvoting vs liking per user probably falls in reddits favor but the content being shared on facebook is much larger in size.
    - 9y ago
ivan_ah 9y ago
Since we're guesstimating numbers a lot, it would be interesting to also guesstimate the human resources costs required to run the infrastructure: X electrical engineers, Y cooling engineers, Z SREs, W infrastructure devops ppls, etc. Using the cloud you don't need X,Y,Z and presumably will need less of W. Assuming a 100+k/year salary, that's like 200+k/year total cost of employment. If X+Y+Z-ΔW ~= 1000, then the cloud provides a 200M savings on that front. The real number is probably 2x 3x higher, but I'm not going to bother going back and updating the guesstimates.
- exelius 9y ago
  Right; but it doesn't change the fact that any company capable of operating technical infrastructure at scale (aka any company with compute/storage needs above a half-dozen long rows of racks at a datacenter) can do it for about 60% of what it would cost on AWS. AWS is still a huge boon to startups from a cash flow perspective - it's like leasing equipment rather than buying it, except you're also leasing a very small fraction of all the folks you need to build/operate the equipment. It's also very cheap compared to trying to build and run the infrastructure yourself at small scale. And don't kid yourself -- you still need SREs in AWS. They're working on different issues, but AWS introduces as many issues in an SRE's scope as it removes.
- p0rkbelly 9y ago
  You are also thinking about this in a perfect vacuum. Global scale presents an entirely different --non-technical-- challenge. Take for example, Dropbox stated they are moving off AWS -- except in Europe (and other non-US areas I'm sure). Having boots on the ground abroad, operating 24/7, dedicated to the mission of the company where they have not even visited ... near impossible. https://blogs.dropbox.com/business/2016/09/making-european-infrastructure-available-to-our-customers/ https://blogs.dropbox.com/business/2016/09/making-european-i... Why? Hiring people around the globe is hard who are going to remove that HDD 24/7 is pretty hard. Hiring lawyers to understand foreign law is hard. Keeping your company focused abroad and building a culture abroad is very hard. How long does it take for a company to expand to India on their own? In AWS, you just change your CloudFormation template to another region.
discodave 9y ago
I don't understand why people think that companies can get special pricing from AWS but not drive feature development. Large customers like Netflix, GE & friends get to drive feature development. Heck, if you're the CIA and give AWS $600 MM they will build a private region for you! This post has two contradictory quotes in my opinion: "It’s worth noting that this AWS price wouldn’t be what Facebook paid in this hypothetical situation. Much like Snapchat and Netflix, Facebook would be a heavy and influential user" "Facebook’s years of specialization for running Facebook are in contrast to AWS, whose storage is designed for multi-purpose (albeit-heavy) use." - Amazon already has several tiers of storage (S3, S3 - Infrequent Access, Glacier, CloudFront)... who is to say that they wouldn't be motivated to introduce more classes by a large customer like FB.
- BurningFrog 9y ago
  Sidenote: It's remarkable that Amazon happily provides server services to Netflix, who is a huge competitor!
  - kinkrtyavimoodh 9y ago
    I can see their thought process— "Netflix will most likely exist even if we don't provide them server services as there exist other cloud providers and / or Netflix is big enough to roll their own, so we might as well have a piece of our competitor's success by having their infra spend come to us."
    - sjg007 9y ago
      Netflix absolutely should roll their own system. At the very least as a meta layer on top of the major cloud providers.
      - ojbyrne 9y ago
        Why do you think they don't?
      - ghaff 9y ago
virtuallynathan 9y ago
I'd have to assume Facebook is using well over 500TB/mo of outbound traffic. I'd Assume on the order of 1-2Tbps or more on average. Disclamer: I work for AWS.
- kondro 9y ago
  Well over. 500TB a month is only 250KB per active user.
  - kondro 9y ago
    Oh. I see this might've just been a typo. Agree with the 1Tbps+ figure for external bandwidth (and maybe 300%+ on that internally and between regions).
- p0rkbelly 9y ago
  500TB/mo is nothing for FB or AWS. Hope you don't work with any anything to do with scale at AWS. You're looking more towards 1 Exabyte a month...
  - virtuallynathan 9y ago
    The author of the article makes the assumption of 500TB/mo in the article. I'd agree, probably more like 1EB+ per mo. I guessed the 1-3Tbps figure, I haven't seen FB publish anything around that.
philip1209 9y ago
Some big players with their own hardware host overflow capacity in cloud data centers, e.g. so that in DDOS they can dip into additional capacity. I don't know if Facebook does this, but I expect that many of the big players already peer with cloud data centers.
vacri 9y ago
> We need to take into account that Facebook not only has double the users but also more data created per person - photos, videos, live streams etc. Plus it now hosts Instagram. So let’s double the number. ... but apparently no efficiency gains in 5 years to reduce the number again? Faster processes, more cores, more ram, better storage, more senior-developer-hours, bespoke datacentres... ?
TheChetan 9y ago
In 2012 Facebook had 1BN users and 180K servers. 1,000,000 users / 180,000 servers = 5,556 users per server In 2017 Facebook has close to 2BN users. 2,000,000 users / 5556 users per server = 360,000 servers Notice how the 1B has only 6 zeros. The awesome part is that this mistake evens itself out later, yielding a correct number for "The of servers".
p0rkbelly 9y ago
Keep in mind, you can bet your ass that prices would be drastically lower than the public prices posted. Large enterprises of any significance, and especially flagship/strategic customers, don't pay list prices. Every cloud provider has private pricing and enterprise discount programs. That goes for all hardware vendors (e.g. Cisco, Palo Alto Networks, Oracle, etc selling to any company) as well. I'm sure Netflix pays nowhere near public pricing on AWS and I'm sure AWS pays less than anyone in the world for an Intel CPU.
- zeep 9y ago
  wholesale vs retail costs...
jaypaulynice 9y ago
"We estimate that Facebook has 830,000 servers in 2017." No way this can be true. Facebook scaled primarily using CDNs like Akamai. Akamai is the largest CDN in the world and has no more than 300k-350k servers all over the world. It's one of the largest distributed system in the world. I worked there, so I know this for sure. Facebook has recently been moving their data in house, but they do not have 830,000 servers. That's almost 3 times Akamai. At this point, Facebook would be more profitable letting people use their server infrastructure instead of using it for themselves. Running Facebook over AWS is a CDN problem that was solved by Akamai!
- tanilama 9y ago
  Servers might not be just for serving. Facebook likely have huge clusters for offline computation/analytics/machine learning stuff. In order to process the logs at their scale, I won't be surprised if they have hadoop cluster that scales over several thousand even 10k nodes, or multiple clusters.
- crispytx 9y ago
  Yeah I thought that math sounded a little fuzzy myself.
- origami777 9y ago
  <insert "when you're a hammer..." joke here> Not to be a jerk, but, running Facebook over AWS is not a CDN problem. That's crazy talk. There are far more Facebook requirements that AWS can meet that Akamai cannot. You have to look beyond their content delivery reqs. Also, they could absolutely have 800k servers. I don't think comparing their size to Akamai is fair. The requirements of each company are drastically different. Do I think the 830k is accurate? No, but only because these estimates are usually wrong (unless informed by an inside source).
  - jaypaulynice 9y ago
    Akamai handles 15-30% of all web traffic on any given day with 200k-300k servers...does Facebook have that kind of load in a day? The problem is mostly a CDN one, crunching data and writing to the servers are second to actually serving users fast. For example, I don't necessarily need to see the latest posts, comments, etc. so you can queue up the writing and sync data later, but when I hit fb.com, I have to see stuff...
deepGem 9y ago
Minor nitpick, 1,000,000 users / 180,000 servers = 5,556 users per server Should be 1,000,000,000 users / 180,000 servers = 5,556 users per server
crispytx 9y ago
I think you could easily build the NEXT facebook on AWS or any of the big cloud computing platforms. But obviously moving facebook to AWS at its current size wouldn't be practical, and no one would probably ever do something like that.
johnsmith21006 9y ago
What I have heard about Facebook infrastructure it would better with Google cloud than Amazon.
luord 9y ago
A mostly good thought experiment, IMO, if pointless since it's never going to happen.

198 points by vinnyglennon 9y ago

> If we take the 2012 estimate to be true, Facebook’s server capacity outstrips Moore’s Law. As well as inflation, and all manner of other totally irrelevant concepts.

jxramos 9y ago
is it really irrelevant? It's a well understood benchmark for tech growth rate is it not?
- positr0n 9y ago
  It's a benchmark for transistor density, nothing else. Of course a single company could grow faster than transistor density can scale. The only possible insight, if you can call it that, is that facebook owns more square centimeters of transistors now than they did in 2012.
  - jxramos 9y ago
    That's the original inspiration, but I thought people adopted the exponential rate from the original context as a bench mark for all sorts of stuff. https://en.wikipedia.org/wiki/Moore%27s_law#Other_formulations_and_similar_observations https://en.wikipedia.org/wiki/Moore%27s_law#Other_formulatio.... At best I can figure the problem with the auhtor's usage is that it's not relative to a per unit performance but rather a per enterprise performance, ie not a faster server but more of them over a unit of time. I'm not entirely convinced the restriction to a per unit basis is valid, but maybe that's a more meaningful definition than just arbitrary exponential growth.
    - jxramos 9y ago
      I'm a bit mystified by all the downvotes, am I mistaken in understanding that Moore's law can be understood in a generalized context with respect to the rate not the quantity the rate was originally measured from? Basically he observed a particular rate constant or exponential growth profile. The quoted wiki page states: "Several measures of digital technology are improving at exponential rates related to Moore's law, including the size, cost, density, and speed of components. Moore wrote only about the density of components, "a component being a transistor, resistor, diode or capacitor,"[102] at minimum cost."". So essentially when comparing the growth of one quantity vs another it's not the quantities that are being compared but the rates.
tyingq 9y ago
It could be relevant if there is some fixed refresh cycle of servers. That is, newer servers can serve more end users because of the faster cpu, drives, etc. The author seems to have skipped the concept though.
- puzzle 9y ago
  Yeah, I looked through the article, but never does he mention that one rack of today's server vintage might have an extra order of magnitude more RAM, if not cores, than one from 5 years ago. Ten years ago, four-core Barcelona Opterons were so hot and in high demand that, if you wanted to be on the latest servers, you had to show a performance boost worth the premium. Sometime this year you'll be able to buy dual processor Naples systems with 64 cores (128 threads).
  - jjeaff 9y ago
    I assume that their assumption is that the facebook application has grown in complexity to keep pace with better servers. There is certainly a lot more going on as they had several years ago.

curiousgal 9y ago

Tech's own Fermi problems.

jzelinskie 9y ago

I think a lot of the smaller details this article glosses over contribute to FB's ability to run on so few machines. For example, comparing my anecdotes of FB's internal network (they probably have the most organized/mature IPv6 network on the planet) with those of my AWS public cloud experience, I'd reckon that a lot of FB applications would require architectural redesigns to operate in that kind of environment. The real question isn't "can FB be hosted on AWS?", it's "why isn't FB competing with AWS?" because what they've already got is much better for the range of applications that they deploy.

ryandvm 9y ago
I suppose the reason Facebook doesn't resell their infrastructure is because their endgame is to _be_ the internet. From their point of view, any site you're on that isn't Facebook is their competition.
- bedros 9y ago
  That's exactly why only a fool will use reactjs in their web design, because FB will put you out of business and if you sue them for infringing your patents, your reactjs license is revoked
  - dgfgfdagasdfgfa 9y ago
    That's exactly why only a fool would build their software product around patent protection.
  - zepolen 9y ago
    I didn't know this, this needs to be more well known. How does this affect libraries like preact?
    - madeofpalk 9y ago
      It is rather well known but it's mostly FUD.
    - 9y ago

pier25 9y ago

> Its application code is still developed using PHP This is surprising since 1) Facebook is most likely using Hack and 2) it could save a lot of money by moving to Go. > Facebook’s entire site runs on HHVM (desktop, API and mobile) Isn't Facebook running mostly on React? I'm guessing HHVM only really powers the API, business logic, etc.

sgslo 9y ago
HHVM is a VM for executing Hack and PHP, used extensively on the backend at Facebook. The majority (all?) of the front end and mobile apps are powered by React and React Native, respectively.
- pier25 9y ago
  Yes, that was why I mentioned it. The article said PHP was powering their web apps, but in fact it's only powering the APIs since their frontends are SPAs.
pryelluw 9y ago
Explain point #2, please.
- pier25 9y ago
  Let's imagine Facebook spends 2 billion on servers every year. If it could reduce the number of servers by 20% because of the better Go performance it would save 200 million every year. Go outperforms HHVM by a wider margin than 20%. I've seen benchmarks as high as 50%. Sure, benchmarks aren't real world applications, but it's safe to say that generally speaking Go is significantly faster than HHVM.
  - dboreham 9y ago
    It would need a much higher return than your estimate if 20% to be worthwhile given the downsides (everything breaks, legions of engineers spend years figuring out how to do the migration). Also most systems end up being I/O or memory pressure bound, so the 20% number could easily be 0% or even negative.

oneplane 9y ago

The answer is no. This is because you can't compare commodity hosting like AWS with specialized hosting like Facebook's data centers and infrastructure software. This question of possibility has nothing to do with 'number of servers' or 'amount of money'. It's architecturally incompatible.

accountyaccount 9y ago
it's still possible, you just have to change the architecture
- oneplane 9y ago
  With that concept, everything is always possible. "Can you do X with Y and Z?" -> "Yes but change Y and Z and then X is possible".

drawkbox 9y ago

Reddit and Netflix are hosted on AWS so yes you could. You wouldn't run it the same as if you had your own infrastructure though, and probably you would run it more horizontal and cached but it could be done much cheaper than this estimate. On your own environments you can use larger servers, dbs and sharding or clustering is closer. But the cloud is different and you'd be able to run it but differently. The cloud definitely influences architecture.

jsmthrowaway 9y ago
> Reddit and Netflix are hosted on AWS so yes you could. Those are read-heavy properties (yes, even Reddit, a miracle of aggressive caching). Don't underestimate Facebook's write load, which is the bloody difficult thing to scale.
- sk1pper 9y ago
  Not sure if I buy that Reddit is read-heavy compared to Facebook, got any data on that?
  - jsmthrowaway 9y ago
    No comparison -- I wouldn't buy that either. Just saying they're an eyeball property, with a somewhat moderate write load all things considered. Many billions of page views versus their ~millions DAUs is my gut feeling there, since there are probably far more people reading threads off of Akamai than interacting. Little bit (somewhat outdated) here: http://highscalability.com/blog/2013/8/26/reddit-lessons-learned-from-mistakes-made-scaling-to-1-billi.html http://highscalability.com/blog/2013/8/26/reddit-lessons-lea...
  - wolco 9y ago
    Facebook is recording screen movement and a variety of other logging that reddit wouldn't be recording. User actions like upvoting vs liking per user probably falls in reddits favor but the content being shared on facebook is much larger in size.
  - 9y ago

ivan_ah 9y ago

Since we're guesstimating numbers a lot, it would be interesting to also guesstimate the human resources costs required to run the infrastructure: X electrical engineers, Y cooling engineers, Z SREs, W infrastructure devops ppls, etc. Using the cloud you don't need X,Y,Z and presumably will need less of W. Assuming a 100+k/year salary, that's like 200+k/year total cost of employment. If X+Y+Z-ΔW ~= 1000, then the cloud provides a 200M savings on that front. The real number is probably 2x 3x higher, but I'm not going to bother going back and updating the guesstimates.

exelius 9y ago
Right; but it doesn't change the fact that any company capable of operating technical infrastructure at scale (aka any company with compute/storage needs above a half-dozen long rows of racks at a datacenter) can do it for about 60% of what it would cost on AWS. AWS is still a huge boon to startups from a cash flow perspective - it's like leasing equipment rather than buying it, except you're also leasing a very small fraction of all the folks you need to build/operate the equipment. It's also very cheap compared to trying to build and run the infrastructure yourself at small scale. And don't kid yourself -- you still need SREs in AWS. They're working on different issues, but AWS introduces as many issues in an SRE's scope as it removes.
p0rkbelly 9y ago
You are also thinking about this in a perfect vacuum. Global scale presents an entirely different --non-technical-- challenge. Take for example, Dropbox stated they are moving off AWS -- except in Europe (and other non-US areas I'm sure). Having boots on the ground abroad, operating 24/7, dedicated to the mission of the company where they have not even visited ... near impossible. https://blogs.dropbox.com/business/2016/09/making-european-infrastructure-available-to-our-customers/ https://blogs.dropbox.com/business/2016/09/making-european-i... Why? Hiring people around the globe is hard who are going to remove that HDD 24/7 is pretty hard. Hiring lawyers to understand foreign law is hard. Keeping your company focused abroad and building a culture abroad is very hard. How long does it take for a company to expand to India on their own? In AWS, you just change your CloudFormation template to another region.

discodave 9y ago

I don't understand why people think that companies can get special pricing from AWS but not drive feature development. Large customers like Netflix, GE & friends get to drive feature development. Heck, if you're the CIA and give AWS $600 MM they will build a private region for you! This post has two contradictory quotes in my opinion: "It’s worth noting that this AWS price wouldn’t be what Facebook paid in this hypothetical situation. Much like Snapchat and Netflix, Facebook would be a heavy and influential user" "Facebook’s years of specialization for running Facebook are in contrast to AWS, whose storage is designed for multi-purpose (albeit-heavy) use." - Amazon already has several tiers of storage (S3, S3 - Infrequent Access, Glacier, CloudFront)... who is to say that they wouldn't be motivated to introduce more classes by a large customer like FB.

BurningFrog 9y ago
Sidenote: It's remarkable that Amazon happily provides server services to Netflix, who is a huge competitor!
- kinkrtyavimoodh 9y ago
  I can see their thought process— "Netflix will most likely exist even if we don't provide them server services as there exist other cloud providers and / or Netflix is big enough to roll their own, so we might as well have a piece of our competitor's success by having their infra spend come to us."
  - sjg007 9y ago
    Netflix absolutely should roll their own system. At the very least as a meta layer on top of the major cloud providers.
    - ojbyrne 9y ago
      Why do you think they don't?
    - ghaff 9y ago

virtuallynathan 9y ago

I'd have to assume Facebook is using well over 500TB/mo of outbound traffic. I'd Assume on the order of 1-2Tbps or more on average. Disclamer: I work for AWS.

kondro 9y ago
Well over. 500TB a month is only 250KB per active user.
- kondro 9y ago
  Oh. I see this might've just been a typo. Agree with the 1Tbps+ figure for external bandwidth (and maybe 300%+ on that internally and between regions).
p0rkbelly 9y ago
500TB/mo is nothing for FB or AWS. Hope you don't work with any anything to do with scale at AWS. You're looking more towards 1 Exabyte a month...
- virtuallynathan 9y ago
  The author of the article makes the assumption of 500TB/mo in the article. I'd agree, probably more like 1EB+ per mo. I guessed the 1-3Tbps figure, I haven't seen FB publish anything around that.

philip1209 9y ago

Some big players with their own hardware host overflow capacity in cloud data centers, e.g. so that in DDOS they can dip into additional capacity. I don't know if Facebook does this, but I expect that many of the big players already peer with cloud data centers.

vacri 9y ago

> We need to take into account that Facebook not only has double the users but also more data created per person - photos, videos, live streams etc. Plus it now hosts Instagram. So let’s double the number. ... but apparently no efficiency gains in 5 years to reduce the number again? Faster processes, more cores, more ram, better storage, more senior-developer-hours, bespoke datacentres... ?

TheChetan 9y ago

In 2012 Facebook had 1BN users and 180K servers. 1,000,000 users / 180,000 servers = 5,556 users per server In 2017 Facebook has close to 2BN users. 2,000,000 users / 5556 users per server = 360,000 servers Notice how the 1B has only 6 zeros. The awesome part is that this mistake evens itself out later, yielding a correct number for "The of servers".

p0rkbelly 9y ago

Keep in mind, you can bet your ass that prices would be drastically lower than the public prices posted. Large enterprises of any significance, and especially flagship/strategic customers, don't pay list prices. Every cloud provider has private pricing and enterprise discount programs. That goes for all hardware vendors (e.g. Cisco, Palo Alto Networks, Oracle, etc selling to any company) as well. I'm sure Netflix pays nowhere near public pricing on AWS and I'm sure AWS pays less than anyone in the world for an Intel CPU.

zeep 9y ago
wholesale vs retail costs...

jaypaulynice 9y ago

"We estimate that Facebook has 830,000 servers in 2017." No way this can be true. Facebook scaled primarily using CDNs like Akamai. Akamai is the largest CDN in the world and has no more than 300k-350k servers all over the world. It's one of the largest distributed system in the world. I worked there, so I know this for sure. Facebook has recently been moving their data in house, but they do not have 830,000 servers. That's almost 3 times Akamai. At this point, Facebook would be more profitable letting people use their server infrastructure instead of using it for themselves. Running Facebook over AWS is a CDN problem that was solved by Akamai!

tanilama 9y ago
Servers might not be just for serving. Facebook likely have huge clusters for offline computation/analytics/machine learning stuff. In order to process the logs at their scale, I won't be surprised if they have hadoop cluster that scales over several thousand even 10k nodes, or multiple clusters.
crispytx 9y ago
Yeah I thought that math sounded a little fuzzy myself.
origami777 9y ago
<insert "when you're a hammer..." joke here> Not to be a jerk, but, running Facebook over AWS is not a CDN problem. That's crazy talk. There are far more Facebook requirements that AWS can meet that Akamai cannot. You have to look beyond their content delivery reqs. Also, they could absolutely have 800k servers. I don't think comparing their size to Akamai is fair. The requirements of each company are drastically different. Do I think the 830k is accurate? No, but only because these estimates are usually wrong (unless informed by an inside source).
- jaypaulynice 9y ago
  Akamai handles 15-30% of all web traffic on any given day with 200k-300k servers...does Facebook have that kind of load in a day? The problem is mostly a CDN one, crunching data and writing to the servers are second to actually serving users fast. For example, I don't necessarily need to see the latest posts, comments, etc. so you can queue up the writing and sync data later, but when I hit fb.com, I have to see stuff...

deepGem 9y ago

Minor nitpick, 1,000,000 users / 180,000 servers = 5,556 users per server Should be 1,000,000,000 users / 180,000 servers = 5,556 users per server

crispytx 9y ago

I think you could easily build the NEXT facebook on AWS or any of the big cloud computing platforms. But obviously moving facebook to AWS at its current size wouldn't be practical, and no one would probably ever do something like that.

johnsmith21006 9y ago

What I have heard about Facebook infrastructure it would better with Google cloud than Amazon.

luord 9y ago

A mostly good thought experiment, IMO, if pointless since it's never going to happen.