4 ms·

Facebook has the world's largest Hadoop cluster

51 points by DanielRibeiro 15y ago

dwhitney 15y ago
Bah! Largest?!? I'm absolutely certain Hadoop Clusters of that size are spawned on Amazon Web Services several times a day. In fact, I'd say something more along the lines of, "wow, look at how old skool Facebook is!"
- jpitz 15y ago
  You're making the arguably extraordinary claim that _someone_ is spinning up Facebook-sized workloads on AWS, multiple times a day. Until you can back that up with a source, you're going to get downvoted.
  - 0x12 15y ago
    There is this article: http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scaling_hadoop_to_4000_nodes_a/ http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scalin... Which is from 2008 and which references a cluster twice the size of the facebook one, and mentions several production clusters the same size as the facebook one. They are (obviously, this being Yahoo!) not on Amazon though.
    - al_james 15y ago
      Define bigger? More nodes yes, but less cores, RAM and disk space.
    - DanielRibeiro 15y ago
      From the article: That's a total of more than 21 PB of configured storage capacity! This is larger than the previously known Yahoo!'s cluster[1] of 14 PB. Guess they are comparing data volumes. But great for pointing out that they do not have the largest in machine count. [1] http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalability_of_the_hadoop_dist/ http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalab...
  - dwhitney 15y ago
    Down voted :) I think this is my first comment. Anyway, yes, I'm the CTO for a small startup and we regularly spin up 500 node clusters on AWS for data processing. Yes that's a quarter of what Facebook is claiming, and yeah I see that their nodes have 32 gigs of RAM and we typically use either the 8 or 16 gig instance types, which are smaller, but machines of the type described in that article are available, and we're 8 person company with 3 developers, which means it's not unlikely that my claim is wrong (i.e. a larger company could easily exceed the capacity we need). Amazon allows this type of thing with ElasticMapReduce. It's normal. Do I need to supply proof? Naw, this is a bulletin board.
miguelrios 15y ago
"Sunday, May 9, 2010" just saying... those numbers may be different now.
- DanielRibeiro 15y ago
  It was mentioned on Apache Hadoop Goes Realtime at Facebook paper[1], presented on June 12-16 2011. At least facebook research guys are still advertising it. [1] ref 16 at http://borthakur.com/ftp/RealtimeHadoopSigmod2011.pdf http://borthakur.com/ftp/RealtimeHadoopSigmod2011.pdf
klausa 15y ago
Does anyone knows what caused such big spike in size of data around August 09? Did they simply move more/all data into Hadoop or is there another explanation? Number of machine stayed relatively the same, so it seems really weird that they increased size of data 10000-fold while not even doubling their machine number.
- zach 15y ago
  Hmm. FarmVille was released shortly before that time, wasn't it? Just a thought.
  - RyanKearney 15y ago
    Except Zynga holds all that information.
- jbooth 15y ago
  If I was going to guess, I'd guess that they started storing logs of every impression at that point, that'd be a lot of data and Hive's initial use case was parsing weblogs.

51 points by DanielRibeiro 15y ago

Bah! Largest?!? I'm absolutely certain Hadoop Clusters of that size are spawned on Amazon Web Services several times a day. In fact, I'd say something more along the lines of, "wow, look at how old skool Facebook is!"

jpitz 15y ago
You're making the arguably extraordinary claim that _someone_ is spinning up Facebook-sized workloads on AWS, multiple times a day. Until you can back that up with a source, you're going to get downvoted.
- 0x12 15y ago
  There is this article: http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scaling_hadoop_to_4000_nodes_a/ http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scalin... Which is from 2008 and which references a cluster twice the size of the facebook one, and mentions several production clusters the same size as the facebook one. They are (obviously, this being Yahoo!) not on Amazon though.
  - al_james 15y ago
    Define bigger? More nodes yes, but less cores, RAM and disk space.
  - DanielRibeiro 15y ago
    From the article: That's a total of more than 21 PB of configured storage capacity! This is larger than the previously known Yahoo!'s cluster[1] of 14 PB. Guess they are comparing data volumes. But great for pointing out that they do not have the largest in machine count. [1] http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalability_of_the_hadoop_dist/ http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalab...
- dwhitney 15y ago
  Down voted :) I think this is my first comment. Anyway, yes, I'm the CTO for a small startup and we regularly spin up 500 node clusters on AWS for data processing. Yes that's a quarter of what Facebook is claiming, and yeah I see that their nodes have 32 gigs of RAM and we typically use either the 8 or 16 gig instance types, which are smaller, but machines of the type described in that article are available, and we're 8 person company with 3 developers, which means it's not unlikely that my claim is wrong (i.e. a larger company could easily exceed the capacity we need). Amazon allows this type of thing with ElasticMapReduce. It's normal. Do I need to supply proof? Naw, this is a bulletin board.

miguelrios 15y ago

"Sunday, May 9, 2010" just saying... those numbers may be different now.

DanielRibeiro 15y ago
It was mentioned on Apache Hadoop Goes Realtime at Facebook paper[1], presented on June 12-16 2011. At least facebook research guys are still advertising it. [1] ref 16 at http://borthakur.com/ftp/RealtimeHadoopSigmod2011.pdf http://borthakur.com/ftp/RealtimeHadoopSigmod2011.pdf

klausa 15y ago

Does anyone knows what caused such big spike in size of data around August 09? Did they simply move more/all data into Hadoop or is there another explanation? Number of machine stayed relatively the same, so it seems really weird that they increased size of data 10000-fold while not even doubling their machine number.

zach 15y ago
Hmm. FarmVille was released shortly before that time, wasn't it? Just a thought.
- RyanKearney 15y ago
  Except Zynga holds all that information.
jbooth 15y ago
If I was going to guess, I'd guess that they started storing logs of every impression at that point, that'd be a lot of data and Hive's initial use case was parsing weblogs.