3 ms·

Bah! Largest?!? I'm absolutely certain Hadoop Clusters of that size are spawned on Amazon Web Services several times a day. In fact, I'd say something more alo

Bah! Largest?!? I'm absolutely certain Hadoop Clusters of that size are spawned on Amazon Web Services several times a day. In fact, I'd say something more along the lines of, "wow, look at how old skool Facebook is!"

jpitz 15y ago
You're making the arguably extraordinary claim that _someone_ is spinning up Facebook-sized workloads on AWS, multiple times a day. Until you can back that up with a source, you're going to get downvoted.
- 0x12 15y ago
  There is this article: http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scaling_hadoop_to_4000_nodes_a/ http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scalin... Which is from 2008 and which references a cluster twice the size of the facebook one, and mentions several production clusters the same size as the facebook one. They are (obviously, this being Yahoo!) not on Amazon though.
  - al_james 15y ago
    Define bigger? More nodes yes, but less cores, RAM and disk space.
  - DanielRibeiro 15y ago
    From the article: That's a total of more than 21 PB of configured storage capacity! This is larger than the previously known Yahoo!'s cluster[1] of 14 PB. Guess they are comparing data volumes. But great for pointing out that they do not have the largest in machine count. [1] http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalability_of_the_hadoop_dist/ http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalab...
- dwhitney 15y ago
  Down voted :) I think this is my first comment. Anyway, yes, I'm the CTO for a small startup and we regularly spin up 500 node clusters on AWS for data processing. Yes that's a quarter of what Facebook is claiming, and yeah I see that their nodes have 32 gigs of RAM and we typically use either the 8 or 16 gig instance types, which are smaller, but machines of the type described in that article are available, and we're 8 person company with 3 developers, which means it's not unlikely that my claim is wrong (i.e. a larger company could easily exceed the capacity we need). Amazon allows this type of thing with ElasticMapReduce. It's normal. Do I need to supply proof? Naw, this is a bulletin board.

jpitz 15y ago

You're making the arguably extraordinary claim that _someone_ is spinning up Facebook-sized workloads on AWS, multiple times a day. Until you can back that up with a source, you're going to get downvoted.

0x12 15y ago
There is this article: http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scaling_hadoop_to_4000_nodes_a/ http://developer.yahoo.com/blogs/hadoop/posts/2008/09/scalin... Which is from 2008 and which references a cluster twice the size of the facebook one, and mentions several production clusters the same size as the facebook one. They are (obviously, this being Yahoo!) not on Amazon though.
- al_james 15y ago
  Define bigger? More nodes yes, but less cores, RAM and disk space.
- DanielRibeiro 15y ago
  From the article: That's a total of more than 21 PB of configured storage capacity! This is larger than the previously known Yahoo!'s cluster[1] of 14 PB. Guess they are comparing data volumes. But great for pointing out that they do not have the largest in machine count. [1] http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalability_of_the_hadoop_dist/ http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalab...
dwhitney 15y ago
Down voted :) I think this is my first comment. Anyway, yes, I'm the CTO for a small startup and we regularly spin up 500 node clusters on AWS for data processing. Yes that's a quarter of what Facebook is claiming, and yeah I see that their nodes have 32 gigs of RAM and we typically use either the 8 or 16 gig instance types, which are smaller, but machines of the type described in that article are available, and we're 8 person company with 3 developers, which means it's not unlikely that my claim is wrong (i.e. a larger company could easily exceed the capacity we need). Amazon allows this type of thing with ElasticMapReduce. It's normal. Do I need to supply proof? Naw, this is a bulletin board.