Messages like this make me sad. Hadoop hasn't been "Yahoo's Hadoop" since at least 2008. MapReduce is deprecated in favor of newer frameworks like Cloudera Im

Messages like this make me sad. Hadoop hasn't been "Yahoo's Hadoop" since at least 2008. MapReduce is deprecated in favor of newer frameworks like Cloudera Impala and Apache Spark. These tools are years old. If you are still discussing 2008 software as if it were current, you're giving people a very false picture of what is out there. Further, avoiding having every reducer have to have high throughput network to every mapper is key; the hadoop design makes it difficult to optimize over network topology like higher within-rack bandwidth. Hadoop was designed around rack topologies where bandwidth within racks was higher than bandwidth between them. That is why it tries to place mappers near where the data is that the mappers will be reading. The general assumption is that mappers will be generating a much smaller data set than what they're reading. If that's not true, then yes, you may have locality issues. You can solve this problem by writing your own InputFormat. There are far better designs implemented: never write to local disk from the mappers, instead writing to a global dfs then performing the sort in the dfs. I worked for a company that took hadoop 0.17 and made it roughly 10x more performant (and not on toy datasets: on petabyte scale datasets) with roughly 10 engineer-years with this design. There have been a bunch of projects to optimize MapReduce, like Sailfish. But ultimately higher-level optimizations like avoiding spilling between different phases of a multi-stage job (like Spark implements) are more important. And Impala implements a ton of join optimizations, of course, as well as doing absolutely everything it can in memory. Finally, the hdfs design is pretty shitty, on two fronts: the namenode gets slow (choking everyone), and the disk-based format is missing lots of things to enable optimizations like certain types of append, or indexing. Indexing is incredibly powerful for things like join, enabling a map-side join instead of having to read all your data once, sort by join keys, then reread in your reducers. In practice, the HDFS NameNode only gets slow if you create lots of small files. There are plenty of external indices you can put on top of HDFS. For example, you can run HBase. Or if you don't want to run another daemon, you can use Apache Parquet or Apache ORCFile... file formats which are inherently indexed. Another problem with hdfs design is capacity adding; hdfs has shitty rebalancing tools. The typical problem is this: hot data is new data. So if you add capacity to hdfs, your old capacity is say 80% full and your new capacity is 0% full so where does all new data go? To the new capacity. Which is quickly 100% saturated and choking your jobs. This is completely untrue. By default, HDFS doesn't make any attempt to load balance between disks or nodes at the moment; it will keep writing until the disk / node fills up. The sysadmin must run the balancer to move things around between full and less-full nodes. I don't think any version of HDFS ever implemented the behavior you talk about as the default. hdfs' use of triple replication (well, you can actually set the replication level) instead of better algorithms like erasure encoding does no favors to performance as well. Erasure encoding isn't better for all use cases, only for cold data. Sometime you want those replicas. At some point, I expect HDFS will get RAID support, as well. To be honest, it's rare to see a cluster where lack of disk space is a problem. With 20 4 TB disk drives on each node, optimizing for reduced disk space is a bit foolish unless you profile and find that I/O is indeed the bottleneck. It's really depressing to read anything about Hadoop on Hacker News. It seems like people talk about 2008 software and Hadoop 0.20 as though it were current. Nobody talks about Windows XP as though that were the latest version of Windows (which it was in 2006), but somehow talking about the shortcomings Hadoop 0.20 and other ancient stuff is considered topical.