2 ms·
While the push-back against Map/Reduce & "Big Data" in general is totally valid, it's important to put it into context. In 2007-2010, when Hadoop first starte
by nl 13y ago
While the push-back against Map/Reduce & "Big Data" in general is totally valid, it's important to put it into context.
In 2007-2010, when Hadoop first started to gain momentum it was very useful because disk sizes were smaller, 64 bit machines weren't ubiquitous and (perhaps most importantly) SSDs were insanely expensive for anything more than tiny amounts of data.
That meant if you had more than a couple of terabytes of data you either invested in a SAN, or you started looking at ways to split your data across multiple machines.
HDFS grew out of those constraints, and once you have data distributed like that, with each machine having a decently powerful CPU as well, Map/Reduce is a sensible way of dealing with it.
- jfxberns 13y agoIt still is a valid pattern. As disks get bigger, the harder it is to move the data and the more efficient it is to move the compute to the data.