3 ms·
You’re right and wrong. MapReduce is two things: a pattern for massively parallel computation, and the name of the initial implementation of the pattern at Goog
by opportune 3y ago
You’re right and wrong. MapReduce is two things: a pattern for massively parallel computation, and the name of the initial implementation of the pattern at Google.
While the initial implementation at Google quickly got replaced with better things, the MapReduce pattern is everywhere in the data space, and almost taken for granted now. Hadoop is basically the same: a shitty (I think HDFS is still pretty good, just not the compute part) initial implementation of the pattern that was quickly iterated and improved upon.
Also, a big reason people stopped having to think about eg rack-local operations is that most people operating on huge amounts of data now aren’t doing it on traditional generic servers, they’re using something like s3 on VMs in Public Cloud datacenters if they’re doing something relatively “low level” or more likely just using something like Snowflake, Spark/Databricks (pretty close to OG mapreduce…), etc.