5 ms·
> Map-reduce is naturally built into the language I hear this often about both array and functional languages but I think this kind of slogan leads to a misund
by iskander 10y ago
> Map-reduce is naturally built into the language
I hear this often about both array and functional languages but I think this kind of slogan leads to a misunderstanding of the MapReduce framework (and what's special about it).
In a functional or array language, map and reduce have the following signatures:
map : key list -> (key -> value) -> value list
reduce : value list -> (value, value -> value) -> value
These are strictly less powerful than the functions used in MapReduce:
read : block -> (input_key, input_value) list
map : (input_key, input_value) -> (output_key, output_value) list
reduce : (output_key, (output_value list)) -> (output_value, output_value -> output_value) -> (output_key, output_value)
Combining this API with a (1) a distributed file system which lets you co-locate computation with data across many computers and (2) a parallel partitioning algorithm enables petabyte-scale computation. It's qualitatively different than in-memory array processing.
- icsa 10y agokdb+ map/reduce implementation operates across data in memory or on machines in a cluster.