4 ms·

The key ideas were part of the MPI interface for a decade before Jeff came along and applied the new branding https://en.wikipedia.org/wiki/Message_Passing_Int

• dekhn 6y ago
  MapReduce and MPI are very different. MapReduce doesn't use message passing- the map phase reads inputs from sharded files in a separate disk system, applies map to the inputs, and writes out the mapped outputs to temp sort files on a seperate disk system. Then the shuffler sorts those and writes the outputs to the appropriate destination output shards in a seperate disk system, at which point the reducer reads them, applies reduce, and writes the final outputs, sharded by key to a seperate disk system. The mappers, shufflers, and reducers are all independent of each other, reading and writing from the filesystem, and managed by a coordinator. There's nothing like MPI, other than the use of the Stubby RPC system, which sort of resembles MPI but has completely different distributed communication semantics.
• bayareaguy 6y ago
  The distributed database systems of the mid 80s such as the Teradata DBC 1012[1] are better prior art for Map/Reduce. 1- https://en.wikipedia.org/wiki/DBC_1012 https://en.wikipedia.org/wiki/DBC_1012