3 ms·

What lays the groundwork any massive-scale parallel data processing is a lot of good engineering, being aware of the common execution scenarios and relations to

by Radim 15y ago

What lays the groundwork any massive-scale parallel data processing is a lot of good engineering, being aware of the common execution scenarios and relations to the environment (incl. HW). Plus tinkering with little devilish "details". Pretty much the opposite of the (true but trivial) concept that ordering of the result is independent of the map function. TL;DR What you wrote at the end sounded a bit analogous to saying startups succeed because of their idea, while they succeed for a number of things -- mostly the execution and how much other people like the idea in the real world.

nostrademons 15y ago
Well, any real-world engineering problem is going to have a lot success factors, far more than can be enumerated in a post on an Internet message board. The general point is that you aren't going to succeed unless you understand those building blocks in enough detail that you can take them apart and put them back together again in new ways. Map & reduce are concepts from functional programming, usually explained in Scheme or Haskell. MapReduce is a C++ framework. To go from one to another, you not only need to understand the nitty-gritty engineering details, but you also need to understand the fundamental computing concepts well enough to translate them into languages and use-cases that they weren't originally intended for. (Side note: the MapReduce framework actually bears less resemblance to map & reduce than most people think. The "map" phase is a combination map & unfold, because you're not only iterating over the input, but you can output multiple times for a given input element. Think of running a word count over web pages: you have to parse the page and output multiple words per page. And the "reduce" phase is only a reduce within keys: it's really a map between keys, because the output is required to have the same keys as the input to the phase. I've often wished for a separate "re-key" phase, where the values output from the reduce phase could be reshuffled under a different key.)
- Radim 15y ago
  I don't like the preachy tone but I like the way you disassembled your own argument in the side note there :-) To avoid misunderstanding, I'm a fan of functional programming too. But the intersection of the MapReduce framework and map&reduce of Scheme and Haskell is mostly in the name and in some core math concepts that are ageless (predate programming and lambda calculus).

by Radim 15y ago

nostrademons 15y ago

Well, any real-world engineering problem is going to have a lot success factors, far more than can be enumerated in a post on an Internet message board. The general point is that you aren't going to succeed unless you understand those building blocks in enough detail that you can take them apart and put them back together again in new ways. Map & reduce are concepts from functional programming, usually explained in Scheme or Haskell. MapReduce is a C++ framework. To go from one to another, you not only need to understand the nitty-gritty engineering details, but you also need to understand the fundamental computing concepts well enough to translate them into languages and use-cases that they weren't originally intended for. (Side note: the MapReduce framework actually bears less resemblance to map & reduce than most people think. The "map" phase is a combination map & unfold, because you're not only iterating over the input, but you can output multiple times for a given input element. Think of running a word count over web pages: you have to parse the page and output multiple words per page. And the "reduce" phase is only a reduce within keys: it's really a map between keys, because the output is required to have the same keys as the input to the phase. I've often wished for a separate "re-key" phase, where the values output from the reduce phase could be reshuffled under a different key.)

Radim 15y ago
I don't like the preachy tone but I like the way you disassembled your own argument in the side note there :-) To avoid misunderstanding, I'm a fan of functional programming too. But the intersection of the MapReduce framework and map&reduce of Scheme and Haskell is mostly in the name and in some core math concepts that are ageless (predate programming and lambda calculus).