3 ms·

I don't know if this really qualify as "map-reduce". It's very cool, anyway :)

by wazoox 16y ago

StavrosK 16y ago
Why not? There appears to be both a mapper and a reducer, so...
- btilly 16y ago
  There is both a mapper and a reducer internally, but a real map-reduce implementation should let you run custom code on both the map and the reduce, and a full-featured one should let you independently specify how many nodes you have mapping and reducing. I may have missed it, but I don't see those things. (I'm ignoring the fact that it is nice to be able to pass around arbitrary data structures. While true, I wouldn't expect that out of a bash implementation.)
  - adamtj 16y ago
    map/reduce is a very simple, very general idea. It is also Google's and Hadoop's specific implementations with lots of bells and whistles. It's also everything in between. This implementation has more than the minimum two necessary features. If you are arguing that it isn't a true map/reduce implementation, then I would argue that you aren't a true software developer. http://en.wikipedia.org/wiki/No_true_scotsman http://en.wikipedia.org/wiki/No_true_scotsman
    - btilly 16y ago
      The idea of MapReduce was introduced to the world in the paper http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/mapreduce.html. The abstract starts off with MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper. If you lack that feature set, then you can't solve most of the problems that MapReduce is actually used for in practice. Conversely if you have that feature set, then you can, though not necessarily with good reliability or at optimal speed. Therefore that description is what I take as the canonical description of what it means to have implemented MapReduce. As for your arguing that I am not a true software developer, guilty as charged. I've been joking for years that I am merely a profitably displaced mathematician, and my current job role is a weird cross between software development and system administration.

I don't know if this really qualify as "map-reduce". It's very cool, anyway :)

by wazoox 16y ago

I don't know if this really qualify as "map-reduce". It's very cool, anyway :)

StavrosK 16y ago

Why not? There appears to be both a mapper and a reducer, so...

btilly 16y ago
There is both a mapper and a reducer internally, but a real map-reduce implementation should let you run custom code on both the map and the reduce, and a full-featured one should let you independently specify how many nodes you have mapping and reducing. I may have missed it, but I don't see those things. (I'm ignoring the fact that it is nice to be able to pass around arbitrary data structures. While true, I wouldn't expect that out of a bash implementation.)
- adamtj 16y ago
  map/reduce is a very simple, very general idea. It is also Google's and Hadoop's specific implementations with lots of bells and whistles. It's also everything in between. This implementation has more than the minimum two necessary features. If you are arguing that it isn't a true map/reduce implementation, then I would argue that you aren't a true software developer. http://en.wikipedia.org/wiki/No_true_scotsman http://en.wikipedia.org/wiki/No_true_scotsman
  - btilly 16y ago
    The idea of MapReduce was introduced to the world in the paper http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/mapreduce.html. The abstract starts off with MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper. If you lack that feature set, then you can't solve most of the problems that MapReduce is actually used for in practice. Conversely if you have that feature set, then you can, though not necessarily with good reliability or at optimal speed. Therefore that description is what I take as the canonical description of what it means to have implemented MapReduce. As for your arguing that I am not a true software developer, guilty as charged. I've been joking for years that I am merely a profitably displaced mathematician, and my current job role is a weird cross between software development and system administration.