3 ms·
a cool hack, but a large part of what makes map reduce map reduce isn't just the "map" and "reduce" but all the necessary fault tolerance features to ensure tha
by gfodor 16y ago
a cool hack, but a large part of what makes map reduce map reduce isn't just the "map" and "reduce" but all the necessary fault tolerance features to ensure that when you run a job that's computable, it damn well is going to finish even if a meteor takes out half your data center.
- timr 16y agoEh. That's a bit like saying that a web server isn't a web server unless it supports the various bells and whistles provided by Apache. Most people don't have a gigantic cluster of computers. Their fault-tolerance needs are minimal, because the most likely point of failure is their own code. In that situation, a minimalist map/reduce implementation can be a good compromise between scale and complexity. And let's face it: for all of the internet hysteria over map/reduce, it's a very simple idea.
- saurik 16y agoThat is not difficult to handle in bash. The expectation going in should be that individual computations and even computers might fail in ways that keep them from reporting failure, at which point the fact that it is written in a language running on a VM with a poor memory manager (as it really wasn't designed for this kind of stress, and rapidly dominates RAM and fragments its heap to hell) is not a problem. (This happens to be a sore point of mine, as I had a rather unenlightened professor in grad school who refused to believe I could effectively solve this one massive distributed problem that came up with a "simple shell script for doing the distribution", giving me a bad grade in his class and never forgiving me for having made the claim. This being despite the fact that I then sent him the finished result three days later, having narrowed the search space with some light intelligence enough to have gotten the same answer his system took a week to do on better hardware, and his beloved not-a-shell-script. sigh ;P)
- gfodor 16y agoIt seems easier said than done, unless you think Hadoop's architecture and fault tolerance logic is over-engineered.
- gcb 16y agojust like you would assume the job a web proxy does is manage files in memory or disk if you only experience was with squid? riiiiiiigth.