Hope the parent comment gets more visibility. In addition to parallelizing IO and automatic management of failures, Hadoop also provides hooks to implement comp

Hope the parent comment gets more visibility. In addition to parallelizing IO and automatic management of failures, Hadoop also provides hooks to implement complex data partitioning schemes - think dynamic range partitoning, compound group partitioning, etc. Unix tools, MPI, scatter-gather are convenient only for embarrassingly parallel jobs.