I'm working on YAMRF (yet another MR framework) in C#. I wanted something like Hadoop but something that was much simpler to setup and to hack on. There's no q

I'm working on YAMRF (yet another MR framework) in C#. I wanted something like Hadoop but something that was much simpler to setup and to hack on. There's no question that Hadoop is the elephant in the room, so to speak. It is very robust and performant, and there's a great ecosystem and community. It is quite complex as a result and getting it set up and tuned can take a lot of time and effort. I've got the distributed file system working and am working on the processing part now. The underlying framework is more general purpose than MR, working at the level of data or record streams which can be run through LINQ, for example. Dryad has this but it's a much more complicated beast. Even though more general purpose computation is possible with such a framework, it turns out that to achieve scale, your problem needs to be parallelizable and MR is a good way to do that. I think that's why we aren't seeing much in the way of alternatives, yet--it's a question of the "enemy of good enough."