3 ms·

And still doesn't work nearly as well.

by Xorlev 11y ago

east2west 11y ago
I have always suspected so since Google still does not open source its MapReduce, but I haven't found any actual evidence directly comparing Hadoop MapReduce and Google's. There could also be technical issues preventing Google from open sourcing its implementation, although I don't know any. Now that we are moving on to more sophisticated models and implementation I am hoping more details can emerge.
- GeneralMayhem 11y ago
  I don't know anything about MR in particular, but the open-sourcing process at Google often gets bogged down by how interconnected the codebase and ecosystem are. Releasing an open source version of a component means reimplementing a lot of stuff that Google isn't ready to release yet but that that component depends on. Look at Bazel/Blaze - it took them a long time to get it out, and the open version doesn't have feature parity. Also take a look at their governance plan going forward, which hints at the difficulties.[1] That said, I suspect a large part of the reason is that Google's MapReduce is highly optimized for running in a Google datacenter, and they're generally pretty tight-lipped about the real secret sauce in their infrastructure. [1]http://bazel.io/governance.html http://bazel.io/governance.html
  - chris_va 11y ago
    https://github.com/chrisvana/repobuild/wiki/Motivation https://github.com/chrisvana/repobuild/wiki/Motivation (Pre bazel)
- duaneb 11y ago
  MapReduce and associated infrastructure at Google is arguably at the center of their massive and proprietary codebase. I suspect opening that would involve opening much of their core code.
- bertil 11y ago
  > There could also be technical issues preventing Google from open sourcing its implementation, although I don't know any. My understanding is that sharing details takes time off very valuable employees, away from fixing important internal issues – a lot more than the company being afraid those details might leak. For the handful of companies who can and needs to implement equivalent, poaching said employees is simpler than back-engineering from blogposts and it makes implementation easier.

And still doesn't work nearly as well.

by Xorlev 11y ago

And still doesn't work nearly as well.

I have always suspected so since Google still does not open source its MapReduce, but I haven't found any actual evidence directly comparing Hadoop MapReduce and Google's. There could also be technical issues preventing Google from open sourcing its implementation, although I don't know any. Now that we are moving on to more sophisticated models and implementation I am hoping more details can emerge.

GeneralMayhem 11y ago
I don't know anything about MR in particular, but the open-sourcing process at Google often gets bogged down by how interconnected the codebase and ecosystem are. Releasing an open source version of a component means reimplementing a lot of stuff that Google isn't ready to release yet but that that component depends on. Look at Bazel/Blaze - it took them a long time to get it out, and the open version doesn't have feature parity. Also take a look at their governance plan going forward, which hints at the difficulties.[1] That said, I suspect a large part of the reason is that Google's MapReduce is highly optimized for running in a Google datacenter, and they're generally pretty tight-lipped about the real secret sauce in their infrastructure. [1]http://bazel.io/governance.html http://bazel.io/governance.html
- chris_va 11y ago
  https://github.com/chrisvana/repobuild/wiki/Motivation https://github.com/chrisvana/repobuild/wiki/Motivation (Pre bazel)
duaneb 11y ago
MapReduce and associated infrastructure at Google is arguably at the center of their massive and proprietary codebase. I suspect opening that would involve opening much of their core code.
bertil 11y ago
> There could also be technical issues preventing Google from open sourcing its implementation, although I don't know any. My understanding is that sharing details takes time off very valuable employees, away from fixing important internal issues – a lot more than the company being afraid those details might leak. For the handful of companies who can and needs to implement equivalent, poaching said employees is simpler than back-engineering from blogposts and it makes implementation easier.