3 ms·

>> Google stopped using MapReduce How are they then querying over big data these days? We don't know, do we? Or did they open-source their search engine? By

>> Google stopped using MapReduce How are they then querying over big data these days? We don't know, do we? Or did they open-source their search engine? By using Hadoop people are trying to not reinvent the big data wheel, partly because it's a motherfucker of a problem to have to solve and party because they want to solve the business problem, not the technical one. I don't see how that is in any way worthy of being frowned upon.

StreamBright 7y ago
Maybe you and the downvoter crew do not know it. https://www.datacenterknowledge.com/archives/2014/06/25/google-dumps-mapreduce-favor-new-hyper-scale-analytics-system https://www.datacenterknowledge.com/archives/2014/06/25/goog... https://www.quora.com/Why-did-Google-stop-using-MapReduce-and-start-encouraging-Cloud-Dataflow https://www.quora.com/Why-did-Google-stop-using-MapReduce-an...
- yourbandsucks 7y ago
  The same concepts are still there, it's just a nicer API now. Think Hadoop -> spark. Map -> map, filter, flatmap, etc Reduce -> reduce, joins, folds, group by, etc Those other concepts were always expressible as map and reduce, of course, just with a bunch of annoying repetitive work
  - mrep 7y ago
    Dataflow looks like it is just their public version of flumejava (both mention pcollections) [0] which indeed is basically a just a bunch of mapreduces pipelined into a fancy directed acyclic graph. Here is the abstract: "MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pipelines can be difficult. We present FlumeJava, a Java library that makes it easy to develop, test, and run efficient dataparallel pipelines. At the core of the FlumeJava library are a couple of classes that represent immutable parallel collections, each supporting a modest number of operations for processing them in parallel. Parallel collections and their operations present a simple, high-level, uniform abstraction over different data representations and execution strategies. To enable parallel operations to run efficiently, FlumeJava defers their evaluation, instead internally constructing an execution plan dataflow graph. When the final results of the parallel operations are eventually needed, FlumeJava first optimizes the execution plan, and then executes the optimized operations on appropriate underlying primitives (e.g., MapReduces). The combination of high-level abstractions for parallel data and computation, deferred evaluation and optimization, and efficient parallel primitives yields an easy-to-use system that approaches the efficiency of hand-optimized pipelines. FlumeJava is in active use by hundreds of pipeline developers within Google" [0]. [0]: https://ai.google/research/pubs/pub35650 https://ai.google/research/pubs/pub35650
    - yourbandsucks 7y ago
      Yeah, reduce should have really been named groupBy or something. The shiffle-sort to group by key is the important part.

StreamBright 7y ago

Maybe you and the downvoter crew do not know it. https://www.datacenterknowledge.com/archives/2014/06/25/google-dumps-mapreduce-favor-new-hyper-scale-analytics-system https://www.datacenterknowledge.com/archives/2014/06/25/goog... https://www.quora.com/Why-did-Google-stop-using-MapReduce-and-start-encouraging-Cloud-Dataflow https://www.quora.com/Why-did-Google-stop-using-MapReduce-an...

yourbandsucks 7y ago
The same concepts are still there, it's just a nicer API now. Think Hadoop -> spark. Map -> map, filter, flatmap, etc Reduce -> reduce, joins, folds, group by, etc Those other concepts were always expressible as map and reduce, of course, just with a bunch of annoying repetitive work
- mrep 7y ago
  Dataflow looks like it is just their public version of flumejava (both mention pcollections) [0] which indeed is basically a just a bunch of mapreduces pipelined into a fancy directed acyclic graph. Here is the abstract: "MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pipelines can be difficult. We present FlumeJava, a Java library that makes it easy to develop, test, and run efficient dataparallel pipelines. At the core of the FlumeJava library are a couple of classes that represent immutable parallel collections, each supporting a modest number of operations for processing them in parallel. Parallel collections and their operations present a simple, high-level, uniform abstraction over different data representations and execution strategies. To enable parallel operations to run efficiently, FlumeJava defers their evaluation, instead internally constructing an execution plan dataflow graph. When the final results of the parallel operations are eventually needed, FlumeJava first optimizes the execution plan, and then executes the optimized operations on appropriate underlying primitives (e.g., MapReduces). The combination of high-level abstractions for parallel data and computation, deferred evaluation and optimization, and efficient parallel primitives yields an easy-to-use system that approaches the efficiency of hand-optimized pipelines. FlumeJava is in active use by hundreds of pipeline developers within Google" [0]. [0]: https://ai.google/research/pubs/pub35650 https://ai.google/research/pubs/pub35650
  - yourbandsucks 7y ago
    Yeah, reduce should have really been named groupBy or something. The shiffle-sort to group by key is the important part.