7 ms·
DeWitt and Stonebraker's "MapReduce: A major step backwards" (2009)
- mrkeen 3y ago> Not novel at all — it represents a specific implementation of well known techniques developed nearly 25 years ago Your old stuff is bad. > We, however, assert that they should not overlook the lessons of more than 40 years of database technology — Our old stuff is good.
- mucle6 3y agoI like your perspective
- deleted 3y ago[deleted]
- sinkasapa 3y agoI think that in academic work this is reasonable. People sometimes claim to have done something novel, when it is clearly a repetition of something older. Perhaps the citations of the works being criticized didn't make it clear that they were repeating an old idea. It allows the reader to see that there were relevant previous works that may have been subjected to criticism in the past and that those previous criticisms remain important for the more recent work. The question then is whether the recent work resolves those criticisms. The authors in this case seem not to think so. A better paraphrase is the following: The new idea is not so new as some claim. It is 20 years old and was previously shown to be flawed in comparison to other approaches. Looking over 40 years of literature, one can see the flaws in the approach and how the flaws were subsequently resolved. Proponents of the "new idea" are either unaware of these advancements or are ignoring them. I'd say that ignoring previous literature and work is a big problem is CS adjacent studies. My experience is being a theoretical linguist who interfaces with computational linguists. I have had colleagues receive criticism for citing work that is 10 years old or more, even if such a work represents the earliest example of a particular idea they are making use of. It is suggested that a more modern work should be cited. There is kind of an "anti-memory" culture that results from trying to make work seem cutting edge, even if a work is clearly an extension or reinvention of very old ideas.
- lewis1028282 3y agoSome of these issues were solved by Spark. Do agree with the overall point, people shouldn’t be reaching for Hadoop when Postgres would suffice. Indexes are fast. Use them.
- ysofunny 3y ago> people shouldn’t be reaching for Hadoop when Postgres would suffice. Indexes are fast. Use them. let me try a riskier one: "people should not be reaching for kubernetes when a system administrator would suffice. sysadmins are cheap(er?) than clouds, use them" it did not come out very well.... I got stuck trying to find what to contrast kube with; all I got in them minutes alloted to comment posting was 'system administrator'. meh.
- rijx 3y agoNowadays K3s is worth the small learning curve with a big payoff as you get a lot of automation included / by installing an operator :-)
- jt2190 3y ago> … K3s… I’m assuming this isn’t a typo and you mean Kubes? https://kubes.guru/getting-started/ https://kubes.guru/getting-started/
- maxcoder4 3y agoI don't know what OP meant, but https://k3s.io/ https://k3s.io/ also fits the context.
- Areading314 3y agoK8s is an abbreviation of kubernetes K3s is an abbreviation of k8s
- 3y ago
- kevindamm 3y agoMapReduce is a good example of choosing tools suitable to the task. I'll agree with the authors that there are certainly large DB systems that a traditional RDBMS is better suited or more optimal. But the designers and avid users of MapReduce did not use it because it seemed a more optimal DB query engine. The utility|cost being optimized was not compute cycles or disk seeks or logappends, it was the developer time needed to construct another large-scale OLAP definition and an overall tolerance to hardware and system failure.
- nopurpose 3y agoPaper evaluates MapReduce in context of RDBMS. Was it ever advocated to replace them? Closest MR product to RDBMS I know is Hive, but it was just a thin SQL layer mostly for convenience, not a killer product by any means. MR was cool and I can't deny that managing Hadoop/Spark clusters was quite fun.
- BoiledCabbage 3y agoI also agree we should stop using MapReduce because of its lack of support for Crystal Reports. /s Kidding aside it's easy to bash a paper like this with the benefit of hindsight and move on. But instead look at what we can take from it. The benefit of new tech isn't always by being better than old tech at its strengths, it can also be by being better at old tech at its weaknesses and finding a way to change the problem domain so that its strengths don't matter much.
- edejong 3y agoThe 'no schema needed' paradigm that persisted within DE is finally replaced with the hard lessons learned in the 70's. Systems die, data stays.
- threeseed 3y agoIt was never about no-schema it was about schema-on-read. And it came about because companies were wasting millions doing pointless up front schema design for data that were never queried. As well as the ongoing challenge of trying to keep schemas up to date for systems they don't control e.g. SaaS.
- dunk010 3y agoI remember this paper, and was at a company at the time that was one of the first to use MapReduce, so saw this all play out first hand. I appreciated the paper. Then and now developers rush to grab new technologies, especially those that stroke their ego-driven fantasies of “working at scale” without considering their underlying constraints or applicability. At the time this was published every company and startup under the sun was rushing to use MapReduce, most often in places where it wasn’t warranted. I’m glad someone surfaced this paper again; people still need to learn the lessons that it outlines. Microservices and k8s: I’m looking straight at you.
- jeffbee 3y agoWhich problem is more serious? 1) your small company has an over-complex system that could have been postgres; 2) your medium-sized company has a postgres that's on fire at the bottom of the ocean every day despite the forty people you hired to stabilize postgres, and your scalable replacement system is still six months away?
- alanbernstein 3y agoProbably 3) the system you overengineered too early solved the wrong problem, and your replacement is six months away, but you've paid for it twice.
- davidw 3y agoI have very rarely seen the second scenario, but the first seems more common.
- fiddlerwoaroof 3y agoThe objection I have to this paper is that I keep seeing SQL forced onto streaming workflows when the functional programming operations provided by the non-SQL APIs of systems like Flink and Spark are a lot easier to think about in this context: SQL stream joins often have very surprising performance properties and it can take a long time to figure out the tunables needed to make them perform at scale.
- mccanne 3y agoNecessity is the mother of invention. MapReduce-based systems were developed because the state-of-the-art RDBMS systems of that age could not scale to the needs of the Googles/Yahoos/Facebooks during the phenomenal growth spurt of the early Web. The novelty here was the tradeoffs they made to scale out and up using the compute and storage footprints available at the time. "We thought of that" vs "we built it and made it work".
- dekhn 3y agoMapReduce was never built to compete with RDBMS systems. It was built to compete with batch-scheduled distribution data processing, typically where there was no index. It was also built to build indices (the search index), not really use them during any of the three phases. It was also built to be reliable in the face of cheap machines (bad RAM and disk). Google built MR because it was in an existential crisis: they couldn't build a new index for the search engine, and freshness and size of the index was important for early search engines. The previous tools would crash part-way through due to the cheap hardware that Google bought. If Google had based search indexing on RDBMS, they would not exist today. Now Google did use RDBMS- they used MySQL at scale. It wasn't unheard-of for mapreduces to run against MySQL (typically doing a query to get a bunch of records, and then mapping over those records). I worked on later mapreduce (long after it was mature) which used all sorts of tricks to extend the MapReduce paradigm as far as possible but ultimately nearly everything got replace with Flume, which is effectively a computational superset of what MR can do. I think the paper must have been pulled because Stonebreaker must have gotten huge pushback for attacking MR for something it wasn't good at. See the original paper for what they proposed as good use cases: counting word occurences in a large corpus (far larger than the storage limits of postgres and others at the time), distributed grep (without an index), counting unique items (where the number of items is larger than the capacity of a database at the time), reversing a graph (convert (source, target) pairs to (target, [source, source, source]), term vectors, inverted index (the original use case for building the index) and distributed sort. None of the RDBMS of that day could handle the scale of the web. That's all.
- makmanalp 3y ago
- jey 3y agoI wonder whether they pulled their article because they changed their mind. Maybe over time they got to understand the difference in use-cases and tradeoffs. Though it's definitely true that plain MapReduce can be shockingly inefficient, systems like Spark which add rich data structures and in-memory caching make a big improvement towards closing that gap. But it's still not a panacea, and I've definitely seen usecases where SQLite could replace and far outperform Spark, but the general programming model behind Spark is still a powerful and widely applicable abstraction. My main complaint with Spark is that it's pretty hard for non-experts to debug crashes and failures. While the programming model beautifully abstracts away all the complexities of parallel and distributed programming with its headaches of managing clusters and dealing with synchronization and communication, the entire abstraction bursts instantly when debugging is needed. Suddenly you need to understand the entire Spark programming model from top-to-bottom, from DataFrames to RDDs and partitions, along with implementation details like shuffle files, DAGScheduler, block caching, Python-JVM interop, OOM killer, and on and on. And there's absolutely no incentive for the commercial vendors to improve this situation in the open source project. (This isn't specific to Spark though, the entire Hadoop ecosystem seems to operate on hidden complexity in the open source version with vendors who make money by providing support and more usable distributions.)
- sitkack 3y agoSpark exists because Python is slow and the Java GC absolutely sucked for large heaps, they were unknowingly optimizing around the wrong things. A single Rust program on a large machine can handle 95% of what any org on earth would need. https://web.archive.org/web/20080509185243/http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html https://web.archive.org/web/20080509185243/http://www.databa... The article died from link rot and website "refreshes".
- VHRanger 3y agoI'll be honest, the performance issue is almost never the language used. You can write Python code that's very fast for almost any usecase (using numpy/pandas/vaex/polars/etc). Similarly for Java. The main thing is the way people use Python/Java is generally extremely inefficient - the code is full of cache misses because of how the idiomatic way to use those languages conflicts with how CPU caches work.
- advisedwang 3y agoInteresting that all of the successors to MapReduce fix all the major criticism DeWitt and Stonebraker have. (I'm thinking BigQuery, Snowflake, Spark) So the lessons were re-learned and features re-introduced, but the massively parallel execution has persisted. Perhaps MapReduce was non-novel and flawed, but it certainly seems to have led to a flowering of rich, large scale data querying systems.
- nmca 3y agoA different criticism of mapreduce-like technologies, and one of my all time favourite papers in any field, is: Scalability, but at what COST? By Frank McSherry. If you enjoy dist-systems bashing or honestly engineersing in general it's a must-read.
- marcinzm 3y agoI was at Yahoo during the time of Hadoop/MapReduce. I'd summarize the value of MapReduce in one sentence as: a solution you can use is infinitely better than a solution you can't. An optimized DB might bo much better for a specific workload. To get an optimized DB would take 6 months of conversations with a half dozen teams and getting approval for hardware. Then you'd use it for a few days before moving to your new workload. Then get yelled t for the waste of resources. To use MapReduce you logged into a cluster and ran your code the same day. MapReduce wasn't replacing databases. It was replacing local scripts running on database dumps and dozen machine clusters cobbled together to improve throughput. It's clear from the backgrounds of the people who wrote that piece that they never had to be in the shoes of the people actually using MapReduce on a daily basis and getting value from it. edit: This is a roughly the same reason cloud took off. Cloud costs more but waiting 6 months for IT to deploy a half-bake compromise solution is significantly more expensive for the business in lost opportunity and productivity.
- mistrial9 3y agoread between the lines -- management giving orders that happen right away, is worth money to management -- full stop.
- indepnd 3y agoA case of "worse is better"
- jdlyga 3y agoIs Apache Spark really a MapReduce replacement? All the recent technical books I've read look to MapReduce as a thing of the past.
- gregw134 3y agoShort answer: yes
- wwarner 3y agothis seems to be an accurate capture of the original "databasecolumn" blog post https://dsf.berkeley.edu/cs286/papers/backwards-vertica2008.pdf https://dsf.berkeley.edu/cs286/papers/backwards-vertica2008....
- ignoreusernames 3y ago100% agree. mapReduce hype always seemed strange to me because it's basically the volcano paper from the 90s but with custom user defined operators instead of pre baked ones in a more traditional engine. To make everything worse, hadoop came along, ignoring every industry advance of the past 40 years with its "one tuple at a time" iterator based model on a garbage collected language. I realize it's very easy for me to say those things in hindsight, but it's not like vectorized execution was a weird obscure secret by the time these things came out. On a side note, it finally looks like the industry is moving towards saner tools that implement a lot of things that this article mentions mapReduce was missing
- rossjudson 3y agoHow good is your vectorized execution engine at dealing with a handful of storage nodes going down for an hour or two? Or figuring out when bit flips have randomly happened? Or at sharing resources with latency-sensitive serving jobs? "Custom user defined operators" did a lot of heavy lifting at Google over decades. The set of appropriate use cases is getting smaller, mostly due to sql-ish systems scaling up towards what could actually (and frequently, only) be done with MapReduce/Flume.
- deleted 3y ago[deleted]
- fifilura 3y agoI am sure this was true 2009. But I feel like the technologies that started with MapReduce have now matured into BigQuery and Presto/Trino. And those have incorporated some of the criticism in the article such as schemas. Not indexes though, they instead optimize on partitions. But I think for the better since thay are more suited for the job and easier to work with.
- ein0p 3y agoDistributed SQL backends such as BigQuery and Spanner basically use MapReduce underneath anyway. As do most other distributed backends, SQL or not, capable of aggregation between shards.
- rossjudson 3y agoNope and nope (unless you're generalizing MapReduce to "stuff that splits up work and combines the results) ;) Scaled, distributed joining has a small army of techniques working hard these days.
- ein0p 3y agoYes, stuff that splits up, shuffles, and combines the result is exactly what I mean. That’s MapReduce. It’s just that e.g. in Dremel/BigQuery the shuffle step happens via a special in-memory distributed system, but the fundamental idea is the same: do what you can on the node, partition/shuffle, reduce on the receiving nodes, sometimes more than once.
- random3 3y agoHaving been an early adopter (2007) and riding the whole wave I think the entire movement ignited with MapReduce, BigTable, etc. was probably one of the best things that happenned in the industry. For me personally, it allowed to break things down to first principles in a way that "industry coding" wouldn't have been able to. It was the practical side of theory that was confined in school. There were, however, two types of big data adopters. I was in the bottom up camp, where passion for learing and finding the best solution to the problem was the driver. The top-down camp that eventually filled the Hadoop conferences by the time they got large (>1000 people) I suspect didn't get much out of it, neither for their organizations, nor personally. So back to Stonebreaker, back then, same as now, looks like a frustration more than anything. I do understand where it comes from, but still a frustration more than anything. Relational algebra is nice, but classical databases and SQL never nailed neither theory nor practice. NoSQL for me was more NoOracle, No<MSSQL, etc. and an ability to learn by doing from the ground up.
- dtjohnnymonkey 3y agoI am responsible for a MapReduce-based system built about 13 years ago and it is the bane of my team’s existence right now. It was the hotness in its day, but did not age well at all. We are working on a replacement using ClickHouse.
- zui 3y agoArchive.org have a copy of the original link: https://web.archive.org/web/20080509185243/http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html https://web.archive.org/web/20080509185243/http://www.databa...
- markhahn 3y agoThe note seems oddly reductionist or strawman-indulging. Nothing about MR says you can't use indices. In a sense, MR is really about sharding! It would be interesting to know how often MR (and its line) has been used on datasets, and execution hardware, which didn't suit standard SQL approaches. But it's also fundamentally different targets (again, both data and hardware). You probably shouldn't be doing MR-like approaches on less than dozens of (commodity, local-disk) nodes, and I think that's a pretty extreme tail for standard monolithic SQL...
- deleted 3y ago[deleted]
- alexnewman 3y agoI was an apache committer/contributor to a few of the hadoop ecosystem and found this paper to be very inspiring. I totally agree with much of it and it really changed the way I look at the world. I would go farther nowadays especially that databases are so smart.
- srj 3y agoThis reads as though they had taken issue with some of the movement away from traditional relational databases in that era. I know it's been retracted, so who knows what the authors opinions are today, but the eureka moment to me was in horizontal scalability - not the map and reduce functions. Until then if you wanted something to go faster you simply bought a bigger computer. In my work many of the MR use cases didn't necessarily compete with activities you'd expect of a database. For example - take the entire Geo database and generate the map tiles. I also must have written hundreds of MRs that essentially transform the entire dataset. You certainly can do that in a relational database but it seems like the wrong approach. Similarly if I was going to perform a bunch of queries I wouldn't write a new MR each time.
- Hendrikto 3y ago> Until then if you wanted something to go faster you simply bought a bigger computer. People were not stupid back then, you know. MR did not introduce horizontal scaling.
- srj 3y agoThis was not my first programming job and I didn't mean to imply stupidity. Maybe you could give some examples? I had seen it in some contexts: distcc, networking, seti@home. At least to me it didn't seem as common a tactic as trying to scale up the hardware on the machine or designing a different algorithm. That's generally what I saw at Microsoft in the years before. Maybe it was more the power of combining it with a "cloud" and containers - being able to change just the value of --mapreduce_machines= to something like 10000 and seeing it happen. That's the first time I had been exposed to that.