8 ms·
The part about Vertex might be right but the establishing story about mapreduce is totally wrong. By the time Hadoop took off, mapreduce at Google already had o
by jeffbee 3y ago
The part about Vertex might be right but the establishing story about mapreduce is totally wrong. By the time Hadoop took off, mapreduce at Google already had one foot in the grave. If you are using Hadoop today you have adopted a technology stack that Google recognized as obsolete 15 years ago. It is difficult to see how Google lost that battle. They effectively disabled an entire industry by suggesting an obsolete stack, while simultaneously moving on to cheaper, better sequels.
- sberens 3y ago[Genuinely curious,] what have people moved on to?
- moandcompany 3y agoGoogle itself moved on to "Flume" and later created "Dataflow" the precursor for Apache Beam. While Dataflow/Beam aren't execution engines for data processing themselves, they abstract away the language of expressing data computation from the engines themselves. At Google for example, a data processing job might be expressed using Beam on top of Flume for processing. Outside of Google, most organizations with large distributed data processing problems moved on to Hadoop2 (YARN/MapReduce2) and later in present day to Apache Spark. When organizations say they are using "Databricks" they are using Apache Spark provided as a service, from a company started by the creators of Apache Spark, which happens to be Databricks. Apache Beam is also used outside of Google on top of other data processing "engines" or runners for these jobs, such as Google's Cloud Dataflow service, Apache Flink, Apache Spark, etc.
- gravypod 3y agoOpinions are my own. Some info on flume: https://research.google/pubs/pub35650/ https://research.google/pubs/pub35650/ To quote from there: "MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pipelines can be difficult."
- itissid 3y agoAren't hash joins done in RDBMS just like a general application of map-reduce? In left joins The big table is hashed on the join key value and sent to N machines and the little table is just everywhere. IIUC this is how any OLAP/Bigdata frameworks think while doing massive joins or partitioning to reduce data later, they just have to deal with additional issues like locality of partition to computation target. So map reduce is in the DNA of many data computation flows instead of a thing in off itself.
- summerlight 3y agoIn addition to Flume/Dataflow, there's a significant push toward SQL engines. In general, SQL (or similar query engines written in more declarative languages/APIs) has some performance benefits over usual Flume codes thanks to vectorized execution and other optimizations.
- chaxor 3y agoSo if I read this right, if you're not a big company (perhaps just a standard dev with maybe a tiny cluster of computers or just one beefy one), you just make a Docker container with pyspark and put your scripts in there, and everyone can reproduce your work easily on any type of machine or cluster? It seems like a reasonable approach, though it would be nice to not need the OS dependencies/docker for spark.
- mrbungie 3y agoIf you are running jobs inside pure Docker containers (i.e. just one node without need for k8s, compose, rancher or whatever), it may be the case you don't even need pyspark.
- opportune 3y agoAlso the second generation of Flume/Spark different vs MapReduce/Hadoop has to be understood in the context of what other assumptions changed at the same time. At Google, GFS was replaced with Colossus (can’t share specifics but this was also accompanied by a change in “data/machine topology” and associated networking changes away from uniform less specialized servers) which made it so “move code to data” became less important. Similarly Spark was originally meant to run on HDFS but became a lot more popular once it started being able to use things like s3 as its storage layer and public cloud VMs for compute (which was a similar transition to GFS->Colossus). In terms of usability the other two main innovations were to make it easier to program a workflow that chained MapReduce operations (without an intermediate, expensive, blocks-until-all-nodes-done disk write step, nor a jankass orchestration engine) and subsequently to declaratively specify the desired output (eg SQL) without requiring the user to specify the implementation. They’ve since added more stuff like streaming, ML, whatever, but the biggest change from 1st to 2nd gen is really in the data topology.
- moandcompany 3y agoYep. Regarding workflows with chains of Map and Reduce operations, the Hadoop ecosystem had a similar improvement with the introduction of Hadoop 2 where YARN as a container resource manager and MapReduce (MapReduce2) were introduced, separating the workflow constraints in original Hadoop/MapReduce. This led to Hadoop projects, such as Tez as an alternate execution engine, replacing MapReduce2, on YARN with the same types of flow optimizations for chained operations and reducing the number of shuffles/writes to disk (i.e. overall much better pipeline performance for typical jobs) -- This was particularly relevant for things like Hive, where Tez could be plugged in as the execution engine when running on a Hadoop 2 cluster.
- itissid 3y agoIsn't that Rama framework(Nathan Marz's new thing from Red Planet labs) the latest iteration of "lets completely abstract computation latency/complexity from the framework"? In my mind it tries to do different things depending on who you are. In the words of a colleague, "I am excited someone is trying to remove SQL" Rama seems like if you are a fullstack or backend dev then it can provide you an easy way to have a(low latency) view of your data to build upon. If you are a Data Scientist you can use the thing to pull necessary data for analysis and slice and dice it.
- disgruntledphd2 3y agoThe above will only be successful if it supports SQL. The best place to end up is something like PySpark/Snowpark as a better API for SQL is really useful when doing complicated things. You still need to have a standard SQL layer though, as otherwise you'll cripple adoption.
- choppaface 3y agoDremel / Parquet, which helps facilitate much more efficient joins and filters versus sawzall on mapreduce. For streaming there is flume and beam, or just load important data into Spanner.
- riku_iki 3y ago> Parquet Parquet is a format, and not execution engine or paradigm?.. You can totally mr over parquet.
- deleted 3y ago[deleted]
- dekhn 3y agoIt wasn't obsolete 15 years ago. There were production mapreduces making double-digit improvements in key metrics (watch time, apps purchased) much more recently than that. The system I worked on, Sibyl, isn't well known outside of Google, but used MR as an engine to do extremely large-scale machine learning. we added a number of features and used MR in ways that would have been extremely challenging to reimplement while maintaining performance targets. I'm not even sure the mapreduce code has been deleted from google3 yet. To be fair, MR was definitely dated by the time I joined- 2007- and I'm surprised it lasted as long as it did. But it was really well-tuned and reliable. Also the MR paper was never intended to stake Google's position as a data processing provider (that came far, far later). The MR, Bigtable, and GFS papers were written to attract software engineers to work on infra at google, to share some useful ideas with the world (specifically, the distributed shuffle in mapreduce, the bloom filter in bigtable, and the single-master-index-in-ram of GFS), and finally, to "show off".
- ithkuil 3y agoI remember the joke CL that deleted mapreduce from google3, even got an approval from Urs IIRC
- dieortin 3y agoAs an outsider, what is a “CL” and who is Urs?
- case 3y agohttps://en.wikipedia.org/wiki/Urs_H%C3%B6lzle https://en.wikipedia.org/wiki/Urs_H%C3%B6lzle
- ReactiveJelly 3y agoProbably Change List https://stackoverflow.com/questions/25716920/what-does-cl-mean-in-a-commit-message-what-does-it-stand-for https://stackoverflow.com/questions/25716920/what-does-cl-me...
- dmoy 3y ago
- choppaface 3y ago> They effectively disabled an entire industry by suggesting an obsolete stack Interesting opinion but not supported at all by evidence. Most non-Google datasets are small and stored on off-the-shelf heterogenous hardware, so HDFS / Mapreduce for streaming OLAP is a great fit. Cassandra (BigTable) and Parquet (Dremel) plus Cloudera’s Impala had much quicker time-to-market when large-scale BI became more relevant. “Obsolete” for Google problems sure, but Google problems largely only happen at Google. Stuff like ad targeting and ML look a lot different for products outside the Chocolate Factory.
- jeffbee 3y agoI can’t imagine HDFS being a “Good fit” for anything. Its mere existence was undoubtedly a disabling force at two large companies where I worked.
- moandcompany 3y agoI recommend reading the GFS paper and consider that there were/are use cases for horizontally scalable, fault-tolerant object storage, with the nuance of understanding that some of your storage/data nodes may also be compute nodes and there can be a benefit or preference for assigning applications to run where underlying data is stored. In the HDFS case with Hadoop's ecosystem, consider Hive, BigTable, Drill, and even Spark when running on YARN. In the peak days of Hadoop, many organizations were primarily on-prem, and S3 or S3 compatible object stores were mostly reserved for people using AWS.
- opportune 3y agoThe problem with using technology X “because Google does it” or “this is the best open source version of what Google uses, so let’s use it because Google does it” is that companies neglect that Google does not just use the technology out of the box they 1. created the software for their own needs 2. maintain a developer team to improve it and address requirements/pain points/integration 3. have an internal pool of experts in the form of the developer team and “customers”/early adopters 4. most likely have other proprietary systems like Borg or Colossus which integrate with the software very well, which OSS like Hadoop may not (another example: OSS Bazel vs Blaze+Forge+Piper+Monorepo structure). Something like HDFS was hugely painful for many teams because they had no idea how it worked or how to debug it, had no idea how to fix it or extend it, and didn’t have any good tooling to understand why something was slow. All they could do was try to configure it, integrate with it, and find answers for their problems online. That’s because HDFS was “free” but a team capable of properly maintaining, supporting/operations, and developing HDFS was extremely expensive.
- moandcompany 3y agoRegarding "obsolete" and missed opportunities: this idea of missed opportunities for Google has been touched upon several times in the last decade, including the launch of Google Cloud Platform itself. Urs Hölzle, the former head of Google TI (Technical Infrastructure), discussed in public some of the challenges and reasons for creating Google Cloud Platform as a platform, and backing projects like Kubernetes. Over time, Google has become a proprietary tech "island" in severals ways and arguably more fragmented than other large tech companies, such as Microsoft and Amazon, which happen to both have commercial cloud offerings, and Meta/Facebook. While all of these companies certainly have challenges with not-invented-here ("NIH") syndrome, and lots of internal, proprietary tools, as a software engineer at one of these three, odds are you will use and touch more commercial and open-source technologies than you would at Google. Google itself still struggles with having teams and projects use GCP for internal work versus Borg/etc; and there are plenty of valid reasons why Google teams don't use GCP. The proprietary tech "island" issue is a non-trival concern when you need to hire new software engineers from industry/outside and ramp-up time with some of these systems may be 6-months or even greater; today Alphabet/Google is at around 200k+ FTE, and you aren't going to be able to find many engineers outside that have experience with Borg/Flume/Spanner/Monarch/etc. Likewise when you are an experienced Google software engineer looking to work elsewhere, you need a translation map to figure out what tools outside are similar to the ones from inside. Google's proprietary tech island has its legitimate reasons for existing, and when people say 'xyz' commercial/open-source thing is "better," they often mean it is better for their problem at hand. At Google, a decade-plus ago many of the problems it had to solve were problems that few other organizations had, such as large-scale data processing (to be made cost-efficient on commodity hardware), and it needed to create a number of tools/platforms as solutions such as Map Reduce/GFS. Many of these tools and platforms were discussed via papers, and inspired open-source work. In the Map Reduce case, it changed how Apache Hadoop itself took shape, and the lessons from all of these later led to things like Apache Spark. The idea of losing a battle can only be applied with the benefit of hindsight, and many of the Google examples given were created at a time where there were no peers, nor at that time was Google interested in selling these things as commercial products at the time (i.e. GCP vs AWS vs Azure); it built these things according to its unique internal needs that few other organizations could relate to. I acknowledge that I am intentionally leaving out organizational politics, and culture (e.g. PERF) as non-trivial contributors for this result).
- opportune 3y agoYou’re right and wrong. MapReduce is two things: a pattern for massively parallel computation, and the name of the initial implementation of the pattern at Google. While the initial implementation at Google quickly got replaced with better things, the MapReduce pattern is everywhere in the data space, and almost taken for granted now. Hadoop is basically the same: a shitty (I think HDFS is still pretty good, just not the compute part) initial implementation of the pattern that was quickly iterated and improved upon. Also, a big reason people stopped having to think about eg rack-local operations is that most people operating on huge amounts of data now aren’t doing it on traditional generic servers, they’re using something like s3 on VMs in Public Cloud datacenters if they’re doing something relatively “low level” or more likely just using something like Snowflake, Spark/Databricks (pretty close to OG mapreduce…), etc.
- renewiltord 3y agoHadoop and then Yarn+Mapreduce gave us quite a lot of value even until 2017. I honestly don't think it was a bad technology choice. Should have moved off it faster but we had cheap commodity hardware running cheap software and early Spark was massively memory-finicky (cliff of performance). I wouldn't use that tech today but back in 2008-2014 it let us run things over a few petabytes relatively cheaply with relatively slow interconnect. We had an impl of the Pregel paper on top of the Yarn manager. The API was painful and easy to err on but it did provide quite a bit of functionality. Now, of course, that stuff is all out of date. Where I am now we have custom job engine and it's way better. I imagine others have something like this too. Things have just changed. Interconnect is now cheap and fast: 40 Gbps is commodity.