4 ms·
> They effectively disabled an entire industry by suggesting an obsolete stack Interesting opinion but not supported at all by evidence. Most non-Google datas
by choppaface 3y ago
> They effectively disabled an entire industry by suggesting an obsolete stack
Interesting opinion but not supported at all by evidence. Most non-Google datasets are small and stored on off-the-shelf heterogenous hardware, so HDFS / Mapreduce for streaming OLAP is a great fit. Cassandra (BigTable) and Parquet (Dremel) plus Cloudera’s Impala had much quicker time-to-market when large-scale BI became more relevant.
“Obsolete” for Google problems sure, but Google problems largely only happen at Google. Stuff like ad targeting and ML look a lot different for products outside the Chocolate Factory.
- jeffbee 3y agoI can’t imagine HDFS being a “Good fit” for anything. Its mere existence was undoubtedly a disabling force at two large companies where I worked.
- moandcompany 3y agoI recommend reading the GFS paper and consider that there were/are use cases for horizontally scalable, fault-tolerant object storage, with the nuance of understanding that some of your storage/data nodes may also be compute nodes and there can be a benefit or preference for assigning applications to run where underlying data is stored. In the HDFS case with Hadoop's ecosystem, consider Hive, BigTable, Drill, and even Spark when running on YARN. In the peak days of Hadoop, many organizations were primarily on-prem, and S3 or S3 compatible object stores were mostly reserved for people using AWS.
- opportune 3y agoThe problem with using technology X “because Google does it” or “this is the best open source version of what Google uses, so let’s use it because Google does it” is that companies neglect that Google does not just use the technology out of the box they 1. created the software for their own needs 2. maintain a developer team to improve it and address requirements/pain points/integration 3. have an internal pool of experts in the form of the developer team and “customers”/early adopters 4. most likely have other proprietary systems like Borg or Colossus which integrate with the software very well, which OSS like Hadoop may not (another example: OSS Bazel vs Blaze+Forge+Piper+Monorepo structure). Something like HDFS was hugely painful for many teams because they had no idea how it worked or how to debug it, had no idea how to fix it or extend it, and didn’t have any good tooling to understand why something was slow. All they could do was try to configure it, integrate with it, and find answers for their problems online. That’s because HDFS was “free” but a team capable of properly maintaining, supporting/operations, and developing HDFS was extremely expensive.
- jeffbee 3y ago> OSS Bazel vs Blaze+Forge+Piper+Monorepo structure Wish more organizations understood this part before adopting Bazel.
- icedchai 3y agoAbout 10 years ago, I worked at a place where we had a "big data" project. Some big wig wanted to use Hadoop. Turns out, the "big data" isn't big. It's not even a gigabyte, spread over less than a dozen files. Everything worked great, but the job orchestration took longer than the actual processing. Despite this misuse of Hadoop, another guy really loves it, and decides to start a project rewriting everything to use MapReduce. A new guy started, got assigned to the "MapReduce project"... he worked on this for over a year. It never made it to production.
- lmm 3y agoHmm, I experienced the opposite - HDFS was one of the main reasons that one of the large companies where I worked was able to work at all.
- riku_iki 3y ago> can’t imagine HDFS being a “Good fit” for anything. what is the better fit if you need to store 1PB of data for cheap?..