3 ms·

My work sent me to a Hadoop workshop in 2016 where in the introduction the instructor said Hadoop would replace the traditional RDBMS within five years. We went

by donatj 3y ago

My work sent me to a Hadoop workshop in 2016 where in the introduction the instructor said Hadoop would replace the traditional RDBMS within five years. We went on to build a system to search the full text of Shakespeare for word instances that took a solid minute to scan maybe 100k of text. An RDBMS with decent indexes could have done that work instantly; hell, awk | grep | sort | uniq -c could have done that work instantly. It’s been 8 years and I think RDBMS is stronger than ever? Colored the entire course with a “yeah right”. Frankly is Hadoop still popular? Sure, it’s still around but I don’t hear much about it anymore. Never ended up using it professionally, I do most of my heavy data processing in Go and it works great. https://twitter.com/donatj/status/740210538320273408 https://twitter.com/donatj/status/740210538320273408

michaelmior 3y ago
Hadoop has largely been replaced by Spark which eliminates a lot of the inefficiencies from Hadoop. HDFS is still reasonably popular, but in your use case, running locally would still be much better.
- wenc 3y ago
  Spark is still pretty non performant. If the workload fits in memory and a single machine, DuckDb is so much more lightweight and faster.
  - cmiles74 3y ago
    My current task at my day job is analyzing a large amount of data stored in a Spark cluster. I'd say, so far, 80% of the job has been extracting data from the cluster so that I can work with it interactively with DuckDB. This data is all read-only, I suspect a set of PostgreSQL servers would perform much better.
    - wenc 3y ago
      Yes. My job involves pulling a ton of data off Redshift into Parquet files, and then working with them using DuckDB (sooo much faster — DuckDB is parallelized, vectorized and just plain fast on Parquet datasets)
    - physicles 3y ago
      Why Postgres? DuckDB is column-based and Postgres is row-based. For analytics workloads, I’m having a hard time thinking of a scenario where Postgres wins in terms of performance. If your data is too big to fit into DuckDB, consider Clickhouse, which is also column-based and understands standard SQL.
      - cmiles74 3y ago
        I hear you. I was thinking something like "they don't have that much data anyway." For sure, there are better choices.
- hobs 3y ago
  In terms of the actual performance? Sure. In terms of the overhead, the mental model shift, the library changes, the version churn and problems with scala/spark libraries, the black box debugging, no, still really inefficient. Most of the companies I have worked with that actively have spark deployed are using it on queries with less than 1TB of data at a time and boy howdy does it make no sense.
  - michaelmior 3y ago
    I haven't really encountered most of the problems you mentioned, but I agree it can certainly be inefficient in terms of runtime. That said, I think if you're already using HDFS for data storage, being able to easily bolt on Spark does make for nice ease of use.

by donatj 3y ago

michaelmior 3y ago

Hadoop has largely been replaced by Spark which eliminates a lot of the inefficiencies from Hadoop. HDFS is still reasonably popular, but in your use case, running locally would still be much better.

wenc 3y ago
Spark is still pretty non performant. If the workload fits in memory and a single machine, DuckDb is so much more lightweight and faster.
- cmiles74 3y ago
  My current task at my day job is analyzing a large amount of data stored in a Spark cluster. I'd say, so far, 80% of the job has been extracting data from the cluster so that I can work with it interactively with DuckDB. This data is all read-only, I suspect a set of PostgreSQL servers would perform much better.
  - wenc 3y ago
    Yes. My job involves pulling a ton of data off Redshift into Parquet files, and then working with them using DuckDB (sooo much faster — DuckDB is parallelized, vectorized and just plain fast on Parquet datasets)
  - physicles 3y ago
    Why Postgres? DuckDB is column-based and Postgres is row-based. For analytics workloads, I’m having a hard time thinking of a scenario where Postgres wins in terms of performance. If your data is too big to fit into DuckDB, consider Clickhouse, which is also column-based and understands standard SQL.
    - cmiles74 3y ago
      I hear you. I was thinking something like "they don't have that much data anyway." For sure, there are better choices.
hobs 3y ago
In terms of the actual performance? Sure. In terms of the overhead, the mental model shift, the library changes, the version churn and problems with scala/spark libraries, the black box debugging, no, still really inefficient. Most of the companies I have worked with that actively have spark deployed are using it on queries with less than 1TB of data at a time and boy howdy does it make no sense.
- michaelmior 3y ago
  I haven't really encountered most of the problems you mentioned, but I agree it can certainly be inefficient in terms of runtime. That said, I think if you're already using HDFS for data storage, being able to easily bolt on Spark does make for nice ease of use.