3 ms·
Spark was a really big step up from hadoop. But these days just use trino or whatever. There are lots of new ways to work on data that are all bigger steps up
by willvarfar 1y ago
Spark was a really big step up from hadoop.
But these days just use trino or whatever. There are lots of new ways to work on data that are all bigger steps up - ergonomically, performance and price - over spark as spark was over hadoop.
- disgruntledphd2 1y agoThe nice thing about spark is the scala/python/R APIs. That helps to avoid lots of the irritating things about SQL (the same transformation applied to multiple columns is a big one).
- lelandbatey 1y agoI really can't speak highly enough of Trino (though I used it as AWS Athena, and this was back when Trino was called Presto). It's impressive how well it took "ever growing pile of CSV/JSON/Excel/Parquet/whatever" and let you query it via SQL as-is without transforming it and putting it into some other system. What an impressive feat of engineering.