3 ms·
I wonder whether they pulled their article because they changed their mind. Maybe over time they got to understand the difference in use-cases and tradeoffs. Th
by jey 3y ago
I wonder whether they pulled their article because they changed their mind. Maybe over time they got to understand the difference in use-cases and tradeoffs. Though it's definitely true that plain MapReduce can be shockingly inefficient, systems like Spark which add rich data structures and in-memory caching make a big improvement towards closing that gap. But it's still not a panacea, and I've definitely seen usecases where SQLite could replace and far outperform Spark, but the general programming model behind Spark is still a powerful and widely applicable abstraction.
My main complaint with Spark is that it's pretty hard for non-experts to debug crashes and failures. While the programming model beautifully abstracts away all the complexities of parallel and distributed programming with its headaches of managing clusters and dealing with synchronization and communication, the entire abstraction bursts instantly when debugging is needed. Suddenly you need to understand the entire Spark programming model from top-to-bottom, from DataFrames to RDDs and partitions, along with implementation details like shuffle files, DAGScheduler, block caching, Python-JVM interop, OOM killer, and on and on.
And there's absolutely no incentive for the commercial vendors to improve this situation in the open source project. (This isn't specific to Spark though, the entire Hadoop ecosystem seems to operate on hidden complexity in the open source version with vendors who make money by providing support and more usable distributions.)
- sitkack 3y agoSpark exists because Python is slow and the Java GC absolutely sucked for large heaps, they were unknowingly optimizing around the wrong things. A single Rust program on a large machine can handle 95% of what any org on earth would need. https://web.archive.org/web/20080509185243/http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html https://web.archive.org/web/20080509185243/http://www.databa... The article died from link rot and website "refreshes".
- VHRanger 3y agoI'll be honest, the performance issue is almost never the language used. You can write Python code that's very fast for almost any usecase (using numpy/pandas/vaex/polars/etc). Similarly for Java. The main thing is the way people use Python/Java is generally extremely inefficient - the code is full of cache misses because of how the idiomatic way to use those languages conflicts with how CPU caches work.
- sitkack 3y agoI agree, we trade horses for the wrong reasons. For big companies, the cloud is about downsizing their tech workforce and speeding up delivery. For tech focused companies, they could get all the advantages of the cloud by adopting cloud dev practices on prem.
- bornfreddy 3y agoIs Spark still used anywhere? I haven't heard about it in a loooong time. At the time it seemed like a very nice abstraction, but while I played around with it, I never had a problem that needed such a complex solution.
- twoodfin 3y agoWell, Databricks is likely to either go public or be bought by Microsoft for $40B or so in the next 12 months, so … yes?
- Kamq 3y agoPySpark instead of spark, but I had a job a couple years back using it in glue to generate financial reports. No longer on the project, but I'm pretty sure they're using it. Honestly wasn't that bad of a model. But, then again, the job didn't actually need spark, someone just sold it that way before I was on the project. Fun to work with though
- faizshah 3y agoSpark probably powers 90% of FANG batch data pipelines. Theres still lots of hive and Hadoop clusters out there that are just being maintained for reporting until they get funding to swap to some fancy DBT setup. Spark is pretty much ubiquitous and the default solution for batch processing right now (especially pyspark). There’s also a lot of users of AWS Glue with spark.
- Twirrim 3y agoAll over the places in cloud infra (I've worked for two large cloud companies). I keep hearing of more and more stuff being based around it and kafka.
- gregw134 3y agoIsn't the Hadoop story also playing out with spark--the open source version is kinda buggy, while the spark committers at Databricks have retained all the fixes for themselves.