13 ms·
Spark exists because Python is slow and the Java GC absolutely sucked for large heaps, they were unknowingly optimizing around the wrong things. A single Rust
by sitkack 3y ago
Spark exists because Python is slow and the Java GC absolutely sucked for large heaps, they were unknowingly optimizing around the wrong things.
A single Rust program on a large machine can handle 95% of what any org on earth would need.
https://web.archive.org/web/20080509185243/http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html https://web.archive.org/web/20080509185243/http://www.databa...
The article died from link rot and website "refreshes".
- VHRanger 3y agoI'll be honest, the performance issue is almost never the language used. You can write Python code that's very fast for almost any usecase (using numpy/pandas/vaex/polars/etc). Similarly for Java. The main thing is the way people use Python/Java is generally extremely inefficient - the code is full of cache misses because of how the idiomatic way to use those languages conflicts with how CPU caches work.
- sitkack 3y agoI agree, we trade horses for the wrong reasons. For big companies, the cloud is about downsizing their tech workforce and speeding up delivery. For tech focused companies, they could get all the advantages of the cloud by adopting cloud dev practices on prem.