4 ms·
I had to work on some "data pipelines" built with Python & Spark. Terrible experience. All the different data structures for tabular data without a common inter
by specialist 8y ago
I had to work on some "data pipelines" built with Python & Spark. Terrible experience. All the different data structures for tabular data without a common interface. Oodles of ETL code for transmuting data structures to apply different algorithms. Foraging the docs to figure out even the inconsistent smallest tidbits.
Madness.
I am still aghast. How does anyone get any work done using those tools?
https://www.datacamp.com/community/tutorials/apache-spark-python#difference https://www.datacamp.com/community/tutorials/apache-spark-py...
Why didn't they implement RowSet interface?
Or create new tabular data holding interface?
Any way. Whenever someone enthuses about Python, Spark, big data, whatever, I now assume they're insane.
--
The whole notebook (Jupyter) idea is awesome. Are the fans conflating different tools, equating Spark with notebooks?
- mruts 8y agoSpark is written in Scala, not Python. I've never used the Python interface, but I've extensively used Spark from Scala and while it's a little annoying to work with sometimes, for "real" Big Data (data set > 1TB maybe), its probably the best tool around.
- stochastic_monk 8y agoHave you looked at Thrill? [0] It claims to basically be a better, C++ Spark. [0] https://github.com/thrill https://github.com/thrill