3 ms·
A bit of both. I wanted an excuse to test out Spark to find the kinks which were ommited from the documentation (and boy did I find kinks), and also provide a p
by minimaxir 10y ago
A bit of both. I wanted an excuse to test out Spark to find the kinks which were ommited from the documentation (and boy did I find kinks), and also provide a practical demo.
- robzyb 10y agoThanks for writing this! I'm thinking about using Spark for a little 2M-data-point project that I'm working on, just for the learning experience. Out of curiosity, what kinks did you find?
- minimaxir 10y agoEssentially the same Spark caveats of lazy evaluation and immutability of caches: neither are a big deal on small datasets, but making a mistake on either on a large dataset can result in a lot of lost time or confusion. Then there are the massive shuffle read/writes that result in 50GB i/o which are not great for SSDs.