4 ms·
I believe that was the intention of the author, write a DIY post to demonstrate usefulness of Spark, not to gain insights out of the Amazon reviews.
by rubayeet 10y ago
I believe that was the intention of the author, write a DIY post to demonstrate usefulness of Spark, not to gain insights out of the Amazon reviews.
- minimaxir 10y agoA bit of both. I wanted an excuse to test out Spark to find the kinks which were ommited from the documentation (and boy did I find kinks), and also provide a practical demo.
- robzyb 10y agoThanks for writing this! I'm thinking about using Spark for a little 2M-data-point project that I'm working on, just for the learning experience. Out of curiosity, what kinks did you find?
- minimaxir 10y agoEssentially the same Spark caveats of lazy evaluation and immutability of caches: neither are a big deal on small datasets, but making a mistake on either on a large dataset can result in a lot of lost time or confusion. Then there are the massive shuffle read/writes that result in 50GB i/o which are not great for SSDs.