3 ms·
This is cool and all, but one could smash this out with gnu parallel and awk with no problem. Almost everything mentioned can be done on a stream including the
by usgroup 8y ago
This is cool and all, but one could smash this out with gnu parallel and awk with no problem.
Almost everything mentioned can be done on a stream including the aggregates with non of the Spark fanfare.
You could also happily do this with PipelineDB in a single SQL query.
What is the magic allure of Spark?
- saberience 8y agoThe magic allure of spark is when you have a team of 20-30 analysts and 10 data scientists working on a 4PB dataset stored on an 80 node cluster. Let me know when you have a solution for this without using big data tools like Hadoop, spark, etc.
- usgroup 8y agoI mean for the problem at hand ... not for anything, ever :) I think usually the argument for Spark and the like proceeds by just making the dataset bigger until whatever you’re comparing becomes absurd (e.g. command line vs Spark). It was the same deal in the map reduce days. Works like shit for 99% of datasets but it’s the best thing ever when there are no other alternatives :)
- kuzehanka 8y agoIf the job can be done on a single node, there is absolutely no reason to use hadoop or spark or anything of the sort.