2 ms·
Totally, I think there are a lot of cases where all that machinery is actually completely superfluous anyway because thinking about the problem you're trying to
by s_Hogg 7y ago
Totally, I think there are a lot of cases where all that machinery is actually completely superfluous anyway because thinking about the problem you're trying to solve could lead you to a way of doing it that doesn't need all that compute. There's a blog post about this I read a while back where someone did a simple graph search using spark and then did it in a slightly smarter way on a commodity laptop and it outran spark on a large-ish amount of data. Wish I could find it.
- buckminster 7y agoThis one? http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
- s_Hogg 7y agoYep, that's the one!
- eb0la 7y agoI remember 4-5 years ago I tried to slice a big CSV file with Spark and took soo loong it didn't finished after some hours. Later on the day I realized it could be done from the command line with 'cut' and 'grep'. Took less than 15 minutes.
- andoriyu 7y agoI wonder how long it would have taken with `cut` and `ripgrep`