> I was struggling with a 28GB CSV file Okay, sounds reasonable, that's larger than most machines' memory... > Tried spark on my laptop Yikes, how did we get

> I was struggling with a 28GB CSV file Okay, sounds reasonable, that's larger than most machines' memory... > Tried spark on my laptop Yikes, how did we get here? Not to shame you or anything, but that's like two orders of magnitude smaller than the minimum you might consider reaching for cluster solutions...and on a laptop...I'm legitimately curious to hear the sequence of events that led you to pick up Spark for this. In my opinion this isn't something you should be leaving the command line for. I'm partial to awk; if you needed to get, say, columns 3, 7, 6, 4 and 1 for every line (in that order): awk '{print $3"\t"$7"\t"$6"\t"$4"\t"$1}' file.csv ...where you're using tab as the delimiter for the output. While I'm at it, this is my favorite awk command, because you can use it for fast deduplication without wasting cycles/memory on sorting: awk '!x[$1]++' file.csv I actually don't know of anything as concise or as fast as that last command, and it's short enough to memorize. It's definitely not intuitive, however...