I did a vaguely similar thing once. We once upon a time had a EMR / Hadoop based process that simply joined a some thousands of files of a few kb-ish from S3 ea

I did a vaguely similar thing once. We once upon a time had a EMR / Hadoop based process that simply joined a some thousands of files of a few kb-ish from S3 each into one big file totaling 20GB or so. It worked, but took a while. After a bit of mucking with it, I noticed that the built-in tools for it used the wrong S3 API for the purpose. I wrote a simple dumb single-threaded Ruby script using the right API, and it ran like 4x faster IIRC. Same lesson - before you reach for Big Data tools, make sure you've fully explored simple conventional solutions.