Yes, I have difficulty imagining that the bottleneck here is anything other than reading the file from a spinning-rust disk. It would probably be faster to iter

Yes, I have difficulty imagining that the bottleneck here is anything other than reading the file from a spinning-rust disk. It would probably be faster to iterate over all the files line by line (eg, sequentially) in a single thread than spinning up 40 threads and having them skip around all over the files, thrashing the disk. I had this come up recently myself. An operation was performing slowly, so I split it up to use multiple cores in parallel. Got slower. Looked at performance monitor and disk utilization was pegged at 100%, while CPU utilization was ~80% of one core. Went back to the old code, and disk utilization was still pegged at 100%, but it had a marginally higher transfer rate because it was reading one file at a time rather than several all at once. I should have examined the performance characteristics more carefully in the beginning, but I didn't, because I assumed I'm smarter than I am. Good lesson. This is almost certainly the wrong tool for the job, unless the job is to learn how to use Spark.