I don't know much about this but have you thought of separating your 1millions files into 2 to 10 lists and launching a single threaded process 2 to 10 times c

I don't know much about this but have you thought of separating your 1millions files into 2 to 10 lists and launching a single threaded process 2 to 10 times concurrently, with some dumb shell script?