If there are floating point numbers in those csvs, I sped up a system like that 10x just by writing a custom (Java equivalent of) atof() that didn't do variable

If there are floating point numbers in those csvs, I sped up a system like that 10x just by writing a custom (Java equivalent of) atof() that didn't do variable decimal separators and scientific notation. That's not even counting the improvements in I/O speed from the size reduction. Any system that works from CSV's is going to be slow. I don't know what sort of computations you're doing of course, but I did all the work on a laptop, couldn't be bothered to scale it out after the improvements I made in the first pass. How much of your code spends its time in I/O (including conversions) vs actually calculating?