The low throughput is definitely a code smell. In my experience, a common mistake in handling files too large for memory is to read the data faster than you pr

The low throughput is definitely a code smell. In my experience, a common mistake in handling files too large for memory is to read the data faster than you process it, consequently running out of memory. For example, maybe they have a loop reading lines and doing a bunch of async DB writes: your process will read at disk speed and write at database speed (relatively slower), and you'll run out of memory due to async tasks/context queuing up in memory. Perhaps this is why it worked to split the file into 8GB units X 25 servers: it didn't blow their memory budget per server.