If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'l

If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'll automatically balance the workload across them. https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1 https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1