I tried to use Dask to do a formst conversion of a 5 TB binary dataset coming from astrophysical simulation. It should have been a trivial job, since it was a s

I tried to use Dask to do a formst conversion of a 5 TB binary dataset coming from astrophysical simulation. It should have been a trivial job, since it was a single huge 3D array coming from a lot of files that were slicing the array in one direction, and I needed to do a conversion in each point, do a couple of transposes and reshapes, and then output a single file. After two weeks tearing my hair out at the appalling speed, I scrapped that solution and spent one day throwing together a pipeline of classical shell tools (dd, sed, xxd etc.), orchestrated using GNU Parallell, and it did the whole job in three hours.