This is what we've found. For a certain kind of simulation, we can process data on a single core at ~.5x line rate (I think this could be a lot faster if we too

This is what we've found. For a certain kind of simulation, we can process data on a single core at ~.5x line rate (I think this could be a lot faster if we took this problem seriously) so if we move to some arrangement where hosts have to download the data in order to process it, they can only use 2 cores each, and every 2 cores we want use requires another storage box to download the data from! Paying for hardware to increase "line rate" and sending zstd-compressed files over the wire might get us a factor of ~60 here, but even then we pretty rapidly run into issues where the data has to be placed on a bunch of distinct hosts in order to be useful, since 120 cores is not a lot of machines. In the future we expect to have workloads that do less computation per piece of data, which makes the need to move our compute to our data much more acute.