3 ms·
As someone not versed in the field, can anyone explain the types of workloads that are ideal for HPC machines, as opposed to, for example, a huge number of netw
by hn_throwaway_99 3y ago
As someone not versed in the field, can anyone explain the types of workloads that are ideal for HPC machines, as opposed to, for example, a huge number of networked GPUs?
E.g. it reportedly cost over $100 million to train GPT 4. My understanding is this was done in a huge number of high performance GPUs, e.g. A100s, etc. So I guess my question then is what would a dedicated "supercomputer" be used for today that couldn't be accomplished on one a more traditional network?
To emphasize, none of these questions are meant to be rhetorical or leading - I honestly don't know and am curious.
- bbatha 3y agoThat’s basically what super computers have been for the last twenty years: large computer optimized data centers. The main differences in hardware are usually infiniband (and its RDMA capabilities) paired with a really powerful parallel file system cluster. Occasionally they’ll be exotic compute accelerators or lately just variants of the gpus specific to the cluster. On the software side it’s usually slurm managing code that leverages MPI, openMP and CUDA. Having a large homogeneous cluster that has a ~10 year lifespan means that you you have a lot of tuning up and down the stack from specific MPI implementations for your infiniband hardware to optimization tricks in the science code tuned for the specific gpu and cpu models. All of this has gotten even less specialized since GPUs started to take off and those trends have accelerated with ML having the same needs. ML is also prompting clouds to provide the same kinds of hard ware with infiniband and rdma.
- trueismywork 3y agoAlgorithms like differential equation solvers for extremely fast wave speed physical systems. Molecular dynamics, electronic structure calculations. Any algorithm that requires FFT over 1 billion cells.