Nvidia has stuff like hardware sparsity support. Modern methods (RigL) can let you train sparse for a 2X speedup. Memory bandwidth (sparsity helps) and netwo

Nvidia has stuff like hardware sparsity support. Modern methods (RigL) can let you train sparse for a 2X speedup. Memory bandwidth (sparsity helps) and networking connectivity (Nvidia bought Mellanox and other networking companies) are important too. They are also using a lot of die space on raytracing stuff that they don't waste on the datacenter versions presumably.