Once models are trained, it is not too difficult to compress them. Have you done any of that? You can start with things like switching the weights and activati

Once models are trained, it is not too difficult to compress them. Have you done any of that? You can start with things like switching the weights and activations to fp16. A100 also supports BFloat16, which is even better and should work right out of the box.