4 ms·

You need activations for inference as well. But all of that (trainable parameters, activations, optimizer state) is like 12 bytes per trainable parameter, not

You need activations for inference as well. But all of that (trainable parameters, activations, optimizer state) is like 12 bytes per trainable parameter, not 80.