4 ms·
It's not per parameter, you also need to hold activations for back prop to work.
by rfoo 3y ago
It's not per parameter, you also need to hold activations for back prop to work.
- deleted 3y ago[deleted]
- mirekrusin 3y agoYou need activations for inference as well. But all of that (trainable parameters, activations, optimizer state) is like 12 bytes per trainable parameter, not 80.