3 ms·
What do 540 billion parameters mean in this case?
by sidcool 5y ago
What do 540 billion parameters mean in this case?
- minimaxir 5y ago540B float32 values in the model. (although since this model was trained via TPUs, likely bfloat16s instead)
- cubefox 5y agoDo those floats encode the weights of synapses? One synapse per float?
- whimsicalism 5y agoThey encode weights. Not sure what a "synapse" is though.
- nynx 5y agoMachine learning models are much more complicated than perceptrons now.
- yobbo 5y agoNetwork weights would take up around 2TB. It also means at least 4TB of TPU/GPU memory required for training, or at least 50 Nvidia A100s.