4 ms·
Woops! I would have thought the number of neurons roughly equals the number of parameters, but you are right. The number of parameters is much higher.
by mg 2y ago
Woops! I would have thought the number of neurons roughly equals the number of parameters, but you are right. The number of parameters is much higher.
- tama_sala 2y agoThe embedding size is only 8k so while the parameters are 70B. So it's a huge difference