3 ms·

Interesting. Let's do the math ... Let's say the model has 50B parameters and 50 layers. That would mean about one billion values have to travel through the wi

by mg 2y ago

Interesting. Let's do the math ... Let's say the model has 50B parameters and 50 layers. That would mean about one billion values have to travel through the wifi for every generated token? I wonder how much data that is in bytes and how long it takes to transfer them.

blackbear_ 2y ago
It's not the parameters that are sent, it's the layer outputs. That makes for a few thousands floats per token
- mg 2y ago
  Woops! I would have thought the number of neurons roughly equals the number of parameters, but you are right. The number of parameters is much higher.
  - tama_sala 2y ago
    The embedding size is only 8k so while the parameters are 70B. So it's a huge difference