4 ms·
I am interested to learn why models move so much data per second. Where could I learn more that is not a ChatGPT session?
by czbond 1y ago
I am interested to learn why models move so much data per second. Where could I learn more that is not a ChatGPT session?
- shorts_theory 1y agoYou might be interested in LLM Systems which talks about how LLMs work at the hardware level and what optimizations can be done to improve the efficiency of them in this course: https://llmsystem.github.io/llmsystem2025spring/ https://llmsystem.github.io/llmsystem2025spring/
- modeless 1y agoThe models (weights and activations and caches) can fill all the memory you have and more, and to a first (very rough) approximation every byte needs to be accessed for each token generated. You can see how that would add up. I highly recommend Andrej Karpathy's videos if you want to learn details.
- pfortuny 1y agoA very simplified version is: you need all the matrix to compute a matrix x vector operation, even if the vector is mostly zeroes. Edit: obviously my simplification is wrong but if you add up compression, etc… you get an idea.
- rs186 1y agoWould you mind specifying which video(s)? He has quite a lot of content to consume.
- Sohcahtoa82 1y agoModels are made of "parameters" which are really weights in a large neural network. For each token generated, each parameter needs to take its turn inside the CPU/GPU to be calculated. So if you have a 7B parameter model with 16-bit quantization, that means you'll have 14 GB/s of data coming in. If you only have 153 GB/sec of memory bandwidth, that means you'll cap out ~11 tokens/sec, regardless of how my processing power you have. You can of course quantize to 8-bit or even 4-bit, or use a smaller model, but doing so makes your model dumber. There's a trade-off between performance and capability.
- adastra22 1y agoI think you mean GB/token
- Sohcahtoa82 1y agoErr...yup. My bad. Can't edit it now.