4 ms·
> Transformers naturally have a big part of the network that is unused on any particular token flowing through the model. You could see that by how little RAM e
by lostmsu 4y ago
> Transformers naturally have a big part of the network that is unused on any particular token flowing through the model. You could see that by how little RAM ended up being used in llama.cpp when they moved to mmaping the model.
Dense transformers (GPT-3 AFAIK is dense) don't.
- VHRanger 4y agoThey do though. A ton of the neurons end up not activating on any particular layer, leading to a huge waste as you're passing a zero down the layers. I spoke with a PM at TT and he told me an important idea is that we're spending a lot of electricity multiplying things with zeros.