4 ms·
They use some improvements on attention mechanisms. It is on the main page ... That is why 7B model requires 24GB VRAM, there might be increased amount of atten
by pk-protect-ai 3y ago
They use some improvements on attention mechanisms. It is on the main page ... That is why 7B model requires 24GB VRAM, there might be increased amount of attention heads as well...
- sp332 3y agoThey clarified https://news.ycombinator.com/item?id=37677311 https://news.ycombinator.com/item?id=37677311 that it should only need 16GB, but the unoptimized runtime currently takes 24.