2 ms·
Why is there still a hard VRAM requirement that's dependent on the model size? Isn't that exactly what this project is supposed to solve?
by ranger_danger 3mo ago
Why is there still a hard VRAM requirement that's dependent on the model size? Isn't that exactly what this project is supposed to solve?
- MakazhanAlpamys 3mo ago[flagged]
- MakazhanAlpamys 3mo agoBecause streaming only removes the decoder stack. The embeddings and lm_head stay resident, that is 2.10 GB of the 3.32 GB peak on 8B. And the logits tensor scales with batch x seq x vocab, not with depth. So it goes from "whole model must fit" to "embeddings plus one layer plus logits must fit". That is why 8B works and why I did not try 14B. The table on the site is the normal resident path, streaming is opt-in and BETA. Should be clearer, my fault.