3 ms·
I get 10 tokens/second on a 4-bit 13B model with 8GB VRAM offloading as much as possible to the GPU. At this speed, I cannot read the LLM output as fast as it g
by tarruda 3y ago
I get 10 tokens/second on a 4-bit 13B model with 8GB VRAM offloading as much as possible to the GPU. At this speed, I cannot read the LLM output as fast as it generates, so I consider it to be sufficient.
- johnbellone 3y agoWhich video card?
- tarruda 3y agoRTX 3070 Max-Q (laptop)