3 ms·
If you want decent performance (more than say 20 tokens/s) for your dev team, you absolutely do need all of the model in VRAM.
by Tepix 4mo ago
If you want decent performance (more than say 20 tokens/s) for your dev team, you absolutely do need all of the model in VRAM.