3 ms·
Your 120B model likely has way more active parameters, so it can probably only fit a few shared layers in the VRAM for your dGPU. You might be better off runnin
by zozbot234 7mo ago
Your 120B model likely has way more active parameters, so it can probably only fit a few shared layers in the VRAM for your dGPU. You might be better off running that model on a unified memory platform, slower VRAM but a lot more of it.
- IceWreck 7mo agoYep, I understand I was giving an example to the person I was replying to.