3 ms·
> Unfortunately that is nowhere close to being able to run a 750B param model. For something like that, we're getting closer to 1TB VRAM You don't have to run
by zozbot234 4mo ago
> Unfortunately that is nowhere close to being able to run a 750B param model. For something like that, we're getting closer to 1TB VRAM
You don't have to run a model from VRAM, or even from a sizeable amount of RAM. These choices only ever make sense when serving the model at scale, to hundreds of simultaneous users or more.
- bitmasher9 4mo agoFor workstation inference a unified memory architecture would be a good cost/performance balance, while keeping COGs reasonable. 512GB unified memory macs are available, with the ram upgrade costing a few grand.