2 ms·
Unless something’s changed you will need the whole model on the HPU anyway, no? So way beyond a 4090 regardless.
by taneq 2y ago
Unless something’s changed you will need the whole model on the HPU anyway, no? So way beyond a 4090 regardless.
- kristopolous 2y agoA habana just for inference? Are you sure? Also I see the 4 bit quants put it at a h100 which is fine ... I've got those at work. Maybe there will be distilled for running at home
- littlestymaar 2y agoYou can still offload most of the model to RAM and use the GPU for compute, but it's obviously much slower than what it would be if everything was on the GPU memory. see ktransformers: https://www.reddit.com/r/LocalLLaMA/comments/1jpi0n9/ktransformers_now_supports_multiconcurrency_and/ https://www.reddit.com/r/LocalLLaMA/comments/1jpi0n9/ktransf...
- kristopolous 2y agoI'm certainly not the brightest person in this thread but has there been effort to maybe bucket the computational cost of the model so that more expensive parts are on the gpu and less expensive parts are on the cpu?
- phonon 2y agoTake a look at https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...