3 ms·DeepSeek R1 671B over 2 tok/s without GPU on local gaming rig17 points by tosh 2y agobuyucu 2y agoI applaud how hardcore this is. Swapping the model from disk and just keeping the KV cache on the CPU ram.Oarch 2y agoCan someone ELI5 please?buyucu 2y agodeepseek is huge with 671b parameters. they keep it in hard disk, and load it piece by piece to the ram. the innovation is that they kick out everything other than the kv cache from the ram.Oarch 2y agoThank you :)htrp 2y agoThe lmsys had a package (flexgen) that did a lot of this similar work (swap GPU to ram to disk) not sure if it's still being maintaineddeleted 2y ago[deleted]