4 ms·
Can any of this realistically run on CPU at some point? (Not training obviously)
by kramerger 4y ago
Can any of this realistically run on CPU at some point?
(Not training obviously)
- GaggiX 4y agoI've run LLaMa models on my CPU before, ViT-L and the Q-former are two transformer models as well, so I can't see why they wouldn't run on a CPU.
- Ambix 4y agoI'm developing framework [1] in Golang with this goal in mind :) It successfully runs relatively big LLM right now, and diffusion models will be the next step [1] https://github.com/gotzmann/llama.go/ https://github.com/gotzmann/llama.go/
- anonzzzies 4y agoIt will do, probably quite soon. Many people are trying.
- muyuu 4y agoquantized Vicuna runs ok-ish in my 16GB i7 laptop (onboard graphics) and the output is usable see this comparison: https://old.reddit.com/r/LocalLLaMA/comments/12ezcly/comparing_models_gpt4xalpaca_vicuna_and_oasst/ https://old.reddit.com/r/LocalLLaMA/comments/12ezcly/compari... these models quantised to 4bit should run in CPU set ups with 16GB of RAM + 16GB of swap (Linux) and perhaps other setups run similarly
- tantony 4y agoSomeone is probably going to port it to llama.cpp soon.
- lhl 4y agoYes, you can run inference at decent speeds on CPU with llama.cpp. A token is about 0.75/words, so you can see lots of people getting 4-8 words/s on their CPUs: https://github.com/ggerganov/llama.cpp/issues/34 https://github.com/ggerganov/llama.cpp/issues/34 There a lot of optimizations that can be done. Here's one w/ potentially a 15X AVX speedup for example: https://github.com/ggerganov/llama.cpp/pull/996 https://github.com/ggerganov/llama.cpp/pull/996