4 ms·

Would it be ever possible to run a GPT-{n}, n>3, similar model in a home computer wihtout GPU? I have a "good" laptop with 32GB, good processor, but no GPU (I w

by 101008 3y ago

Would it be ever possible to run a GPT-{n}, n>3, similar model in a home computer wihtout GPU? I have a "good" laptop with 32GB, good processor, but no GPU (I was never interested in gaming, crypto or ML), but I found GPT very useful and I'd prefer to run a local version instead of keep feeding OpenAI.

whynotmaybe 3y ago
Have you checked Gpt4all or Faraday.dev ?
- 101008 3y ago
  I haven't but I will - thanks!!
- __xor_eax_eax 3y ago
  Or StableDiffusion
deleted 3y ago
[deleted]
convery 3y ago
Most laptops these days have a pretty sizable GPU on the same chip. IIRC Triton makes proper use of the Intel graphics while AMDs equivalents work well with OpenCL out of the box. Apple's M1-3 architecture saw some major speedups on llama.cpp etc. as well. Worth noting is that some may need special drivers, my Xeons from 2010 has support for executing OpenCL but needed extra drivers; no comment on modern processors.
emadm 3y ago
StableLM-Zephyr outperforms original GPT3, Falcon 40b instruct, Guacano 65b etc and runs on cpu On windows you can get it via lmstudio.ai for example Here is ur running on a MacBook M2 Air, we have smaller, more performant models coming https://x.com/emostaque/status/1732912442282312099?s=46 https://x.com/emostaque/status/1732912442282312099?s=46
MacsHeadroom 3y ago
OpenHermes-2.5-Mistral-7B is better than GPT-3 (and scores even better than GPT-3.5-Turbo in human evaluations) and can even run on a raspberry pi or in the browser. On a laptop CPU it uses about 5GB of RAM (in 5bit) and runs around 20-30 tokens per second, which is very fast. I recommend downloading and running OpenHermes inside LM Studio. https://lmstudio.ai/ https://lmstudio.ai/ In LM Studio, search for OpenHermes. Pick the Q5_K_M version (this is the best quality/speed trade off). Then go to the chat tab. On the chat tab, set the context length to 4096 (or up to 16k if you want longer context) and set the number of CPU cores you have under "Hardware Settings." Select the model from the drop down and start chatting!

by 101008 3y ago

whynotmaybe 3y ago

Have you checked Gpt4all or Faraday.dev ?

101008 3y ago
I haven't but I will - thanks!!
__xor_eax_eax 3y ago
Or StableDiffusion

deleted 3y ago

[deleted]

convery 3y ago

Most laptops these days have a pretty sizable GPU on the same chip. IIRC Triton makes proper use of the Intel graphics while AMDs equivalents work well with OpenCL out of the box. Apple's M1-3 architecture saw some major speedups on llama.cpp etc. as well. Worth noting is that some may need special drivers, my Xeons from 2010 has support for executing OpenCL but needed extra drivers; no comment on modern processors.

emadm 3y ago

StableLM-Zephyr outperforms original GPT3, Falcon 40b instruct, Guacano 65b etc and runs on cpu On windows you can get it via lmstudio.ai for example Here is ur running on a MacBook M2 Air, we have smaller, more performant models coming https://x.com/emostaque/status/1732912442282312099?s=46 https://x.com/emostaque/status/1732912442282312099?s=46

MacsHeadroom 3y ago

OpenHermes-2.5-Mistral-7B is better than GPT-3 (and scores even better than GPT-3.5-Turbo in human evaluations) and can even run on a raspberry pi or in the browser. On a laptop CPU it uses about 5GB of RAM (in 5bit) and runs around 20-30 tokens per second, which is very fast. I recommend downloading and running OpenHermes inside LM Studio. https://lmstudio.ai/ https://lmstudio.ai/ In LM Studio, search for OpenHermes. Pick the Q5_K_M version (this is the best quality/speed trade off). Then go to the chat tab. On the chat tab, set the context length to 4096 (or up to 16k if you want longer context) and set the number of CPU cores you have under "Hardware Settings." Select the model from the drop down and start chatting!