4 ms·
Where are you using that 30B?
by flangola7 3y ago
Where are you using that 30B?
- mclightning 3y agoyou can run those models on google collab or runpod easily
- flangola7 3y agoit looks like colab requires a google account I'll lookup runpod
- moffkalast 3y agoWell not really using it per-se, just running it and others for fun on my home pc, 4bit 30B models need about 20G of regular RAM. CPU inference is kinda slow, but it's not too terrible (e.g. 1 min for a full response on my 6 core Ryzen 2nd gen). There's a web gui for llama.cpp that is really straightforward to set up for huggingface models: https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui