4 ms·
Got the ops-6.7b chatbot running on a windows machine with a 3090 in mere minutes. The only difference was to install the cuda pytorch `pip install torch==1.13.
by lxe 4y ago
Got the ops-6.7b chatbot running on a windows machine with a 3090 in mere minutes. The only difference was to install the cuda pytorch `pip install torch==1.13.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117 https://download.pytorch.org/whl/cu117` just like in stable diffusion's case.
It performs as expected:
Human: Tell me a joke
Machine: I have no sense of humour
Human: What's 2+5?
Machine: I cannot answer that.
- A4ET8a8uTh0 4y agoHey. So did anyone try doing it with AMD cards ( I know Nvidia seems preferable now )?
- Ajedi32 4y ago6.7b is pretty small, no? Do you even need offloading for that on a 3090? I'd be curious to see what's needed to run opt-30b or opt-66b with reasonable performance. The README suggests that even opt-175b should be doable with okay performance on a single NVIDIA T4 if you have enough RAM.
- nathan_compton 4y agoIt is entirely possible to run 6.7B parameter models on a 3090, although I believe you need 16 bit weights. I think you can squeeze a 20b parameter model onto the 3090 if you go all the way down to 8.
- rjb7731 4y agoLooks like it might be no bueno on google colab for now, chatbot.py takes prompts via input() too rather then a command line argument.
- hackernewds 4y agoCould it work on Google Colab?