4 ms·
Great to see this advancing! I’m curious if anyone knows what the best repo is for running this stuff on an Nvidia GPU with 16GB vram. I ran the official repo w
by TaylorAlexander 4y ago
Great to see this advancing! I’m curious if anyone knows what the best repo is for running this stuff on an Nvidia GPU with 16GB vram. I ran the official repo with the leaked weights and the best I could run was the 7B parameter model. I’m curious if people have found ways to fit the larger models on such a system.
- terafo 4y agoI'd assume that 33B model should fit with this(only repo that I know of that implements SparseGPT and GPTQ for LLaMa), I, personally, haven't tried though. But you can try your luck https://github.com/lachlansneff/sparsellama https://github.com/lachlansneff/sparsellama
- enlyth 4y agohttps://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- TaylorAlexander 4y agolooks great thank you!