2 ms·
Does that also mean 6GB VRAM? And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4 https://huggingface.co/elinas/a
by brucethemoose2 4y ago
Does that also mean 6GB VRAM?
And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4 https://huggingface.co/elinas/alpaca-30b-lora-int4
- terafo 4y agoNo(llama.cpp is cpu-only) and no(you need to requantize the model).
- sp332 4y agoAccording to https://mobile.twitter.com/JustineTunney/status/1641902010198036480 https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result. If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.
- brucethemoose2 4y agoYeah, or deepspeed presumably. Maybe torch.compile too. I dunno why I thought llama.cpp would support gpus. shrug
- sp332 4y agoLots of C++ programs use the GPU. It's irrelevant.
- KingMachiavelli 4y agoI think those specific Alpaca models are all in safetensor now and there isn't simple converter to ggml.
- sp332 4y agoLate edit: Deep Speed, not FlexGen. I don't know if FG could work, but that repo only supports it for Opt models.