4 ms·

According to https://mobile.twitter.com/JustineTunney/status/1641902010198036480 https://mobile.twitter.com/JustineTunney/status/164190201019... you can probabl

by sp332 4y ago

According to https://mobile.twitter.com/JustineTunney/status/1641902010198036480 https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result. If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.

brucethemoose2 4y ago
Yeah, or deepspeed presumably. Maybe torch.compile too. I dunno why I thought llama.cpp would support gpus. shrug
- sp332 4y ago
  Lots of C++ programs use the GPU. It's irrelevant.
KingMachiavelli 4y ago
I think those specific Alpaca models are all in safetensor now and there isn't simple converter to ggml.
sp332 4y ago
Late edit: Deep Speed, not FlexGen. I don't know if FG could work, but that repo only supports it for Opt models.