4 ms·
One should be able to run Mistral 7B locally on something as low as 1070 8GB would they not? That is assuming 8bit weights. I have not kept up with local LLM
by sireat 3y ago
One should be able to run Mistral 7B locally on something as low as 1070 8GB would they not?
That is assuming 8bit weights.
I have not kept up with local LLM news. I assume the steps needed would be somewhat similar to https://followfoxai.substack.com/p/how-to-run-llama-in-an-old-gpu https://followfoxai.substack.com/p/how-to-run-llama-in-an-ol... which is from April 2023.
- brucethemoose2 3y agoIts already kind of outdated, lol. The backends de joure are either llama.cpp frontends (I use Kobold.cpp at the moment) or oobabooga as the guide specifies, but with the exLlamav2 backend. If you are serving a bunch of people, run a vLLM backend instead since it supports batching, and host it on the Horde if you are feeling super nice: https://lite.koboldai.net/# https://lite.koboldai.net/# Technically only vLLM will work with this new model at the moment, but I'm sure cpp/ooba support will be added within days. This comment will probably be obsolete within a month, when llama.cpp gets batching, MLC gets a better frontend, or some other breakthrough happens :P
- sp332 3y agollama.cpp support is here already via https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF And yes it works fine in oobabooga.