2 ms·
You can use a raspberry pi with 8 GB of ram to run a quantised 7B model (eg. https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF https://huggingface.co/TheBlok
by wokwokwok 2y ago
You can use a raspberry pi with 8 GB of ram to run a quantised 7B model (eg. https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF), or any cheap stick pc.
For larger models, or GPU accelerated inference, there is no “cheap” solution.
Why do you think everyone is so in love with the 7B models?
It’s not because they’re good. They’re just ok, and it’s expensive to run larger models.