4 ms·
I downloaded two files, ran `llamafile-server-0.4 -ngl 35 -m mistral-7b-instruct-v0.1.Q4_K_M.gguf` and was getting 53 tokens/second on my RTX 2070 Super 8GB. It
by pseudosavant 3y ago
I downloaded two files, ran `llamafile-server-0.4 -ngl 35 -m mistral-7b-instruct-v0.1.Q4_K_M.gguf` and was getting 53 tokens/second on my RTX 2070 Super 8GB. It couldn't be easier.
This is really going to make me want to try using a local model to help with coding. I never get responses that fast on ChatGPT with GPT-4.