6 ms·
Unfortunately a 13B model quantized down to even 2 or 3 bits won't fit into 8gb of ram. You might have to settle for 7B models. I'm currently getting great re
by gorbypark 3y ago
Unfortunately a 13B model quantized down to even 2 or 3 bits won't fit into 8gb of ram. You might have to settle for 7B models. I'm currently getting great results out of the Mistral-Openorca-7B model. At 5 bits it's using about 5.5gb of ram and running at 14 tokens/sec on my M2 MacBook Air (24GB ram, but in theory would work on 8gb if MacOS will allow you to allocate that much ram to the GPU). As a quick test, forcing it into CPU mode, I'm still getting 11 tokens/sec. It does seem to take about 2-3 times longer to initialize itself and get into the "ready state" when using the CPU, however.
- calgoo 3y agoI have been running some 7b models on my 8gb MacBook Air m2. It works, as long as you don’t want to do anything else at the same time. It brings my system down to a crawl, but it works, and having cli ollama binary is enough atm.