2 ms·
Thanks to this commit from 8 hours ago: https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6eb56678fdee4c09219ddb85b640 https://github.com/ggerganov/llam
by simonw 4y ago
Thanks to this commit from 8 hours ago: https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6eb56678fdee4c09219ddb85b640 https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e...
I have now successfully run the 13B model too! I updated my TIL with details: https://til.simonwillison.net/llms/llama-7b-m2#user-content-running-13b https://til.simonwillison.net/llms/llama-7b-m2#user-content-...
13B is the model that Facebook claim is competitive with original GPT3:
> LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla70B and PaLM-540B
- mrtksn 4y ago13B runs fine on my M1 Air with 16GB Ram and 8C8G. It's not even stressing out the memory and the swap as much as Stable Diffusion. I'm getting like 350-450ms per token and it feels as fast as ChatGPT on a busy day. This obviously isn't using the Neural Engine. With Apple's Stable Diffusion implementation, when the neural engine is used I can see how my CPU and GPU stays mostly idle and the temp on the Neural Engine cores is rising but it is rising significantly less than when run on the GPU or the CPU. I wonder of it's not possible to have this run on the Neural Engine? Given that it's mostly idle, running this locally will only impact the RAM use and on a machine with a large RAM it might feel like doesn't have a performance hit and run continuously for various task.