5 ms·
@antimatter15 - how much ram does your M2 MacBook Air have? Performance looks very good. Curious if we can extrapolate about the requirements for the larger mod
by phren0logy 4y ago
@antimatter15 - how much ram does your M2 MacBook Air have? Performance looks very good. Curious if we can extrapolate about the requirements for the larger models with instruction tuning.
- zitterbewegung 4y agoUsing the scripts from llama.cpp I was able to run the 65B model on my M1 Max which used ~40 GB of RAM. With 32GB you should be able to run 33B and lower. See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3db526d5d6fd https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...
- AndroidKitKat 4y agoNo chance I'd be able to run this on an 8GB M1 Mac mini then? Looking to hook this up to an IRC bot for a bit of fun with my friends and have this spare Mac lying around.
- zitterbewegung 4y ago7B will be fine I have heard people getting llama working on a nexus 6
- madmads 4y agoThe 7b model loads on a lot less than 8GB RAM, so you should be good to use it on an 8GB M1 Mac Mini. It takes very little time to set up, so it's worth trying to find out.
- AndroidKitKat 4y agoYeah I'll get this set up when I'm home from work! Perhaps the trickiest part for me is going to be hooking it up to IRC, but that's the fun part!
- phren0logy 4y agoThanks!
- recuter 4y agoThere are efforts like https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa that might further compress it so 65B runs on 32GB machines, 33B and lower would run on 16GB machines, and 13B (minimal useful one seems like) would fit on 8GB -- which is wild and as good as you could hope for.
- gorbypark 4y agoThe 30B variant of LlaMa quantized to 4bit and running with llama.cpp takes approximately 20GB of ram. It runs pretty decently on my M2 MacBook Air with 24GB of ram. I assume Alpaca will take the same amount of ram.