3 ms·
Running LLMs on that kind of hardware will be very slow (expect responses with only a few words per second, which is probably pretty annoying). LM Studio [1] m
by smokel 2y ago
Running LLMs on that kind of hardware will be very slow (expect responses with only a few words per second, which is probably pretty annoying).
LM Studio [1] makes it very easy to run models locally and play with them. Llama 3.1 will only run in quantized form with 16GB RAM, and that cripples it quite badly, in my opinion.
You may try Phi-3 Mini, which has only 3.8B weights and can still do fun things.
[1] https://lmstudio.ai/ https://lmstudio.ai/
- eth0up 2y agoMuch appreciated. Thanks for this!
- deleted 2y ago[deleted]
- wkat4242 2y agoI don't find llama3.1 noticeably worse on 8 bit integer quantised than the original fp16 to be honest. It's also a lot faster. Of course even then you're not going to reach the whole 128k context window on 16GB but if you don't need that it works great.