3 ms·
My argument here is that this represents a tipping point. Prior to LLaMA + llama.cpp you could maybe run a large language model locally... if you had the right
by simonw 4y ago
My argument here is that this represents a tipping point.
Prior to LLaMA + llama.cpp you could maybe run a large language model locally... if you had the right GPU rig, and if you really knew what you were doing, and were willing to put in a lot of effort to find and figure out how to run a model.
My hunch is that the ability to run on a M1/M2 MacBook is going to open this up to a lot more people.
(I'm exposing my bias here as a M2 Mac owner.)
I think the race is now on to be the first organization to release a good instruction-tuned model that can run on personal hardware.
- stonerri 4y agoAs someone who just got the 7B running on a base MacBook M1/8GB, I strongly agree. The rate of tool development & prompt generation should see the same increase that Stable Diffusion did a few months (weeks?) ago. And given how early the cpp port is, there is likely plenty of performance headroom with more m1/m2-specific optimization.
- Vetch 4y agoIt's not just that it's accessible, it's also significantly higher in quality than previous local runnable causal LMs. I suspect people saying it's not good are prompting it like ChatGPT, not realizing how much trickier a raw model is to prompt. Getting the hyperparameters for good sampling is another stumbling block. The models are very good if you do everything properly.
- reasonabl_human 4y agoInteresting, where can I learn more about prompting, and tuning a raw model?
- simonw 4y agoThere are a few initial tips here in the LLaMAA FAQ: https://github.com/facebookresearch/llama/blob/main/FAQ.md#2-generations-are-bad https://github.com/facebookresearch/llama/blob/main/FAQ.md#2...