Useful LLMs are probably still a tad to large to run locally on device. Loading and unloading the model takes several seconds, so ideally the model would just s

Useful LLMs are probably still a tad to large to run locally on device. Loading and unloading the model takes several seconds, so ideally the model would just sit in RAM at all times. In my experience playing with local LLaMA variants, ChatGPT level responses are really only possible with 13B parameter models and up. With 4bit quantization that's 6.5GB of RAM usage just for the model weights. You could totally run that on a phone today, but you'll be getting 2 tokens/s and using the majority of the phone's RAM. A responsive, local LLM on a phone is going to happen someday soon, but we need some serious hardware improvements first. I'd speculate you'll need at least 24GB of shared memory, and considerably more dedicated tensor multiplication cores. Alternatively, some sort of GPT architecture change that reduces the memory overhead could also drive local LLMs. Who knows, maybe Apple has a trick or two up their sleeves.