If you have 8GB you can play around with heavily quantized 7B models, or up to moderately quantized 13GB models w/ 16GB. Something like a Mistral-7B Dolphin fi

If you have 8GB you can play around with heavily quantized 7B models, or up to moderately quantized 13GB models w/ 16GB. Something like a Mistral-7B Dolphin finetune actually is surprisingly useful, like GPT-3.5 in some respects. I imagine it would render ~10 tok/sec on an M2 for at least short bursts until throttling sets in. As I mentioned in another post, try this out with LM Studio. Super simple GUI that even a non-tech person could probably figure out for finding/downloading/loading models w/ a ChatGPT like interface for chatting. https://lmstudio.ai/ https://lmstudio.ai/