You probably can do this now. Non-generative LLMs don't need to be as big so something like Gemma 4B on the CPU will work. You may have better results with sem

You probably can do this now. Non-generative LLMs don't need to be as big so something like Gemma 4B on the CPU will work. You may have better results with semi-templated responses though.