There is no need to use smaller models. You can run the biggest models such as llama 3.1 405B on a fairly low end desktop today: https://github.com/lyogavin/ai

There is no need to use smaller models. You can run the biggest models such as llama 3.1 405B on a fairly low end desktop today: https://github.com/lyogavin/airllm https://github.com/lyogavin/airllm However, it will be far slower as you said.