There are also things like Oobabooga's text-generation-webui[0] which can present a similar interface to ChatGPT for local models. I've had great success in ru

There are also things like Oobabooga's text-generation-webui[0] which can present a similar interface to ChatGPT for local models. I've had great success in running Qwen3-8B-GGUF[1] on my RTX 2070 SUPER (8GB VRAM) using Oobabooga (everyone just calls it via the author's name, it's much catchier) so this is definitely doable on consumer hardware. Specifically, I run the Q4_K_M model as Oobabooga loads all of its layers into the GPU by default, making it nice and snappy. (Testing has shown that I can actually load up to the Q6_K model before some layers have to be loaded into the CPU, but I have to manually specify that all those layers should be loaded into the GPU, as opposed to leaving it auto-determined.) It does obviously hallucinate more often than ChatGPT does, so care should be taken. That said, it's really nice to have something local. There's a subreddit for running text gen models locally that people might be interested in: https://www.reddit.com/r/LocalLlama https://www.reddit.com/r/LocalLlama [0] https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui [1] https://huggingface.co/Qwen/Qwen3-8B-GGUF https://huggingface.co/Qwen/Qwen3-8B-GGUF