I'm running everything on my own PC. With a 3060 12 GB at under $300, it's very cheap to build a system capable of running inference with smaller models. But e

I'm running everything on my own PC. With a 3060 12 GB at under $300, it's very cheap to build a system capable of running inference with smaller models. But even bigger models (13B or 30B) can be handled with a dual GPU setup (24 GB total memory at < $600), with quantization. The Discord bot works as a client (outgoing connection to the Discord public server), so it can run from behind NAT.