4 ms·
This looks good, thanks for sharing. Can you share a little more about where the compute is performed, the transcription and llm? Its a local machine or a machi
by wanderingmind 3y ago
This looks good, thanks for sharing. Can you share a little more about where the compute is performed, the transcription and llm? Its a local machine or a machine in cloud? If its local, i assume it cant be used by other people in server? If its on a network/cloud machine, how is the communication handled between discord and the machine? Also what modifications need to be done to do longer streams of audio data?
- tynmann 3y agoI'm running everything on my own PC. With a 3060 12 GB at under $300, it's very cheap to build a system capable of running inference with smaller models. But even bigger models (13B or 30B) can be handled with a dual GPU setup (24 GB total memory at < $600), with quantization. The Discord bot works as a client (outgoing connection to the Discord public server), so it can run from behind NAT.
- tynmann 3y agoHowever if one wants to run something like this in production, it would make more sense to use a server with at least an A10, which has better performance per power consumption.