3 ms·
Show HN: Chat with Zephyr 7B over voice message using whisper.cpp and llama.cpp
Just my small weekend project, experimenting with LLMs. I made a Discord bot that uses the Zephyr 7B model to reply, and whisper.cpp to transcribe voice messages into text. So you can send it voice messages and it replies.
- wanderingmind 3y agoThis looks good, thanks for sharing. Can you share a little more about where the compute is performed, the transcription and llm? Its a local machine or a machine in cloud? If its local, i assume it cant be used by other people in server? If its on a network/cloud machine, how is the communication handled between discord and the machine? Also what modifications need to be done to do longer streams of audio data?
- tynmann 3y agoI'm running everything on my own PC. With a 3060 12 GB at under $300, it's very cheap to build a system capable of running inference with smaller models. But even bigger models (13B or 30B) can be handled with a dual GPU setup (24 GB total memory at < $600), with quantization. The Discord bot works as a client (outgoing connection to the Discord public server), so it can run from behind NAT.
- tynmann 3y agoHowever if one wants to run something like this in production, it would make more sense to use a server with at least an A10, which has better performance per power consumption.