4 ms·
You shouldn't be having slow response issues with LLM/TTS/STT for HA on a mbp m3 max 128gb. I'd either limit the entities exposed or choose a smaller model.
by nickthegreek 2y ago
You shouldn't be having slow response issues with LLM/TTS/STT for HA on a mbp m3 max 128gb. I'd either limit the entities exposed or choose a smaller model.
- joshstrange 2y agoOh, I can get smaller models to run reasonably fast but I'm very interested in tool calling and I'm having a hard time finding a model that runs fast and is good at calling tools locally (I'm sure that's due to my own ignorance).
- nickthegreek 2y agoI decided on openai api for now after setting up so many differnt methods. the local stuff isn't up to snuff yet for what I am trying to accomplish but decent for basic control.
- joshstrange 2y agoI use a combo of Anthropic and OpenAI for now through my bots and my chat UIs and that lets me iterate faster. My hope is once I've done all my testing I could consider moving to local models if it made sense.