4 ms·
Any plans for being able to run the entire thing locally with local models?
by danielbln 2y ago
Any plans for being able to run the entire thing locally with local models?
- _joel 2y agoYou can make an OpenAI compatible local server using LMStudio[1] and load any model you want. It'd have to be on another host though, the s3 has some inference capabilities with addons afair, but nowhere enough grunt to run locally at any usable token/s [1] https://lmstudio.ai/ https://lmstudio.ai/
- homarp 2y agoor with open source tools like llama.cpp - https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md https://github.com/ggerganov/llama.cpp/blob/master/examples/... or mistral.rs - https://github.com/EricLBuehler/mistral.rs/blob/master/docs/HTTP.md https://github.com/EricLBuehler/mistral.rs/blob/master/docs/... lmstudio and ollama use llama.cpp underneath. cut the middle man
- zq2240 2y agoThanks for asking. We will launch local LLM, STT, and TTS models in the near future versions.