3 ms·
You can make an OpenAI compatible local server using LMStudio[1] and load any model you want. It'd have to be on another host though, the s3 has some inference
by _joel 2y ago
You can make an OpenAI compatible local server using LMStudio[1] and load any model you want. It'd have to be on another host though, the s3 has some inference capabilities with addons afair, but nowhere enough grunt to run locally at any usable token/s
[1] https://lmstudio.ai/ https://lmstudio.ai/
- homarp 2y agoor with open source tools like llama.cpp - https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md https://github.com/ggerganov/llama.cpp/blob/master/examples/... or mistral.rs - https://github.com/EricLBuehler/mistral.rs/blob/master/docs/HTTP.md https://github.com/EricLBuehler/mistral.rs/blob/master/docs/... lmstudio and ollama use llama.cpp underneath. cut the middle man