2 ms·
Jan is a good starting point. Desktop app that will guide you through downloading models you can run locally. Then https://jan.ai/docs/tools/retrieval https://
by jonahbenton 2y ago
Jan is a good starting point. Desktop app that will guide you through downloading models you can run locally. Then
https://jan.ai/docs/tools/retrieval https://jan.ai/docs/tools/retrieval
A similar tool, local desktop app, is LMStudio. Same deal.
https://lmstudio.ai/docs/basics/rag https://lmstudio.ai/docs/basics/rag
- rco8786 2y agoThanks so much! Any resources for deploying something like this into a cloud/datacenter and standing up an API?
- jonahbenton 2y agoThey both can be told via command line to run headless and start a server that is OpenAI compatible and can receive documents. Can be run in docker. The API modes are nevertheless not intended for concurrent use as a true server. Another project called VLLM, can be found on github, is intended to support concurrent OpenAI compatible API service, multiplexing over the available GPU.
- jaredsohn 2y agoHow expensive is it to run these in the cloud? Last time I looked into this it looked like you have to pay significantly in server costs to be able to run a local LLM.