3 ms·

Thanks so much! Any resources for deploying something like this into a cloud/datacenter and standing up an API?

• jonahbenton 2y ago
  They both can be told via command line to run headless and start a server that is OpenAI compatible and can receive documents. Can be run in docker. The API modes are nevertheless not intended for concurrent use as a true server. Another project called VLLM, can be found on github, is intended to support concurrent OpenAI compatible API service, multiplexing over the available GPU.
  • jaredsohn 2y ago
    How expensive is it to run these in the cloud? Last time I looked into this it looked like you have to pay significantly in server costs to be able to run a local LLM.