3 ms·
I'm serving AI models on Lambda Labs and after some trial and error I found having a single vllm server along with caddy, behind cloudflare dns, to work really
by rfurmani 2y ago
I'm serving AI models on Lambda Labs and after some trial and error I found having a single vllm server along with caddy, behind cloudflare dns, to work really well and really easy to set up
vllm serve ${MODEL_REPO} --dtype auto --api-key $HF_TOKEN --guided-decoding-backend outlines --disable-fastapi-docs &
sudo caddy reverse-proxy --from ${SUBDOMAIN}.sugaku.net --to localhost:8000 &
- mococa 2y agoYou can use Cloudflare Tunnel, which is even better and simple than having an extra service.
- homebrewer 2y agoIt's really best to avoid running web servers as root. It's easy to forward the port 80 with iptables, change the kernel knob to let unprivileged users use port 80 and above, or set the network capability on the binary. https://stackoverflow.com/questions/413807/ https://stackoverflow.com/questions/413807/