3 ms·
Congrats on the launch! Your service looks interesting, but I think you need to be more transparent about your infrastructure. To which "inference providers" d
by imiric 2y ago
Congrats on the launch!
Your service looks interesting, but I think you need to be more transparent about your infrastructure. To which "inference providers" do you proxy to, and when? Who is hosting the GPU clusters?
Also, a privacy policy and ToS document are pretty important, even at this stage.
- brianjking 2y agoyeah, I called this out too. How this compares to OpenRouter is something I'd like to know as well. https://openrouter.ai/models/meta-llama/llama-3.1-405b-instruct https://openrouter.ai/models/meta-llama/llama-3.1-405b-instr...
- adr1an 2y agoiirc openrouter doesn't support any vLLM or huggingface pasted link.
- reissbaker 2y agoGood callout! Currently: 1. If Together.ai has the model, we proxy to them, since they're faster than us. We might switch to Fireworks for the Llama-3.1 models since they offer them at lower cost; in general that's why I didn't specify the inference providers, since we'll probably change and optimize that quite a bit. (Also would be interesting to try using Groq, since they're so fast.) 2. If the model isn't hosted somewhere else, e.g. a lot of the Llama 3 finetunes, we run the models on our own GPU clusters hosted in Fly.io. This will probably change in the future as well, since some models would really benefit from NVLink (which Fly doesn't support currently).