2 ms·
Nvidia not name products after existing things in the ML space challenge: IMPOSSIBLE More seriously, though: > OpenAI Compatible Frontend – High performance O
by saagarjha 2y ago
Nvidia not name products after existing things in the ML space challenge: IMPOSSIBLE
More seriously, though:
> OpenAI Compatible Frontend – High performance OpenAI compatible http api server written in Rust.
Is this normal in this space? I know everyone has settled on copying the S3 API for block storage but I’m unsure if we’ve done the same for LLM serving.
- DeveloperErrata 2y agoIncreasingly so. Many other popular inference tools in this space also expose an OpenAI compatible API: VLLM, Llama.cpp, and LiteLLM all do.