3 ms·

So... It's a language model..? As in, not "large"? I'm a bit unsure of the magnitudes here, but surely "nano" and "large" cancel out

• IanCal 1y ago
  No, vLLM is a thing for serving language models: https://github.com/vllm-project/vllm https://github.com/vllm-project/vllm
  • barrenko 1y ago
    Is it more like llama.cpp then? I don't have access to the good hardware.
    • jasonjmcghee 1y ago
      llama.cpp is optimized to serve one request at a time. vllm is optimized to serve many requests at one time. If you were to fine tune a model and wanted to serve it to many users, you would use vllm, not llama.cpp
      • jasonjmcghee 1y ago
        Here's a super relevant comment from another post https://news.ycombinator.com/item?id=44366418 https://news.ycombinator.com/item?id=44366418
        • barrenko 1y ago
          Appreciate it!