2 ms·
Another great way to understand how vllm works is to read the code of nano-vllm[1]. It's basically "vllm but cut down to size. It's ~5kloc, supports just one mo
by miki123211 3mo ago
Another great way to understand how vllm works is to read the code of nano-vllm[1]. It's basically "vllm but cut down to size. It's ~5kloc, supports just one model, disposes of some of the abstraction layers that vllm needs due to its codebase size, but contains all the major pieces that make an inference engine fast.
[1] https://github.com/GeeeekExplorer/nano-vllm https://github.com/GeeeekExplorer/nano-vllm
- yu3zhou4 2mo agoIf you prefer C++ and CUDA, then there's also tiny-vllm of mine [0] - recently we broke 1k gh stars [0] https://github.com/jmaczan/tiny-vllm https://github.com/jmaczan/tiny-vllm