5 ms·
Nano-Vllm: Lightweight vLLM implementation built from scratch
- deleted 1y ago[deleted]
- unwind 1y agoMeta: the Title Casing in the title is pretty obnoxious, "Vllm" is exactly the inverse, casing-wise, of how the project spells its name.
- msephton 1y agoFwiw op has a small window of time to correct the casing after posting
- futurecliff 1y agohow did u do it? which portion of vllm refactoring allowed u to get such gains.
- zackify 1y agoWill this end up getting an open ai compatible web server or is that out of scope.
- jimmySixDOF 1y agoLittle sparse on the documentation side can't tell at a glance if there is a 1:1 hyperperameter tuneability or if this is an opinionated single path locked soft fpga eval-hacking kind of thing. EDIT: -- Ok, it's legit, here is an example of it put to use by the makers of the Dolphin OpenSource series of FineTunes: > Here I implement in nano-vllm, efficient sample-K logit extraction, as described in "Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs" by Anshumann et. al. Sampling occurs on the GPU, the non-sampled logits do not get copied out of GPU space. I tried to implement this in @vllm_project, but it was a bit too heavy for me to figure out. https://github.com/GeeeekExplorer/nano-vllm/pull/34 https://github.com/GeeeekExplorer/nano-vllm/pull/34
- baalimago 1y agoSo... It's a language model..? As in, not "large"? I'm a bit unsure of the magnitudes here, but surely "nano" and "large" cancel out
- IanCal 1y agoNo, vLLM is a thing for serving language models: https://github.com/vllm-project/vllm https://github.com/vllm-project/vllm
- barrenko 1y agoIs it more like llama.cpp then? I don't have access to the good hardware.
- jasonjmcghee 1y agollama.cpp is optimized to serve one request at a time. vllm is optimized to serve many requests at one time. If you were to fine tune a model and wanted to serve it to many users, you would use vllm, not llama.cpp
- jasonjmcghee 1y agoHere's a super relevant comment from another post https://news.ycombinator.com/item?id=44366418 https://news.ycombinator.com/item?id=44366418
- barrenko 1y agoAppreciate it!
- fractorial 1y agoDid anyone else click in excitedly after misreading ‘Vllm’ as ‘LLVM?’
- omneity 1y agoThis is an incredible achievement for a solo developer. The dev is from the Deepseek team by the way.
- Imustaskforhelp 1y agoThat is crazy! This is so cool ngl.
- tt726259 1y agoAfter seeing the Docker image for vllm jump +5Gb (to 10Gb!) over the past five months, I grew suspicious of vllm's development practices [1]. It's not easy, for sure, to deal with all those flaky python modules [2]. But having the CUDA packages four times in different layers is questionable! [3] Yet again, as a college mate of mine used to say, "Don't change it. It works." -- [1]: https://hub.docker.com/r/vllm/vllm-openai/tags https://hub.docker.com/r/vllm/vllm-openai/tags [2]: https://github.com/vllm-project/vllm/issues/13306 https://github.com/vllm-project/vllm/issues/13306 [3]: These kinds of workarounds tend to end up accumulating and never get reviewed back: - https://github.com/vllm-project/vllm/commit/b07d741661570ef199ba92528b11c0bd1294fb15 https://github.com/vllm-project/vllm/commit/b07d741661570ef1... - https://github.com/vllm-project/vllm/commit/68d37809b9b52f4d012fa0dfbb187f0fe978bdbc https://github.com/vllm-project/vllm/commit/68d37809b9b52f4d... (this one in particular probably accounts for +3Gb)
- b0a04gl 1y ago[dead]
- mountainriver 1y agoLove this project, we need more simplifications like this in the current ML environment