4 ms·
How is the time sharing handled? I assume if I submit a unit of work it will load to VRAM and then run (sharing time? how many work units can run in parallel?)
by kaoD 6mo ago
How is the time sharing handled? I assume if I submit a unit of work it will load to VRAM and then run (sharing time? how many work units can run in parallel?)
How large is a full context window in MiB and how long does it take to load the buffer? I.e. how many seconds should I expect my worst case wait time to take until I get my first token?
- ninjha 6mo ago> how many work units can run in parallel not original author but batching is one very important trick to make inference efficient, you can reasonably do tens to low hundreds in parallel (depending on model size and gpu size) with very little performance overhead
- jrandolf 6mo agovLLM handles GPU scheduling, not sllm. The model weights stay resident in VRAM permanently so there's no loading/unloading per request. vLLM uses continuous batching, so incoming requests are dynamically added to the running batch every decode step and the GPU is always working on multiple requests simultaneously. There is no "load to VRAM and run" per request; it's more like joining an already-running batch. TTFT is under 2 seconds average. Worst case is 10-30s.
- kaoD 6mo ago> The model weights stay resident in VRAM permanently so there's no loading/unloading per request. Yes, I was thinking about context buffers, which I assume are not small in large models. That has to be loaded into VRAM, right? If I keep sending large context buffers, will that hog the batches?
- deleted 6mo ago[deleted]
- jrandolf 6mo agoNot if you are the only one. We have rate limits to prevent this in case, idk, you share your key with 1000 people lol.