3 ms·
> The model weights stay resident in VRAM permanently so there's no loading/unloading per request. Yes, I was thinking about context buffers, which I assume ar
by kaoD 6mo ago
> The model weights stay resident in VRAM permanently so there's no loading/unloading per request.
Yes, I was thinking about context buffers, which I assume are not small in large models. That has to be loaded into VRAM, right?
If I keep sending large context buffers, will that hog the batches?
- deleted 6mo ago[deleted]
- jrandolf 6mo agoNot if you are the only one. We have rate limits to prevent this in case, idk, you share your key with 1000 people lol.