3 ms·
Cool idea. I didn't understand what causes the bad response on the first query. Does it mean the first response in every new conversation, or just the first ser
by smallerize 2mo ago
Cool idea. I didn't understand what causes the bad response on the first query. Does it mean the first response in every new conversation, or just the first served response after startup?
- nmitchko 2mo agoAfter vLLM startup I observed this on first request only then all further conversations were fine, new and original. Can’t figure out why but seems to be fine after that.