2 ms·
If it takes so much resource to run, how does the sharing of a single llm works? There is some interface that basically submits context/cache plus current promt
by blks 2mo ago
If it takes so much resource to run, how does the sharing of a single llm works? There is some interface that basically submits context/cache plus current promt, from each user, doing basically time-sharing compute?
- mnahkies 2mo agoI believe they batch requests so the same weights in vram are shared across many requests https://www.baseten.co/blog/continuous-vs-dynamic-batching-for-ai-inference/ https://www.baseten.co/blog/continuous-vs-dynamic-batching-f...