3 ms·
Their performance also varies depending on load (concurrent users).
by layer8 1y ago
Their performance also varies depending on load (concurrent users).
- BriggyDwiggs42 1y agoDear god does it really? That’s very funny.
- wiseowise 1y agoWhy are you surprised? It’s a computational thing, after all.
- BriggyDwiggs42 1y agoIt’s not that crazy, just the architecture of differently quantized models and so on that you’d need to do that is impressive considering.
- layer8 1y agoThe models are the same, it's the surrounding processing like "thinking" iterations that are adjusted.
- BriggyDwiggs42 1y agoThat only works for LRMs no? Not traditional LLM inference.