4 ms·

I'll be honest that I've never actually considered tokens per second as something to focus on for my projects, I'm much more concerned with quality of the outpu

• oscarrovira 3y ago
  There’s a percentage of users that do care about token generation speed as they chain multiple API calls. The performance is all thanks to TensorRT-LLM, Mystic takes care of the engineering of getting a scalable endpoint out of it, i.e, not having to manage your infra.
• MareAlfred24 3y ago
  [dead]
• v1vek 3y ago
  the latency to serve the end-user esp with chained calls impacts ux quality
• MrBra 3y ago
  Improvements in inference speed would also manifest itself on those bigger models that may only partially fit into GPU VRAM. In some cases, the improvement on the GPU side alone, is strong enough to basically turn what you would previously consider a too-slow-to-be-usable higher quality model, into a faster, usable one.