You can leverage Nvidia Triton Inference server to do this. In case you want a managed Infra service look at this cookbook on how you can use Transformers, Inf

You can leverage Nvidia Triton Inference server to do this. In case you want a managed Infra service look at this cookbook on how you can use Transformers, Inferless to serve multiple LLMs - https://docs.inferless.com/cookbook/serverless-voice-chatbot https://docs.inferless.com/cookbook/serverless-voice-chatbot