We use transformers in production for T5 models, which aren't supported by the usual suspects. I did a bit of poking around and might try out TensorRT or CTran

We use transformers in production for T5 models, which aren't supported by the usual suspects. I did a bit of poking around and might try out TensorRT or CTranslate2 at some point; for now the single GPU we have dedicated to the project has been able to keep up with incoming data, but I'll probably have to address it in the next couple of months. If anyone has suggestions for other T5 inference options I would welcome them.