You can have load balanced inference and also fine tune models on Modal (disclaimer: I wrote this guide) You can get a load balanced endpoint serving hundreds

You can have load balanced inference and also fine tune models on Modal (disclaimer: I wrote this guide) You can get a load balanced endpoint serving hundreds of tok/s per GPU using vLLM or Text Generation Inference in a few lines of code: https://modal.com/docs/guide/ex/text_generation_inference https://modal.com/docs/guide/ex/text_generation_inference