A Survey of Multi-Tenant Deep Learning Inference on GPU (2022) https://arxiv.org/pdf/2203.09040.pdf https://arxiv.org/pdf/2203.09040.pdf "As one of the most c

A Survey of Multi-Tenant Deep Learning Inference on GPU (2022) https://arxiv.org/pdf/2203.09040.pdf https://arxiv.org/pdf/2203.09040.pdf "As one of the most competitive kernel auto-tuning framework, TVM comes with a built-in assumption of single-tenant execution setting, the tuning configuration of which aims to saturate all SMs and memory bandwidth of the GPU. Such assumption and the single-tenant targeted tuning however becomes unsuitable for multi-kernel concurrent execution with partial resource available for each kernel. According to [7], the maximum throughput gap comparing single-tenant vs. multi-tenant tuned configurations for the same computing kernel could reach 5× difference." TIL: somehow I had assumed Apache TVM permitted multi-tenancy.