So you can go with Kubernetes. This is my preferred tool. With Kubernetes, you can either wrap your model inside a container or mount it into the container fro

So you can go with Kubernetes. This is my preferred tool. With Kubernetes, you can either wrap your model inside a container or mount it into the container from a persistent volume. As for scaling you have two options: 1) Horizontal Pod Autoscaler https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ https://kubernetes.io/docs/tasks/run-application/horizontal-... 2) Knative, which is Kubernetes serverless on-prem solution.