Lambda made compute cheap by treating code as ephemeral and execution state as disposable. LLM inference breaks that model. The execution state (weights shards

Lambda made compute cheap by treating code as ephemeral and execution state as disposable. LLM inference breaks that model. The execution state (weights shards, CUDA context, KV cache) is expensive and GPU-resident, so “scale to zero” usually means full re-initialization and cold starts. We’ve been working on a runtime that treats model execution state as the unit of scheduling, not the container or service. Instead of tearing everything down, we snapshot per-GPU execution state and restore it on demand, so models can scale down without paying the full cold-start cost again. This makes “model = function” closer to practical: fast scale-up, bounded latency, and GPUs shared across many models without keeping them always on. Let us know What you think about serverless inference, especially where the abstraction breaks today (KV cache, memory pressure, multi-GPU models, scheduling).