Inference is still very computationally taxing for deep learning models. Using GPUs in a cluster for inference often proves far more efficient than large number

Inference is still very computationally taxing for deep learning models. Using GPUs in a cluster for inference often proves far more efficient than large numbers of CPUs (eg; spark) but then that suffers from issues like I/O bottlenecks, memory bandwidth bottlenecks, etc. So yeah, it's "read-only" but it's not so trivial to naively scale due to the large amounts of computations involved and often enormous datasets.