3 ms·

I'm confused. I always thought that once you trained a model then you can use to do inference and at that point the model is only a read-only data structure hen

I'm confused. I always thought that once you trained a model then you can use to do inference and at that point the model is only a read-only data structure hence is trivial to scale out inference. I guess you're talking about something else. Could you please elaborate?

alfalfasprout 8y ago
Inference is still very computationally taxing for deep learning models. Using GPUs in a cluster for inference often proves far more efficient than large numbers of CPUs (eg; spark) but then that suffers from issues like I/O bottlenecks, memory bandwidth bottlenecks, etc. So yeah, it's "read-only" but it's not so trivial to naively scale due to the large amounts of computations involved and often enormous datasets.