3 ms·

I feel like we screwed up the AI chip segmentation with separate “inference” and “training” chips Every machine learning application ever involves both trainin

I feel like we screwed up the AI chip segmentation with separate “inference” and “training” chips Every machine learning application ever involves both training and inference. What is the point of a model that can’t learn? What is the point of learning a great model that is never used? It seems like these lead to different optimizations, and they do. But the bigger lock-in is the SW stack. If you own one side of it, you have a huge advantage in SW compatibility that you can leverage into the other.

dagmx 2y ago
Most inference can be easily abstracted over multiple ISV implementations so the inference side rarely is locked in. It’s usually the training side or software above the inference layer that’s locked in. But as to your question as to why: silicon size and frequency of use. You’re almost always going to have a high ratio of inference to training. Silicon for training needs to be much more capable due to how intensive it is. That’s then a block of silicon that isn’t used to its full potential, takes up valuable space and in turn jacks up cost to buy and cost to run (energy). It’s simply not a worthwhile trade off.
flembat 2y ago
The point of the separation is economic, owners create and train models, and renters are charged to execute them on their edge machines.
RamblingCTO 2y ago
> Every machine learning application ever involves both training and inference. Not really, no. There are a lot of "classical" models that you don't want to update on the fly. I'd even argue that it's a minority.