For large models, there are two main ways folks have been optimizing machine learning execution: 1. lowering precision of the operations (reducing compute "wid

For large models, there are two main ways folks have been optimizing machine learning execution: 1. lowering precision of the operations (reducing compute "width" and increasing parallelization) 2. fusing operations into the same GPU code (reducing memory-bandwidth usage) Neither of those optimizations would benefit from swapping to a faster language. Why? The typical "large" neural network operation runs on the order of a dozen microseconds to milliseconds. Models are usually composed of hundred if not thousands of these. The overhead of using Python is around 0.5 microseconds per operation (best case on Intel, worst case on Apple ARM). So that's maybe a 5% net loss if things were running synchronously. But they're not! When you call GPU code, you actually do it asynchronously, so the language latency can be completely hidden. So really, all you want in an ML language is the ability to 1. change the type of the underlying data on the fly (Python is really good at this) and 2. rewrite the operations being dispatched to on the fly (Python is also really good at this). For smaller models (i.e. things that run in sub-microsecond world), Python is not the right choice for training or deploying.