I think you misunderstand the training of many types of large statistical models. It’s almost always I/O bound (getting batches of data into memory or shipped t

I think you misunderstand the training of many types of large statistical models. It’s almost always I/O bound (getting batches of data into memory or shipped to GPU) and the CPU bound part is intensely optimized SIMD matrix algebra operations. The hard part is always that you have more data than can fit into memory, and need complex prefetch solutions to parallelize getting the next batch of data while one batch’s numerical linear algebra computation is in progress. Many libraries handle this transparently for the user in an extension module, such as Keras & PyTorch. I think it’s misleading to act incredulous that this is the bottleneck ... it doesn’t have anything to do with Python, GIL, etc. (and there are easy-to-maintain solutions for this in Python).