This strikes me as an extremely difficult but not intractable problem. I'm not sure what the state of the art in compiler optimisation is with regard to data p

This strikes me as an extremely difficult but not intractable problem. I'm not sure what the state of the art in compiler optimisation is with regard to data positioning and targeting maximum processor usage There was a video on optimisation a while back that showed small optimisations caused increases in speed that were insignificant when compared to the speed variance induced by the memory layout that the optimisation (or even a random change) caused. While that talk was more focused on getting a signal past the noise. That noise itself is an artifact of compilers being not particularly good at handling a much simpler form of the problem you describe. CPU and memory architectures are complex when caches and access patterns impact upon speed. When you add in GPU architectures to the mix I think you might be in fairly uncharted territory. Maybe one day. Of course since we are in the field of AI there is also the question of could a sufficiently smart AI do this. It depends on the value of sufficient. I would like to think that an extremely high level test for an AI model could be to give it something like micrograd and tell it to produce something with the same interface that outperforms torch. We're not even in the ballpark of being able to do that yet, but it will be interesting when and if that happens.