I think that whatever combination of tools/docs/compilers that makes row vs. column major notationally equivalent, amenable to introspection, and equally amenab

I think that whatever combination of tools/docs/compilers that makes row vs. column major notationally equivalent, amenable to introspection, and equally amenable to optimization will win big. I’ve written godawful python extensions to feed TF or PyTorch mmapped regions to the pipeline in an efficient way. This is currently the way of things. But half the trouble in model architecture engineering is keeping the ranks straight and having tools to debug when you messed up. Einstein struggled with tensor math, read his letters. But he didn’t have modern computers or autodiff either. For that era, thinking in higher dimensions was a job for a genius. I harbor a vague suspicion that some people get paid more because it’s needlessly hard.