There have been huge advances in the mathematics of neural networks from Greg Yang (formerly of Microsoft). This allowed predictable training-hyperparameter tra

There have been huge advances in the mathematics of neural networks from Greg Yang (formerly of Microsoft). This allowed predictable training-hyperparameter transfer from smaller versions of GPT-4 where they could be tuned, to the final large model. https://www.microsoft.com/en-us/research/uploads/prod/2021/11/TP5.pdf https://www.microsoft.com/en-us/research/uploads/prod/2021/1... He has proofs and theorems about frontiers of maximal feature learning before things devolve into equivalent to kernel methods, and more: a whole bunch of breakthrough math making deep links with random matrix theory.