I had the same inspiration! I spent a days reviewing the lecture notes for CS336: Language Modeling from Scratch - and then trained a nanoGPT-esque model in Py

I had the same inspiration! I spent a days reviewing the lecture notes for CS336: Language Modeling from Scratch - and then trained a nanoGPT-esque model in PyTorch. I'd recommend trying it for those who are curious. Computational bottlenecks become much more intuitive when you've looked at the overall process.