I don't think implementing autograd is relevant or in-scope for learning about how transformers work (or writing out the gradient for transformer by hand, I can

I don't think implementing autograd is relevant or in-scope for learning about how transformers work (or writing out the gradient for transformer by hand, I can't even imagine doing that).