The author says to add a unity vector to the context, i presume of each layer, to not mess with gradient calculations. But most modern DL frameworks compute the

The author says to add a unity vector to the context, i presume of each layer, to not mess with gradient calculations. But most modern DL frameworks compute the gradient for you, (i know this is true for JAX and Pytorch). Is it maybe that hand coded gradient for a well-known enough dl architecture like transformer is faster than letting the framework autodiff it? Otherwise, i fear some of the 'magic' of transformer networks is that this amplification effect allows it to encode/memorize some results verbatim. And we often are seeing a heavily tuned internet regurgitator. So similar to the rise of RNNs with attention, which supposedly allowed them to focus on some things and ignore others but really often was just overfitting stuff, yielded more interesting results with the overfitting than without.