One thing people generally don't highlight about the transformer architecture, which I think is very important, are the pass through connections. This allows th

One thing people generally don't highlight about the transformer architecture, which I think is very important, are the pass through connections. This allows the model to "spread" out it's learnings across the network rather than them being forced to very local. I think this is also a reason why the Resnet was so successful.