In [1] I think a commenter actually speculated about a design just like this, where later layers can directly access outputs of previous layers instead of havin

In [1] I think a commenter actually speculated about a design just like this, where later layers can directly access outputs of previous layers instead of having to store it in the residual stream [1] https://news.ycombinator.com/item?id=46362579 https://news.ycombinator.com/item?id=46362579