4 ms·
Attention Residuals: Rethinking depth-wise aggregation [pdf]
- krackers 7mo agoIn [1] I think a commenter actually speculated about a design just like this, where later layers can directly access outputs of previous layers instead of having to store it in the residual stream [1] https://news.ycombinator.com/item?id=46362579 https://news.ycombinator.com/item?id=46362579