3 ms·
Here is a paper that made a similar observation recently: https://www.alphaxiv.org/abs/2512.19941 https://www.alphaxiv.org/abs/2512.19941
by WithinReason 7mo ago
Here is a paper that made a similar observation recently:
https://www.alphaxiv.org/abs/2512.19941 https://www.alphaxiv.org/abs/2512.19941
- tgw43279w 7mo agoVery cool, thanks for sharing! Recovering 96% using just two blocks on IMN-1k, wow!
- dnhkng 7mo agoThanks for the link! I think that these models have to learn to efficiently use their parameters, and the best way to do that is 'evolve' (yes, a bad word for it), structures over pretraining time. Unfortunately, they don't have a way to access these structures 'from the inside'. I hope this new approach lets up boost performance in s more experimentally rigorous way
- WithinReason 7mo agoI think the recurrence is a consequence of using a residual connection, seems like that makes the representation stay consistent across layers