3 ms·
May be worth pointing out, that this is not the first residual connection innovation to be in production. Gemma 3n is also using a low-rank projection of the r
by cpldcpu 9mo ago
May be worth pointing out, that this is not the first residual connection innovation to be in production.
Gemma 3n is also using a low-rank projection of the residual stream called LAuReL. Google did not publicize this too much, I noted it when poking around in the model file.
https://arxiv.org/pdf/2411.07501v3 https://arxiv.org/pdf/2411.07501v3
https://old.reddit.com/r/LocalLLaMA/comments/1kuy45r/gemma_3n_architectural_innovations_speculation/ https://old.reddit.com/r/LocalLLaMA/comments/1kuy45r/gemma_3...
Seems to be what they call LAuReL-LR in the paper, with D=2048 and R=64
- taykolasinski 9mo agoThis is a fantastic catch. I hadn't realized Gemma 3n was already shipping with a variant of this in production. It feels like we are entering the era of residual stream engineering. For a long time, the standard x + F(x) additive backbone was treated as untouchable. Now, between mHC (weighted scaling) and LAuReL (low-rank projections), labs are finally finding stable ways to make that signal path more dynamic. I'm curious if the Low-Rank constraint in LAuReL acts as a natural stabilizer against the gradient explosion I saw with unconstrained hyper-connections. Thanks for the paper link, definitely reading that tonight.
- cpldcpu 9mo agoThanks! Would be quite interesting to see how this fares compared to mHC. I noted that LAuReL is cited in the mHC paper, but they refer to it as "expanding the width of the residual stream", which is rather odd.