3 ms·
I’m most surprised that the embedding is actually linear. I wonder if that makes extrapolation perform better
by klysm 3y ago
I’m most surprised that the embedding is actually linear. I wonder if that makes extrapolation perform better
- nyrikki 3y agoFeed forward networks are pairwise linear in respect to inputs, they are effectively DAGs. In theory you could represent an entire LLM as a single 2 dimensional graph of linear line segments. It wouldn't be useful for much as the parameters are clustered in dense patches.
- kridsdale3 3y agoSo are my synapses, and by and large my cortex is 2-dimensional as well (folded and scrunched in to a super complex shape by the limitations of my roughly spherical cranium)
- nyrikki 3y agoEverything from ion channel response up is non-linear and not a deterministic Turing machine. Random cite https://www.jneurosci.org/content/13/1/334?ijkey=9f8730dd3d028900055afd202f57b98d5498b776&keytype2=tf_ipsecsha https://www.jneurosci.org/content/13/1/334?ijkey=9f8730dd3d0...
- xpe 3y agoThe papers cites earlier papers that posit a ‘linear representation hypothesis’. Does your surprise “factor in” these papers? Do you suspect something different in play for the current paper?
- klysm 3y agoDidn’t have that factored into my surprise because I’m not that informed. I just thought that the latent space representation being linear was interesting.
- toxik 3y agoMaybe something to do with the Transformers are mesa optimizers thing?