3 ms·

"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead." It just baffles me that this even works at all

"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead." It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?

itkovian_ 2mo ago
Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
- ModelForge 2mo ago
  And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"
- cma 2mo ago
  The paper says: "Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."
dwohnitmok 2mo ago
As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!). And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.
fspeech 2mo ago
Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.
natrys 2mo ago
At least the SSM layers should have relative PE built in via recurrence, in a hybrid attention model like this.
yhocd 2mo ago
The primary author of RoPE, Jianlin Su, is currently working at Moonshot (KIMI)
augment_me 2mo ago
When you have recurrent blocks in your model, you implicitly have a timestep T(amount of recurrent steps). Similar to Diffusion Transformers, it then becomes valuable to encode the knowledge of where you are in this chain somehow. NoPE is more flexible than RoPE for this.

itkovian_ 2mo ago

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.

ModelForge 2mo ago
And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"
cma 2mo ago
The paper says: "Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."

dwohnitmok 2mo ago

As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!). And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.

fspeech 2mo ago

Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.

natrys 2mo ago

At least the SSM layers should have relative PE built in via recurrence, in a hybrid attention model like this.

yhocd 2mo ago

The primary author of RoPE, Jianlin Su, is currently working at Moonshot (KIMI)

augment_me 2mo ago

When you have recurrent blocks in your model, you implicitly have a timestep T(amount of recurrent steps). Similar to Diffusion Transformers, it then becomes valuable to encode the knowledge of where you are in this chain somehow. NoPE is more flexible than RoPE for this.