4 ms·
I don't think that's accurate. The logits actually have high dimensionality, and they are intermediate outputs used to sample tokens. The latent representations
by jacob019 1y ago
I don't think that's accurate. The logits actually have high dimensionality, and they are intermediate outputs used to sample tokens. The latent representations contain contextual information and are also high-dimensional, but they serve a different role--they feed into the logits.
- valine 1y agoThe dimensionality I suppose depends on the vocab size and your hidden dimension size, but that’s not really relevant. It’s a single linear projection to go from latents to logits. Reasoning is definitely not happening in the linear projection to logits if that’s what you mean.
- deleted 1y ago[deleted]
- pyinstallwoes 1y agoWhere does it happen ?
- valine 1y agoMy personal theory is that it’s an emergent property of many attention heads working together. If each attention head is a bird, reasoning would be the movement of the flock.