4 ms·
I think within the framework of "almost-orthogonal axes" you can still create a vector that has the desired mix of projections onto any combination of these axe
by anonymoushn 1y ago
I think within the framework of "almost-orthogonal axes" you can still create a vector that has the desired mix of projections onto any combination of these axes?
- yorwba 1y agoNo. You can fit an exponential number of almost-orthogonal vectors into the input space, but the number of not-too-similar probability distributions over output tokens is also exponential in the output dimension. This is fine if you only care about a small subset of distributions (e.g. those that only assign significant probability to at most k tokens), but if you pick any random distribution, it's unlikely to be represented well. Fortunately, this doesn't seem to be much of an issue in practice and people even do top-k sampling intentionally.
- anonymoushn 1y agoI see. You're right. I was either badly mistaken or only thinking about small k.