4 ms·
That’s the probability to observe a token given the prompt and the seed. The probability assigned to a token given the prompt alone is a separate thing, which i
by canjobear 2y ago
That’s the probability to observe a token given the prompt and the seed. The probability assigned to a token given the prompt alone is a separate thing, which is objectively defined independent of any observer and can be found by reading out the model logits.
- kgwgk 2y agoYes, that’s a purely mathematical abstract concept that exists outside of space and time. The labels “objective” and “subjective” are usually used to talk about probabilities in relation to the physical world.
- deleted 2y ago[deleted]
- canjobear 2y agoAn LLM distribution exists in the physical world, just as much as this comment does. It didn’t exist before the model was trained. It has relation to the physical world: it assigns probabilities to subword units of text. It has commercial value that it wouldn’t have if its objective probability values were different.
- kgwgk 2y ago> It has relation to the physical world: it assigns probabilities to subword units of text. How is that probability assignment linked to the physical world exactly? In the physical world the computer will produce a token. You rejected before that it was about predicting the token that would be produced.
- kgwgk 2y agoOr maybe you mean that the probability assignments are not about the output of a particular LLM implementation in the real world but about subword units of text in the wild. In that case how could two different LLMs do different assigments to the same physical world without being wrong? Would they be “objective” but unrelated to the “object”?