4 ms·
I have a simple question. Suppose that to answer a question I can use different phrases, I know the answer but I have several ways to express it. Then a LLM in
by sillying 2y ago
I have a simple question. Suppose that to answer a question I can use different phrases, I know the answer but I have several ways to express it. Then a LLM in this case produces tokens with high or low entropy?
Edited several times: I think to avoid this problem the answer of the LLM should be constrained in expression (say Yes or No, fill the blanks, etc). I think in that case we would have a decreasing sequence of the entropy for next token predictions.
- trq_ 2y agoIn this case it would be a low entropy, high varentropy situation. It's confident in a few possible answers, like if it's a set of synonyms.