3 ms·
One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token." From the post: > Surprisingly, in our work, w
by jawns 7y ago
One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token."
From the post:
> Surprisingly, in our work, we discover that perplexity, an automatic metric that is readily available to any neural seq2seq model, exhibits a strong correlation with human evaluation, such as the SSA value. Perplexity measures the uncertainty of a language model. The lower the perplexity, the more confident the model is in generating the next token (character, subword, or word).
- aalleavitch 7y agoIs perplexity related to the Shannon entropy? Seems like this might make sense, with the exception of social niceties the point of human communication is to transfer information that was hitherto unknown.
- phreeza 7y agoYea, perplexity is ~exp(entropy)
- bkanber 7y agoActually, the key point is that they aren't using perplexity. They built a chatbot that aims to maximize "SSA" (Sensibleness and Specificity Average), and then found that SSA is correlated with perplexity.
- phreeza 7y ago"The training objective is to minimize perplexity, the uncertainty of predicting the next toke"" SSA is used as an evaluation metric. It is generated by human raters, and as such is not differentiable, making I hard to directly optimize for with current methods.