3 ms·
True, but LLM token probability doesn't map nearly as cleanly to "how readable was the text".
by rafram 2y ago
True, but LLM token probability doesn't map nearly as cleanly to "how readable was the text".
- hansvm 2y agoWhy not though? Both kinds of models jumble around the data and spit out a probability distribution. Why is the tesseract distribution inherently more explainable (aside from the UI/UX problem of the uncertainty being per-token instead of per-character)?