4 ms·

Will ChatGPT drag this downward even more?

• kqr 3y ago
  I couldn't resist asking it about the word entropy of its training corpus and quite predictably it wasn't able to understand my question and instead answered similar questions. However, it stuck quite firmly to a figure of 10--12 bits per word! I'm fairly sure that refers to the vocabulary entropy and not word entropy of English.
  • fooker 3y ago
    This is like forcing a human to answer questions about brain anatomy. You’ll be lucky to get anything reasonable at all.
• zw123456 3y ago
  I was wondering the same thing. Even the prediction as I type this in is pretty good. It seems like LLM's could be even better.
• canjobear 3y ago
  Yes, the GPT's are literally trained to minimize (cross-)entropy of text. You can see the bits-per-byte of text for lots of models here: https://crfm.stanford.edu/helm/v0.2.0/?group=targeted_evaluations https://crfm.stanford.edu/helm/v0.2.0/?group=targeted_evalua...