4 ms·
Ask HN: Is it significant that token length of source is close to e?
What's the relationship between Shannon entropy of the distribution, and token length? For example, English is quoted as having a token length of ~4 characters, but source code (that I've tested) seems to be closer to 2.7. Is this significant that it's close to e (i.e., the base of the natural logarithm)? Is source code a more efficient and natural representation of structure/knowledge/information than English? Any thoughts? Any connection with how log appears in thermodynamic entropy?
- uberman 11mo agoMy guess is that you are not naming your variables correctly.
- keepamovin 11mo ago*Based on file size / token count (as reported by Claude).
- gus_massa 11mo agoDoes token include operators like + - * / ? Does the average length include the space?
- keepamovin 11mo agoEr, sir, Idk. Ask anthropic.
- pestatije 11mo agonot too many comments either