8 ms·
Vision is how humans see text. So text must have built in adaptations to protect from visual noise. For example, two words that look similar must never appear i
by f33d5173 1y ago
Vision is how humans see text. So text must have built in adaptations to protect from visual noise. For example, two words that look similar must never appear in similar contexts, or else they would be conflated. Hence we can safely reduce such words to the same token. Or something like that.
- ffsm8 1y agoIs that really factual/true? Lots of words have multiple meanings and can mean different things even if used in the same sentence/context just from the interpretation of the person reading it. Heck, it'd argue that most (not all) dayjob conflicts are down to such differences in interpretation /miscommunications
- fxtentacle 1y agoThat also works purely on text and it's the trick I used in my German speech recognition engine ( https://arxiv.org/abs/2206.12693 https://arxiv.org/abs/2206.12693 ). "I'm studying at Oxford Univ" has basically no loss in meaning even though "University" was truncated to less than half its characters.
- UltraSane 1y agoThis is like how many CLIs accept the shortest unique version of commands.