3 ms·
Would the natural analogy of “tokens” be “words”, or something more like, “portion of mouth-movement”?
by drdeca 4y ago
Would the natural analogy of “tokens” be “words”, or something more like, “portion of mouth-movement”?
- b33j0r 3y agoI more imagine someone speaking another language, when I break down written phonemes for tokenization purposes. As a native English speaker who grew up around code-switching Spanish speakers, I often heard speech that sounded really fast. But I was hearing “hay-un-a-al-pa-ca-per-si-gui-en-do-un tren-de-car-ga” I heard that as individual tokens, while my friends simply heard “¡There is an alpaca chasing that freight train!”