4 ms·

>bit under a token/second When you say 'token' is this a word? A character? I've never gotten a good definition for it beyond 'a unit of text the llm process

>bit under a token/second When you say 'token' is this a word? A character? I've never gotten a good definition for it beyond 'a unit of text the llm processes'

jchw 3y ago
More common words can be just one token, but most words will be a few tokens. A token is neither a character nor a word, it's more like a word fragment.
tga_d 3y ago
My amateur intuition, having played around with local llms a little bit and seeing things load a token at a time, is that they're conceptually like if you took all n-grams for all lengths n, then sorted them by frequency in the training data, and truncated that list at some point. So the most common words, or even most common words+punctuation, will be one token, less common words with "normal" spelling will be a few tokens, while unusual words with atypical letter combinations will be many tokens. So, e.g., " the" will probably be one token, but "qzxv" will probably be four, depending on what the training set was (something mostly trained on Wikipedia will have different tokens than something mostly trained on code).
adw 3y ago
It is a unit of text the LLM processes. :-) Everyone uses (byte pair encoding)[https://en.wikipedia.org/wiki/Byte_pair_encoding https://en.wikipedia.org/wiki/Byte_pair_encoding] to generate their tokens; the tokens are whatever emerge from this. They will typically correspond to the most common substrings in the training corpus in, handwaving a bit, a max-cover sense; it's an encoding which attempts to best compress the data the tokenizer was trained on.
pulse7 3y ago
1 Token ~= 3/4 word
kcorbitt 3y ago
A good way to build intuition for how much text fits in a token is by pasting a block of text into a tokenizer playground, like this one: https://huggingface.co/spaces/Xenova/the-tokenizer-playground https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...

>bit under a token/second When you say 'token' is this a word? A character? I've never gotten a good definition for it beyond 'a unit of text the llm processes'

jchw 3y ago

More common words can be just one token, but most words will be a few tokens. A token is neither a character nor a word, it's more like a word fragment.

tga_d 3y ago

My amateur intuition, having played around with local llms a little bit and seeing things load a token at a time, is that they're conceptually like if you took all n-grams for all lengths n, then sorted them by frequency in the training data, and truncated that list at some point. So the most common words, or even most common words+punctuation, will be one token, less common words with "normal" spelling will be a few tokens, while unusual words with atypical letter combinations will be many tokens. So, e.g., " the" will probably be one token, but "qzxv" will probably be four, depending on what the training set was (something mostly trained on Wikipedia will have different tokens than something mostly trained on code).

adw 3y ago

It is a unit of text the LLM processes. :-) Everyone uses (byte pair encoding)[https://en.wikipedia.org/wiki/Byte_pair_encoding https://en.wikipedia.org/wiki/Byte_pair_encoding] to generate their tokens; the tokens are whatever emerge from this. They will typically correspond to the most common substrings in the training corpus in, handwaving a bit, a max-cover sense; it's an encoding which attempts to best compress the data the tokenizer was trained on.

pulse7 3y ago

1 Token ~= 3/4 word

kcorbitt 3y ago

A good way to build intuition for how much text fits in a token is by pasting a block of text into a tokenizer playground, like this one: https://huggingface.co/spaces/Xenova/the-tokenizer-playground https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...