6 ms·
This tool is really useful for helping develop a better intuition for how GPT models actually work. Paste in some text and switch to the token IDs view. Note h
by simonw 4y ago
This tool is really useful for helping develop a better intuition for how GPT models actually work.
Paste in some text and switch to the token IDs view. Note how common words (like "the ") have low integer token IDs, while things like emojis are split into several numbers.
An LLM is a function that takes an array of integers and returns a new array of integers. Seeing the tokens like this helped me reinforce that mental model.
- psychphysic 4y agoTakes an array and returns a single integer is more correct and usefully so? What I still can't wrap my head around is that tokens often don't align with word structures. "Antidepressants" I'd imagine tokenizes as "anti" "depress" "ant". But nope. And "antipsychotic" tokenizes differently from it too! I assumed the output is a token i.e. a single integer and that's rarely even a full word?
- shagie 4y agoImagine a circular list (in however you want to construct that) that matches the input size for the model. The prompt is initially loaded at the start of the list and the model is run and produces high activation on a single output. That token output is then fed to the end of the input circular list and also added to the "this is what the model returned." This process of running the model, getting the token output and sending one copy to the input list and one copy to the return string is repeated until the number of tokens generated hits a numeric limit or a token that represents the stop token is encountered.
- bastawhiz 4y ago> "Antidepressants" I'd imagine tokenizes as "anti" "depress" "ant". But nope. And "antipsychotic" tokenizes differently from it too.. Tokens are symbols. You're thinking of them like embedding vectors. Tokens represent the step before a meaning is assigned to the text: it turns some unit of text into what's essentially an identifier. Which is to say, two homonyms would have the same token id, even though they have different meanings. Tokens have no notion of context.
- crakenzak 4y agowhat is the benefit to such splitting of text based on seemingly meaningless lines? isn't there a better way to do it?
- rmellow 4y agoBreaking text into sub-word units... 1. Greatly reduces memory usage. Instead of memorizing every inflection of the word "walk", it memorizes the root (walk) and the modifiers (ing, ed, er, ...). These modifiers can be reused for other words. 2. Allows for word compositions that weren't in the training set. This is great for uncommon or new expressions like "googlification" or "unalive".
- shagie 4y agoThe walk example doesn't quite hold up. If you put: test walk walker walking walked into the tokenizer you will see the following tokens: [test][ walk][ walk][er][ walking][ walked] Only walker is broken up into two different tokens. I added "test" to that because walk at the start doesn't include the leading space and [walk] and [ walk] are different tokens. For even more fun, [walker] is a distinct token if it doesn't include the leading space. test walker floorwalker foowalker becomes: [test][ walk][er][ floor][walker][ fo][ow][alker] How we think of words doesn't cleanly map to tokens. (Late edit) walker floorwalker becomes tokenized as: [walker][ floor][walker] So in that case, they're the same token. It's curious how white space influences the word to token making.
- nonfamous 4y agoThere’s no syntax or structure to the token set. The actual tokens were algorithmically selected based on the training data to (putting things loosely) optimize compression of the training data given a token set size.
- PeterisP 4y agoSure, but what I'm hearing in the parent post is a question about why we don't use linguistically motivated subword units (of similar length/vocabulary size and thus memory usage) e.g. cutting across morpheme boundaries instead of whatever an algorithm like BPE caclulates.
- raldi 4y agoHow does it decide whether to split "antid" into "ant" "id" or "anti" "d"?
- eternalban 4y agoTry this: 'this is a day that this sentence with clarify that day. Is this not a good day?' [5661, 318, 257, 1110, 326, 428, 6827, 351, 18282, 326, 1110, 13, 1148, 428, 407, 257, 922, 1110, 30] Note 'day' is solidly 1110 here. Now start a sentence with day. "day began with laughter" [12393, 2540, 351, 20263, 13] So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well. "Day after day after this day" [12393, 706, 1110, 706, 428, 1110] "Day day Home home home" [12393, 1110, 5995, 1363, 1363] "day day day home home home" [820, 1110, 1110, 1363, 1363, 1363] [corrected/edited: so case-sensitive and position sensitive as well.] btw doesn't the output array contain the prompt as well (because of the transformer architecture? not entirey sure ~iirc)
- fenomas 4y ago> So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well. You're missing that it groups in spaces. The position isn't relevant, but "day" is a different token than " day".
- eternalban 4y agoAh, you're right. [try "datedate". interesting how it partitions that as 'dated' + 'ate'. Compare with "matemate" -> 'mat', 'emate'.] p.s. "ifthiswasagermanword if this was a german word." It's not even spaces. That second sequence ' german' is chopped up as 'ag' 'erman'.
- hn_throwaway_99 4y agoIt's a lot simpler than that. You can see in the tokenizer that the boundary for words includes the preceding space. So, since the first word doesn't have a preceding space, it has a different token.
- sebzim4500 4y agoThere's no position parameter, it's just that " day" and "day" are different tokens.
- smaddox 4y ago> An LLM is a function that takes an array of integers and returns a new array of integers. To refine this a bit more, a LLM is a function that takes an array of integers (or really, a batch of arrays of integers), and returns a probability distribution for each possible integer, with the array shifted left by one place to enable prediction.
- sorokod 4y agoTo refine further: takes an array of integers and draws the rest of the f**king owl.
- turzmo 4y agoI would argue that my hard drive too can take an array of integers and produce an owl.
- wilg 4y agoOh let's see! I'll start: 1
- x-complexity 4y ago1
- VHRanger 4y agoMoreover: The integers are really indeces into the embedding space. So you'd want to think more that the model maintains a giant matrix (one row = one token; one column is an embedding feature). The array of indices gets the relevant embeddings to shove through the rest of the model's forward pass.
- flir 4y agoA probability distribution for each token in the array, not just the last one? I don't understand that, because wouldn't the probabilities later in the sentence be impacted by the tokens chosen earlier in the sentence?