4 ms·
It goes something like this: input text -> input tokens -> input embeddings -> model -> output embeddings -> output tokens -> output text Tokens aren't nece
by vore 4y ago
It goes something like this:
input text -> input tokens -> input embeddings -> model -> output embeddings -> output tokens -> output text
Tokens aren't necessarily words: they can be fragments of words and you can check out this behavior here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
For instance, "an eagle" is tokenized to [an][ eagle], but "anoxic" is tokenized to [an][oxic], so just looking for the [an] token is not sufficient. Therefore, you would need to map the output text all the way back into the model to figure out what neuron(s) in the model would generate "an" over "a". Since the bulk of GPT is all unsupervised learning, any connections it makes in its neural network is all emergent.
- caf 4y agoFurther to this, as I understand it the "embedding" is mapping the tokens into vectors in a space where tokens semantically similar will be close together in the space.
- nerdponx 4y agoLest we forget that the ability to even construct this sort of "word vector" was somewhat recently considered both fascinating and cutting-edge (word2vec turns 10 this year). We have come a very long way, very quickly.
- hgsgm 4y agoSurely "an eagle" is [ an][ eagle] since the an starts a new word.
- phkahler 4y agoSo is the output compared to the embedding vectors (via dot product) and the strongest one output its token? How is it "clocked" to get successive tokens?
- vore 4y agoGPT is autoregressive, so it uses the previously generated output to produce the next prediction.
- sebzim4500 4y ago> So is the output compared to the embedding vectors (via dot product) and the strongest one output its token? Yeah except it doesn't necessarily take the strongest one, you can set a 'temperature' parameter which determines how often you should pick the second choice, etc. >How is it "clocked" to get successive tokens? You add each new token to the input prompt to generate the next one.
- phkahler 4y ago>> You add each new token to the input prompt to generate the next one. OMFG so it really is a super advanced auto-complete!! Yeah, this is not AGI or anywhere close. That explains how it picks up context, and also how it can lose the plot after a bit due to limited input size.