16 ms·
N00b to this. How are the neurons outputs read to produce text? They talk about tokens as if a token is a word. But if token==word then every word would have a
by phkahler 4y ago
N00b to this. How are the neurons outputs read to produce text? They talk about tokens as if a token is a word. But if token==word then every word would have a specific output and there's nothing to see here. So again, how are neuron outputs converted to letters/text?
- vore 4y agoIt goes something like this: input text -> input tokens -> input embeddings -> model -> output embeddings -> output tokens -> output text Tokens aren't necessarily words: they can be fragments of words and you can check out this behavior here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer For instance, "an eagle" is tokenized to [an][ eagle], but "anoxic" is tokenized to [an][oxic], so just looking for the [an] token is not sufficient. Therefore, you would need to map the output text all the way back into the model to figure out what neuron(s) in the model would generate "an" over "a". Since the bulk of GPT is all unsupervised learning, any connections it makes in its neural network is all emergent.
- caf 4y agoFurther to this, as I understand it the "embedding" is mapping the tokens into vectors in a space where tokens semantically similar will be close together in the space.
- nerdponx 4y agoLest we forget that the ability to even construct this sort of "word vector" was somewhat recently considered both fascinating and cutting-edge (word2vec turns 10 this year). We have come a very long way, very quickly.
- hgsgm 4y agoSurely "an eagle" is [ an][ eagle] since the an starts a new word.
- phkahler 4y agoSo is the output compared to the embedding vectors (via dot product) and the strongest one output its token? How is it "clocked" to get successive tokens?
- vore 4y agoGPT is autoregressive, so it uses the previously generated output to produce the next prediction.
- sebzim4500 4y ago> So is the output compared to the embedding vectors (via dot product) and the strongest one output its token? Yeah except it doesn't necessarily take the strongest one, you can set a 'temperature' parameter which determines how often you should pick the second choice, etc. >How is it "clocked" to get successive tokens? You add each new token to the input prompt to generate the next one.
- phkahler 4y ago>> You add each new token to the input prompt to generate the next one. OMFG so it really is a super advanced auto-complete!! Yeah, this is not AGI or anywhere close. That explains how it picks up context, and also how it can lose the plot after a bit due to limited input size.