3 ms·

Uhh. How is that even possible? I thought I had a basic understanding of Neural Networks and inuput-, hidden- and output layers and those things. So how can it

Uhh. How is that even possible? I thought I had a basic understanding of Neural Networks and inuput-, hidden- and output layers and those things. So how can it possibly backreference to it's own previous answers and then follow another prompt based on this? Mind = Blown

armchairhacker 4y ago
Previous answers are stored in an earlier layer, nodes are densely connected so the answers can "drift" to later layers. In the simplified diagram below the network reaches the answer in the second layer at node "X", and reports derivations of it at 2 positions (obviously there are many more nodes and its a bit more complicated, see https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) https://en.wikipedia.org/wiki/Transformer_(machine_learning_... as GPT3 is a transformer neural network) O O. O X.' 'O. O 'O. 'X O 'O. O O 'X O O O O O
cercatrova 4y ago
This video by Computerphile is a great overview of transformers and how we got to this point [0]. Basically the networks we used before, recurrent neural networks, "forgot" prior information so they're not good at long tasks. The transformer architecture however does not forget (or at least as easily). [0] https://www.youtube.com/watch?v=rURRYI66E54 https://www.youtube.com/watch?v=rURRYI66E54
wcoenen 4y ago
The model only predicts the next token. This is appended to the original input (i.e. prompt), then the model predicts the next token again, etc. So those previous answers are given as input.
nodja 4y ago
GPT is an autoregressive model, this means that it's a recurring prediction model that works one "word"[*] at a time until it predicts that the text should end, feeding it's own guesses as input for the next word. It's basically one of those markov chain bots, except with a very advanced statistical model behind it. [*] Technically it's not words, but tokens. GPT tokenizes text to better compress the amount of data being fed, it's basically a big vocabulary list that compresses text into a list of ints, for example the string "hello world" would be converted to the list [31373, 995]. For this case it's an int per word, but less common words will not be compressed this well, with the worst case scenario being a token per letter. I should also note that while the model's forward pass only works with one token at a time, the text generation is more advanced than that, there's multiple methods like beam search and top k-sampling, each with their own settings and tunables, but the gist of it is that during generation it'll try multiple combinations of token sequences and check which one is the most likely. The limitation is memory, transformer networks are notoriously memory hungry, and IIRC GPT-3 grows quadratically with the number of tokens given, usually the limit is around 2048 tokens, or roughly 1000-2000 words.

armchairhacker 4y ago

Previous answers are stored in an earlier layer, nodes are densely connected so the answers can "drift" to later layers. In the simplified diagram below the network reaches the answer in the second layer at node "X", and reports derivations of it at 2 positions (obviously there are many more nodes and its a bit more complicated, see https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) https://en.wikipedia.org/wiki/Transformer_(machine_learning_... as GPT3 is a transformer neural network) O O. O X.' 'O. O 'O. 'X O 'O. O O 'X O O O O O

cercatrova 4y ago

This video by Computerphile is a great overview of transformers and how we got to this point [0]. Basically the networks we used before, recurrent neural networks, "forgot" prior information so they're not good at long tasks. The transformer architecture however does not forget (or at least as easily). [0] https://www.youtube.com/watch?v=rURRYI66E54 https://www.youtube.com/watch?v=rURRYI66E54

wcoenen 4y ago

The model only predicts the next token. This is appended to the original input (i.e. prompt), then the model predicts the next token again, etc. So those previous answers are given as input.

nodja 4y ago

GPT is an autoregressive model, this means that it's a recurring prediction model that works one "word"[*] at a time until it predicts that the text should end, feeding it's own guesses as input for the next word. It's basically one of those markov chain bots, except with a very advanced statistical model behind it. [*] Technically it's not words, but tokens. GPT tokenizes text to better compress the amount of data being fed, it's basically a big vocabulary list that compresses text into a list of ints, for example the string "hello world" would be converted to the list [31373, 995]. For this case it's an int per word, but less common words will not be compressed this well, with the worst case scenario being a token per letter. I should also note that while the model's forward pass only works with one token at a time, the text generation is more advanced than that, there's multiple methods like beam search and top k-sampling, each with their own settings and tunables, but the gist of it is that during generation it'll try multiple combinations of token sequences and check which one is the most likely. The limitation is memory, transformer networks are notoriously memory hungry, and IIRC GPT-3 grows quadratically with the number of tokens given, usually the limit is around 2048 tokens, or roughly 1000-2000 words.