4 ms·
No, it isn't taught anything. GPT3 text generation is effectively a really fancy autocompletion algorithm based on the n-many previous tokens in a rolling windo
by f38zf5vdt 4y ago
No, it isn't taught anything. GPT3 text generation is effectively a really fancy autocompletion algorithm based on the n-many previous tokens in a rolling window. You can only "teach" GPT3 something within that window, and it doesn't "learn" there, it just tries its best to generate content based on what is stored in its massive n-dimension table of graph edges for tokens.
That is also why it has such a strong propensity to lose the plot once you are outside of that window size and it's generating new content based on self-generated content.
- yunyu 4y agoYou can update the "graph edges" with content longer than the window by fine tuning: https://beta.openai.com/docs/guides/fine-tuning https://beta.openai.com/docs/guides/fine-tuning
- f38zf5vdt 4y agoYes, training the model is where it learns, not in prompts. Prompting might be considered meta-learning but it will always need a reference point given to it from its training data, and beyond the prompt the original model is never altered.
- sdenton4 4y agoEh, one could argue that this is similar to the short term/long term memory divide in humans. We tend to suck at new things until we sleep on it and update our weights...