3 ms·

I only took a beginner course in ML over 5 years ago, so this probably is a stupid question, but does this mean, the trained GPT-2 model encodes the source text

I only took a beginner course in ML over 5 years ago, so this probably is a stupid question, but does this mean, the trained GPT-2 model encodes the source text into it's parameters somehow? Is this resilient — will it still remember it if we randomize the weights just a little tiny bit? Will it remember it if we clear a small portion of parameters? Does the human memory work the same way?

jakear 6y ago
If you can recall text verbatim, the text has been encoded into your neurons’ connections. Tautologically.
minimaxir 6y ago
GPT-2 was trained with a Dropout of 0.1, which gives it a bit more resilience to noise and unusual input versus no Dropout.