Those responses sound very similar (albeit a little less extreme) to glitch tokens [1]; these are created by the tokenizer using the full dataset, but the corre

Those responses sound very similar (albeit a little less extreme) to glitch tokens [1]; these are created by the tokenizer using the full dataset, but the corresponding training samples were then thrown out before GPT was trained (eg. some glitch tokens were usernames participating in r/counting). Such words resulted in GPT not learning their meaning and assigning an embedding vector near zero, making the tokens "meaningless". The sole existence of these in a prompt broke the model completely. OpenAI has since fixed that in the hosted APIs. [1] https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...