For example, early versions of GPT would effectively treat the string “SolidGoldMagikarp” as a random word. It was the username of a prolific poster on the /r/

For example, early versions of GPT would effectively treat the string “SolidGoldMagikarp” as a random word. It was the username of a prolific poster on the /r/counting subreddit, which consists entirely of posters counting upward. This subreddit was excluded from the training data for being useless, but was still used in making the tokenizer. As a result, the string “SolidGoldMagikarp” was a single token with no training data about it. This was later fixed by updating the tokenizer, but it demonstrates the importance of clean datasets at all stages.