2 ms·
Clean datasets are critical in machine learning. Its kind of a miracle that LLMs work as well as they do now, but every drop of garbage (like SEO garbage) makes
by brucethemoose2 3y ago
Clean datasets are critical in machine learning. Its kind of a miracle that LLMs work as well as they do now, but every drop of garbage (like SEO garbage) makes them worse and less efficient.
- MereInterest 3y agoFor example, early versions of GPT would effectively treat the string “SolidGoldMagikarp” as a random word. It was the username of a prolific poster on the /r/counting subreddit, which consists entirely of posters counting upward. This subreddit was excluded from the training data for being useless, but was still used in making the tokenizer. As a result, the string “SolidGoldMagikarp” was a single token with no training data about it. This was later fixed by updating the tokenizer, but it demonstrates the importance of clean datasets at all stages.