You can see the strong influence of WikiHow in the training data for that one, probably from slurping WikiHow itself and also the infinite blogspam that inspire

You can see the strong influence of WikiHow in the training data for that one, probably from slurping WikiHow itself and also the infinite blogspam that inspired/borrowed from its style. What could go wrong when you train your models on 99% barely-attended garbage? Sure, they learn to complete sentences and arrange larger blocks of text, but there's sooo much noise in the content that it creates a bias towards blogspam's plausible garbage (which we so often see). There's going to have to be a whole new wave of training-data pruning and from-scratch retraining once some of the other technical goals are achieved because the feedback of LLM blogspam back into LLM training data is just going to amplify all the bad qualities.