Because data sets are very heterogeneous, and a lot of work already goes into curating them to try and get the best model possible. Curation is limited though,

Because data sets are very heterogeneous, and a lot of work already goes into curating them to try and get the best model possible. Curation is limited though, because it reduces the data set size and the model is still equally weighting everything in the data set, just without the worst stuff. By annotating the data set, you avoid reducing data set size while simultaneously teaching the model the difference between good and bad content. Many models already use annotated data, but LLMs haven't leveraged it much thus far because they're still new-ish tech with architectures in flux, and they've been focused on improving general quality by scaling size/data. As LLMs become more commercially important and we run out of high quality training data, annotating existing data will become the cheapest route to improving output quality.