That does sounds invasive, but on the other hand, math teachers do tell the kids to “show their work” for good reasons. And the consent issues don’t apply for L

That does sounds invasive, but on the other hand, math teachers do tell the kids to “show their work” for good reasons. And the consent issues don’t apply for LLM training. I wonder if the trend towards using synthetic, AI-generated training data will make it easier to train models that use <aloud> effectively? AI’s could be trained to use reasoning and show their work more than people normally do when posting on the Internet. It’s not going to create information out of nothing, but it will better model the distribution that the researchers want the LLM to have, rather than taking distributions found on the Internet as given. It’s not a natural distribution anyway. For example, I believe it’s already the case that people train AI with weighted distributions - training more on Wikipedia, for example. My guess is that the quest for the best training data has only just begun.