Synthetic data without some kind of external validation is garbage. E.g. you can't just synthetically generate code, something or someone needs to run it and s

Synthetic data without some kind of external validation is garbage. E.g. you can't just synthetically generate code, something or someone needs to run it and see if it performs the functions you actually asked of it. You need to feed the LLM output into some kind of formal verification system, and only then add it back to the synthetic training dataset. Here, for example - dumb recursive training causes model collapse: https://www.nature.com/articles/s41586-024-07566-y https://www.nature.com/articles/s41586-024-07566-y