Training models on generated data is a thing. But it needs to be validated in order to filter out the crap. This works better in math and code because you can r

Training models on generated data is a thing. But it needs to be validated in order to filter out the crap. This works better in math and code because you can rely on exact answers and tests. For fake news the model needs to team up with human annotators. For generated images and text in general there are a few ML approaches to detect, and if they fail detection maybe they are that good it's ok to let them be.