GPT can be the generative (imagination) part of a system. The second part would be a system to validate. This means you can filter out the junk and keep the goo

GPT can be the generative (imagination) part of a system. The second part would be a system to validate. This means you can filter out the junk and keep the good samples, extending the training set. This is how the model can learn by performing massive search for solutions all on its own. A bit like AlphaGo. So you might want to reevaluate it. It can sample really well, and it is hard to sample coherently in this combinatorial space. Sampling is half the task, validation needs more work.