Yes this seems like a major downside especially considering this will be used for larger complex outputs and the user will essentially need to verify correctnes

Yes this seems like a major downside especially considering this will be used for larger complex outputs and the user will essentially need to verify correctness via a black box approach. This will lead to distrust in even bothering with complex GPT problem solving.