That's basically what companies like Mistral do. Many open source models are trained on OpenAI API request output. That's how a couple guys in Europe are able t

That's basically what companies like Mistral do. Many open source models are trained on OpenAI API request output. That's how a couple guys in Europe are able to build something nearly as good as GPT4 almost overnight and license it Apache 2.0. If you want the training data to be both human and open source, then there aren't many good options besides things like https://en.wikipedia.org/wiki/The_Pile_%28dataset%29 https://en.wikipedia.org/wiki/The_Pile_%28dataset%29 which has Hacker News, Wikipedia, the Enron Emails, GitHub, arXiv, etc.