I've been busy with a number of projects and haven't had time to look into this but have been dying to know; has anyone recreated the architecture that OpenAI u

I've been busy with a number of projects and haven't had time to look into this but have been dying to know; has anyone recreated the architecture that OpenAI uses for text-davinci-003, InstructGPT, and ChatGPT that simply doesn't have training data? This is a reproducibility problem of its own sort. I mean, the papers are there out in the open if I understand correctly, but I don't know if anyone's actually built their own transformer architecture 1:1 against what OpenAI claims they're doing in the open. I've seen maybe one or two models that supposedly do something similar on HuggingFace, but I'm itching to find the time to build my own. If someone out there has already built it, I'd be fascinated to know what it looks like to train this architecture on a completely limited naive subset of knowledge that ChatGPT itself claims to be trained on: > As an AI language model, I have been trained on a large corpus of text data from various sources, including but not limited to: > 1. Wikipedia > 2. Books from Project Gutenberg > 3. Web pages from Common Crawl > 4. News articles from various sources, including CNN, Reuters, and BBC > 5. Academic papers from arXiv > 6. Reddit posts and comments > 7. Movie scripts > 8. Song lyrics > 9. Transcripts of speeches and interviews > 10. User-generated content from various forums and social media platforms. > This list is not exhaustive, and my training data is constantly updated and expanded to ensure that I can provide the most accurate and up-to-date information possible. Like, can you imagine how a ChatGPT-like model would respond if only trained on particular discussions from subset communities online? I think there's an interesting opportunity to basically collect communal knowledge from specific isolate communities and understand what a statistically probable output might be from particular groups of people. It may turn particular soft science studies into hard science questions. But you'd only know presumably if you had a working architecture with a near empty dataset. This would also be tremendously useful for building automated chat AI for products that doesn't need to know the entirety of Clint Eastwood's career or the specific details of the features of a Boeing 747.