3 ms·
The title seems rather speculative given that they didn't use any textbooks as training data, or even anything especially similar to textbooks. They used mostl
by mkl 3y ago
The title seems rather speculative given that they didn't use any textbooks as training data, or even anything especially similar to textbooks. They used mostly StackOverflow data and source code from The Stack dataset, and also explanations, examples, and exercises generated by GPT3.5.
- gench 3y agoExactly.. Considering the model that generated their exercises was also trained on Stack Overflow data, one may say, not all but what you definitely need for LLM training is Stack Overflow data. People keep getting it granted but I dont think it will continue that way for a long time.