2 ms·
Something that made me a bit sad about LLMs like ChatGPT, is that they're primarily trained on free/open data sets such as Reddit, Wikipedia, and ArXiV. Instea
by jiggawatts 4y ago
Something that made me a bit sad about LLMs like ChatGPT, is that they're primarily trained on free/open data sets such as Reddit, Wikipedia, and ArXiV.
Instead, we could be training LLMs on published scientific papers, textbooks, and high-quality reference materials, but we can't, because not only are they behind paywalls, they're behind thousands of individual paywalls.
- TrapLord_Rhodo 4y agoOpenAI spent millions of dollars on building the data fed into the models. No one has any idea what it's trained on. Also, with the advent of AutoGPT/llamaIndex, you can train it yourself on a massive stockpile of pirated data if you wanted.