3 ms·

<rant>We need an AI-generated encyclopedia - not for us, but for AI. It should have a trillion articles covering all known entities and concepts, written using

<rant>We need an AI-generated encyclopedia - not for us, but for AI. It should have a trillion articles covering all known entities and concepts, written using RAG over the web. Controversial topics should report the controversy or the distribution of opinions. We can put this big synthetic text corpus in the training set of future models. Why? Because AI needs long form, in-depth texts to train on, and the web doesn't provide it in sufficient quantity and quality. We need chain-of-thought to capture relations between concepts in explicit language. Synthetic data makes it possible to have balanced coverage of topics and combinatorial coverage of skills to improve reasoning. It's also better from a copyright stand point to train models on synthetic data.</>

deleted 3y ago
[deleted]
jfk13 3y ago
> the web doesn't provide it in sufficient quantity and quality Do you seriously think that "an AI-generated encyclopedia" would provide a better-quality training set? What would the "AI generator's" articles be derived from?
- Philpax 3y ago
  The idea is that you can standardise the quality of the training data by taking source articles and synthesizing new data with the same "voice" and structure, as well as being able to collate insights from multiple sources. This is the line of thinking behind the Phi lineup of models [0], as well as efforts to generate synthetic textbooks for training [1]. [0]: https://arxiv.org/abs/2309.05463 https://arxiv.org/abs/2309.05463 [1]: https://twitter.com/ocolegro/status/1712327588255809667 https://twitter.com/ocolegro/status/1712327588255809667
- visarga 3y ago
  So the way I see it, in the first stage the model can take all concepts in Wikipedia and other knowledge bases, and do web search, collect a bunch of references, study and compile a report. That's straight forward search + summarization. The advantage would be that models get to bring together information sitting in separate examples and synthesize or draw conclusions. The second stage would be to generate research questions, then solve them with LLM+web search+code execution+other tools. The results would be compiled in reports. So it's a loop of problem generation, problem solving and validation. You can validate with highly trusted sources, or you can run code or simulations, ensemble multiple attempts, or even leave it to ranking by a preference model.
- thatxliner 3y ago
  What about model collapse?

deleted 3y ago

[deleted]

jfk13 3y ago

> the web doesn't provide it in sufficient quantity and quality Do you seriously think that "an AI-generated encyclopedia" would provide a better-quality training set? What would the "AI generator's" articles be derived from?

Philpax 3y ago
The idea is that you can standardise the quality of the training data by taking source articles and synthesizing new data with the same "voice" and structure, as well as being able to collate insights from multiple sources. This is the line of thinking behind the Phi lineup of models [0], as well as efforts to generate synthetic textbooks for training [1]. [0]: https://arxiv.org/abs/2309.05463 https://arxiv.org/abs/2309.05463 [1]: https://twitter.com/ocolegro/status/1712327588255809667 https://twitter.com/ocolegro/status/1712327588255809667
visarga 3y ago
So the way I see it, in the first stage the model can take all concepts in Wikipedia and other knowledge bases, and do web search, collect a bunch of references, study and compile a report. That's straight forward search + summarization. The advantage would be that models get to bring together information sitting in separate examples and synthesize or draw conclusions. The second stage would be to generate research questions, then solve them with LLM+web search+code execution+other tools. The results would be compiled in reports. So it's a loop of problem generation, problem solving and validation. You can validate with highly trusted sources, or you can run code or simulations, ensemble multiple attempts, or even leave it to ranking by a preference model.
thatxliner 3y ago
What about model collapse?