2 ms·
i'm not in the AI world but i've been curious how the level of effort compares between inventing and writing the model vs finding, tagging, and curating the tra
by chasd00 3y ago
i'm not in the AI world but i've been curious how the level of effort compares between inventing and writing the model vs finding, tagging, and curating the training. Is creating the training data analogous to inventing a complete schooling curriculum from scratch? I bet that takes a long freaking time.
- MacsHeadroom 3y agoMuch of GPT-4's training data was made by GPT-3.5. Info on the synthetic data in GPT-4's pretraining is starting to leak, including a reference to "Synthetic-Data(2)" in a recent OpenAI paper. In the near future, if not already, practically all training data for state of the art models will be synthetic. In the words of OpenAI CEO Sam Altman, they have "boostrapped" and are "past the synthetic data generation event horizon."