4 ms·

Sam Altman and Ilya Sutskever have made several mentions of the idea that models are better at teaching models than humans are, in many cases. The next GPT iter

• unshavedyak 4y ago
  Is there a high level explanation as to why? Best I can imagine it’s that it’s easier to improve a model than make one from scratch, and so AI training AI means you can start fresh and have the benefits of the previous version train the foundation for the next, and then yet again refine that new version and repeat. … but this is 100% speculation, I have no clue. Thoughts?
  • thomashop 4y ago
    Probably because you can use the error gradients from the generative model somehow
  • visarga 4y ago
    The task of generating images is harder than that of classifying, the training signal is also much denser. So it means the diffusion model implicitly learned a better visual model, and now they found a way to invert that and improve the discriminative task as well. We can do the same thing in text. A recent paper showed GPT-4 is better than most human labellers at tagging NLP datasets. I personally used it to generate samples for smaller models, basically the same thing this paper showed. You can even top that with a second LLM round to reflect on its generated data and filter the noisy examples. In the end it still makes errors. And the errors GPT makes are hard to detect at scale.
    • bagels 4y ago
      Can you give an outline of how you used gpt4 to generate samples?
• ShamelessC 4y ago
  Alan Turing actually talked about this way back in Computing Machinery and Intelligence and it’s already used for distillation of information from large models to more compact ones in the so called “student/teacher” training regime.
• visarga 4y ago
  It's not so simple. Garbage in garbage out. But if you can get a validation signal in there, maybe using an external tool, a game, an outcome of some sort, then maybe it can work. For example generate code, and then run it to see if it passes tests - this is a good filtering signal.
  • cubefox 4y ago
    These guys did that by letting a model generate unit tests and then let the interpreter check it: https://arxiv.org/abs/2207.14502 https://arxiv.org/abs/2207.14502
  • sebzim4500 4y ago
    I don't think you necessarily need any validation signal to improve performance with synthetic data. Here's an example I posted further up: > Suppose you want to optimize the ability for an LLM to do arithmetic. You prompt the LLM to generate a bunch of arithmetic questions, prompting it to show its working. Then you take that output, remove the intermediate steps, and train on the results.
• sharemywin 4y ago
  isn't reinforcement learning from human feedback (RLHF) the process of training a model for a reward signal from humans?