4 ms·

Do these scale-dependent (I like this adjective better than "emergent") properties survive model distillation? It may be that our training/optimization processe

Do these scale-dependent (I like this adjective better than "emergent") properties survive model distillation? It may be that our training/optimization processes are inefficient and require these scales to achieve, but the underlying model may not actually require the number of parameters that we are giving them. I haven't read any of the papers about distillation yet, does anyone know if this has been tested?

visarga 4y ago
Good question, my guess is that you can't distill chain-of-thought or zero shot prompting in small models, they got to be 15-20B parameters or larger. Maybe someone has a link to a related paper?
- lossolo 4y ago
  For all models smaller than 62B, direct prompting outperforms CoT. The first model where CoT outperforms direct prompting is Flan-cont-PaLM 62B on BBH. For 540B models, there are more settings where CoT outperforms direct prompting, but not all of them. Also, the number can be smaller than 540B. In Suzgun et. al. 2022, the authors show that the 175B InstructGPT and 175B Codex also have better CoT performance than direct prompting. Combining all the results, we get the two numbers 62B and 175B. So yes indeed, to enter the game of scale you do need a ticket to larger models than average. However, there are also other large models like OPT, BLOOM, and the first version of GPT-3. They all have 175B, yet their CoT performance is significantly worse, or even cannot do CoT. source: https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Mo...
  - visarga 4y ago
    Found a paper myself: > Teaching Small Language Models to Reason https://arxiv.org/abs/2212.08410 https://arxiv.org/abs/2212.08410
    - CGamesPlay 4y ago
      Wow this paper is from 4 days ago, but it's exactly what I was asking about!

visarga 4y ago

Good question, my guess is that you can't distill chain-of-thought or zero shot prompting in small models, they got to be 15-20B parameters or larger. Maybe someone has a link to a related paper?

lossolo 4y ago
For all models smaller than 62B, direct prompting outperforms CoT. The first model where CoT outperforms direct prompting is Flan-cont-PaLM 62B on BBH. For 540B models, there are more settings where CoT outperforms direct prompting, but not all of them. Also, the number can be smaller than 540B. In Suzgun et. al. 2022, the authors show that the 175B InstructGPT and 175B Codex also have better CoT performance than direct prompting. Combining all the results, we get the two numbers 62B and 175B. So yes indeed, to enter the game of scale you do need a ticket to larger models than average. However, there are also other large models like OPT, BLOOM, and the first version of GPT-3. They all have 175B, yet their CoT performance is significantly worse, or even cannot do CoT. source: https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Mo...
- visarga 4y ago
  Found a paper myself: > Teaching Small Language Models to Reason https://arxiv.org/abs/2212.08410 https://arxiv.org/abs/2212.08410
  - CGamesPlay 4y ago
    Wow this paper is from 4 days ago, but it's exactly what I was asking about!