Googler here, but not in the research team that authored this paper. It seems that using a "teacher" LLM to train a smaller model in this step-by-step fashion

Googler here, but not in the research team that authored this paper. It seems that using a "teacher" LLM to train a smaller model in this step-by-step fashion you get much more out of your parameters. Specifically, if you look at section 3 in the paper, they mention that they use LLMs generated rationales as additional guidance for the smaller model. This approach has already been tried, but it had some limitations (section 3.2) which they circumvent by doing things differently: "In this work, instead of using rationales as additional model inputs, we frame learning with rationales as a multi-task problem. ... (this) enables the model to learn to generate the intermediate reasoning steps for the prediction, and could therefore guide the model in better predicting the resultant label." It seems that this is the key for the success they are showing. I am still digesting the paper and I am not 100% sure that is the key factor.