4 ms·

(1) This is not knowledge distillation, which requires knowing the logits and not just the output text. (2) "Training a smaller model off davinci using the l

(1) This is not knowledge distillation, which requires knowing the logits and not just the output text. (2) "Training a smaller model off davinci using the logits" would have been vastly more expensive than what they did, which was to fine-tune another pretrained foundation model, which happened to be smaller, using output of another foundation model, which happened to be larger. The relative size isn't the key idea, it's that they were able to extract the fine-tuning for instruction-following (which is relatively computationally cheap to train, but requires a custom expensive small dataset) and port it from one foundation model to another. This is what has implications for the companies that planned to have this custom fine-tuning as part of their competitive moat. (3) I understand that ad hominem is against the culture that Hacker News tries to cultivate here. Consider reading or rereading the local guidelines on discussion, as well as reading or rereading an article on knowledge distillation, the Stanford publication of Alpaca, and my tweet.

shenberg 4y ago
It's referred to as distillation in the literature in separate sub-fields (e.g. LRspeech, a paper about low-resource speech-to-text & text-to-speech from Microsoft Research, 2020, uses datasets generated by a large model to train a small one and refers to this as knowledge distillation: https://arxiv.org/abs/2008.03687 https://arxiv.org/abs/2008.03687 )
Imnimo 4y ago
Matching logits is merely a special case of knowledge distillation. See section 2.1 https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531
- Eliezer 4y ago
  Literally retraining on inputs and outputs without doing anything clever, does not to me seem to deserve quite such a fancy name. How do we end up with a smaller model if the dataset formats are exactly similar, except in some other special case, like using much more generated data to train the smaller model? Of course there are other deservedly clever things you could do besides matching logits, that would enable a smaller student model. Here, the Stanford authors are not doing anything clever to enable a smaller model. They're just yoinking the fine-tuning onto what happens to be a smaller model. The destination model being smaller is not the point. The cheap yoinking of just the instruction tuning is the point. They used a small model as the destination because that was cheaper.
  - Imnimo 4y ago
    Sure, but the value of logit matching is that it allows the smaller model to access richer representations that it has sufficient capacity to express but insufficient capacity to learn. In this case, we're not trying to distill the full capabilities of the large model. We just want to mimic instruction following, and we don't have any reason to believe the smaller model has insufficient capacity. So doesn't the lesson of knowledge distillation tell us that this should work just by matching low temperature logits (in other words, matching argmax only)? I don't see how this is fundamentally different from the standard knowledge distillation setting - it's just an easier instance of it
ipsum2 4y ago
1) It is a general version of knowledge distillation. For example, this paper from 2016 describes the same technique: Sequence-Level Knowledge Distillation [0] > This sequence-level approximation leads to a simple training procedure wherein the student network is trained on a newly generated dataset that is the result of running beam search with the teacher network 2) Fine-tuning is a step in the training process. Language models are first pre-trained, then fine-tuned. This is a pedantic quibble. 3) It is unsurprising that you don't understand ad hominem. Giving background information and pointing out the style of writing is relevant to arguments made. [0] https://arxiv.org/abs/1606.07947 https://arxiv.org/abs/1606.07947
- xdavidliu 4y ago
  It's arguable that saying "EY has a very shallow understanding of ML" is even lower than ad hominem (which is DH1) on the pg scale [4], since pg specifically gives "The author is a self-important dilettante." as an example of DH0. [4] http://www.paulgraham.com/disagree.html http://www.paulgraham.com/disagree.html
YeGoblynQueenne 4y ago
>> the mask of the shoggoth I threw a null cultural reference exception on that one. Can you please clarify? Edit: I know the "Masks of Nyarlathotep" and what a shoggoth is; I just don't know what is the "mask of the shoggoth". Edit 2: Really, the one thing I wanted to be able to do with a language generator has always been to get it to GM a CoC campaign for me. It's still the case that there's no chance of that. Just as an aside.

shenberg 4y ago

It's referred to as distillation in the literature in separate sub-fields (e.g. LRspeech, a paper about low-resource speech-to-text & text-to-speech from Microsoft Research, 2020, uses datasets generated by a large model to train a small one and refers to this as knowledge distillation: https://arxiv.org/abs/2008.03687 https://arxiv.org/abs/2008.03687 )

Imnimo 4y ago

Matching logits is merely a special case of knowledge distillation. See section 2.1 https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531

Eliezer 4y ago
Literally retraining on inputs and outputs without doing anything clever, does not to me seem to deserve quite such a fancy name. How do we end up with a smaller model if the dataset formats are exactly similar, except in some other special case, like using much more generated data to train the smaller model? Of course there are other deservedly clever things you could do besides matching logits, that would enable a smaller student model. Here, the Stanford authors are not doing anything clever to enable a smaller model. They're just yoinking the fine-tuning onto what happens to be a smaller model. The destination model being smaller is not the point. The cheap yoinking of just the instruction tuning is the point. They used a small model as the destination because that was cheaper.
- Imnimo 4y ago
  Sure, but the value of logit matching is that it allows the smaller model to access richer representations that it has sufficient capacity to express but insufficient capacity to learn. In this case, we're not trying to distill the full capabilities of the large model. We just want to mimic instruction following, and we don't have any reason to believe the smaller model has insufficient capacity. So doesn't the lesson of knowledge distillation tell us that this should work just by matching low temperature logits (in other words, matching argmax only)? I don't see how this is fundamentally different from the standard knowledge distillation setting - it's just an easier instance of it

ipsum2 4y ago

1) It is a general version of knowledge distillation. For example, this paper from 2016 describes the same technique: Sequence-Level Knowledge Distillation [0] > This sequence-level approximation leads to a simple training procedure wherein the student network is trained on a newly generated dataset that is the result of running beam search with the teacher network 2) Fine-tuning is a step in the training process. Language models are first pre-trained, then fine-tuned. This is a pedantic quibble. 3) It is unsurprising that you don't understand ad hominem. Giving background information and pointing out the style of writing is relevant to arguments made. [0] https://arxiv.org/abs/1606.07947 https://arxiv.org/abs/1606.07947

xdavidliu 4y ago
It's arguable that saying "EY has a very shallow understanding of ML" is even lower than ad hominem (which is DH1) on the pg scale [4], since pg specifically gives "The author is a self-important dilettante." as an example of DH0. [4] http://www.paulgraham.com/disagree.html http://www.paulgraham.com/disagree.html

YeGoblynQueenne 4y ago

>> the mask of the shoggoth I threw a null cultural reference exception on that one. Can you please clarify? Edit: I know the "Masks of Nyarlathotep" and what a shoggoth is; I just don't know what is the "mask of the shoggoth". Edit 2: Really, the one thing I wanted to be able to do with a language generator has always been to get it to GM a CoC campaign for me. It's still the case that there's no chance of that. Just as an aside.