14 ms·

Interesting that they use T5 for the distilled model. I was under the impression that encoder-decoder architectures were on the way of the Dodo, but it seems th

Interesting that they use T5 for the distilled model. I was under the impression that encoder-decoder architectures were on the way of the Dodo, but it seems they may still be relevant after all. Also interesting is that this isn't an inconceivably clever and out of the box idea. It shows there's still a lot of low hanging fruit to explore, and the future of LLMs isn't set in stone yet. Could be that the real deal is a mixture of experts trained in this style. It's exciting that it feels the holy grail is close to being achievable if only the right combination of ideas is tried.

behnamoh 3y ago
> I was under the impression that encoder-decoder architectures were on the way of the Dodo,... Why is that?
- janalsncm 3y ago
  Decoder-only is pretty good on some tasks. Translation for example was a classic encoded-decoder task but it seems decoders can handle it.
  - mike_hearn 3y ago
    If there's no encoder, then what is being decoded?
- joemoon 3y ago
  It's definitely a very similar method but fundamentally different in that the 'Distilling step-by-step' approach is a multi-task model. As I understand it, rather that training the smaller model to produce the CoT/rationale as a part of the (decoded) response, it actually has two output layers. One output is for the label and the other is for the rationale. The other layers are shared, which is how/why the model is able to have an improved "understanding" of which nuances matter in the labeling task.
- euclaise 3y ago
  They actually have a performance edge, but they aren't well suited to chat models because you can't do caching of past states like with decoder-only models
rolisz 3y ago
T5 family is awesome. FastChat-T5 has amazing text generation quality (eg for RAG chatbots) and it can easily run on CPU fast enough to do a live conversation.
isaacfung 3y ago
The paper mentioned was submitted in May. I think encoder-decoder architectures still make a lot of sense for multimodal models. There are still many low hanging fruits. I have probably seen dozens of variations of chain-of-thoughts, tree-of-thoughts, graph-of-thoughts, self-ask, self-critique, self-plan, self-reflect, etc.
- shanusmagnus 3y ago
  Is there a paper, or some other source, that synthesizes this grab-bag of techniques? As an interested outsider, it's hard to keep up with all the action. Would appreciate any pointers that try to bring some of the pieces together.
  - deleted 3y ago
    [deleted]
  - martingoodson 3y ago
    We discussed this at our meetup last night. See here for paper reference and subscribe to get an alert when the recording is uploaded: https://www.meetup.com/london-machine-learning-meetup/events/295928138/ https://www.meetup.com/london-machine-learning-meetup/events...
    - shanusmagnus 3y ago
      Thank you for that. I wonder if you'd be able to just post a link to the paper, or give its title? I didn't have a Meetup account, tried to create one, spent 4 minutes dicking with it, and still can't get it to let me in.
      - martingoodson 3y ago
        Sorry! Here's the paper https://arxiv.org/abs/2307.10169 https://arxiv.org/abs/2307.10169
  - isaacfung 3y ago

behnamoh 3y ago

> I was under the impression that encoder-decoder architectures were on the way of the Dodo,... Why is that?

janalsncm 3y ago
Decoder-only is pretty good on some tasks. Translation for example was a classic encoded-decoder task but it seems decoders can handle it.
- mike_hearn 3y ago
  If there's no encoder, then what is being decoded?
joemoon 3y ago
It's definitely a very similar method but fundamentally different in that the 'Distilling step-by-step' approach is a multi-task model. As I understand it, rather that training the smaller model to produce the CoT/rationale as a part of the (decoded) response, it actually has two output layers. One output is for the label and the other is for the rationale. The other layers are shared, which is how/why the model is able to have an improved "understanding" of which nuances matter in the labeling task.
euclaise 3y ago
They actually have a performance edge, but they aren't well suited to chat models because you can't do caching of past states like with decoder-only models

rolisz 3y ago

T5 family is awesome. FastChat-T5 has amazing text generation quality (eg for RAG chatbots) and it can easily run on CPU fast enough to do a live conversation.

isaacfung 3y ago

The paper mentioned was submitted in May. I think encoder-decoder architectures still make a lot of sense for multimodal models. There are still many low hanging fruits. I have probably seen dozens of variations of chain-of-thoughts, tree-of-thoughts, graph-of-thoughts, self-ask, self-critique, self-plan, self-reflect, etc.

shanusmagnus 3y ago
Is there a paper, or some other source, that synthesizes this grab-bag of techniques? As an interested outsider, it's hard to keep up with all the action. Would appreciate any pointers that try to bring some of the pieces together.
- deleted 3y ago
  [deleted]
- martingoodson 3y ago
  We discussed this at our meetup last night. See here for paper reference and subscribe to get an alert when the recording is uploaded: https://www.meetup.com/london-machine-learning-meetup/events/295928138/ https://www.meetup.com/london-machine-learning-meetup/events...
  - shanusmagnus 3y ago
    Thank you for that. I wonder if you'd be able to just post a link to the paper, or give its title? I didn't have a Meetup account, tried to create one, spent 4 minutes dicking with it, and still can't get it to let me in.
    - martingoodson 3y ago
      Sorry! Here's the paper https://arxiv.org/abs/2307.10169 https://arxiv.org/abs/2307.10169
- isaacfung 3y ago