5 ms·

I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability. When you have hundreds or thousands of examples

I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability. When you have hundreds or thousands of examples, BERT works great. But that is very restricting.

jerrygenser 2y ago
Yes but you can use an llm to label data and then train a bert model which then costs a small fraction of time and money to run than the original llm.
- hdhshdhshdjd 2y ago
  Shhh, don’t tell everybody the secret. ;-)
  - Karrot_Kream 2y ago
    Lol isn't everyone doing it? That's how I bootstraped my BERT fine-tunes.
    - hdhshdhshdjd 2y ago
      I would say everybody smart is doing that, but a lot of the dumb money in AI right now is just wrappers on the GPT API That makes for a flashy demo with no underlying substance or expertise.
- robrenaud 2y ago
  Is the encoder style arch better for representing classification tasks at a given compute budget than a causal LM? Is this because the final represention in bert style models more globally focused, rather than being optimized for next token prediction?
  - jerrygenser 2y ago
    They are 100% better for classification at a given compute budget. They can account for information before and after e.g. a token for token classification and use that information to classify.
byefruit 2y ago
Yes, no zero shot. Few shot is possible for some use cases with setfit: https://github.com/huggingface/setfit https://github.com/huggingface/setfit and the very recent Fastfit: https://github.com/IBM/fastfit https://github.com/IBM/fastfit ( https://arxiv.org/pdf/2404.12365 https://arxiv.org/pdf/2404.12365 )
deepsquirrelnet 2y ago
They are there, you just have to look. Tasksource, NuNER, Flan, T0. There’s not a lot, but still at least a few good zero shot models in both architectures.
visarga 2y ago
It's because you need to mess with embeddings or even train new heads on top of a network to use it. LLMs just use tokens-in tokens-out, they don't classify with softmax over classes, they softmax over vocabulary tokens. LLMs are more convenient

jerrygenser 2y ago

Yes but you can use an llm to label data and then train a bert model which then costs a small fraction of time and money to run than the original llm.

hdhshdhshdjd 2y ago
Shhh, don’t tell everybody the secret. ;-)
- Karrot_Kream 2y ago
  Lol isn't everyone doing it? That's how I bootstraped my BERT fine-tunes.
  - hdhshdhshdjd 2y ago
    I would say everybody smart is doing that, but a lot of the dumb money in AI right now is just wrappers on the GPT API That makes for a flashy demo with no underlying substance or expertise.
robrenaud 2y ago
Is the encoder style arch better for representing classification tasks at a given compute budget than a causal LM? Is this because the final represention in bert style models more globally focused, rather than being optimized for next token prediction?
- jerrygenser 2y ago
  They are 100% better for classification at a given compute budget. They can account for information before and after e.g. a token for token classification and use that information to classify.

byefruit 2y ago

Yes, no zero shot. Few shot is possible for some use cases with setfit: https://github.com/huggingface/setfit https://github.com/huggingface/setfit and the very recent Fastfit: https://github.com/IBM/fastfit https://github.com/IBM/fastfit ( https://arxiv.org/pdf/2404.12365 https://arxiv.org/pdf/2404.12365 )

deepsquirrelnet 2y ago

They are there, you just have to look. Tasksource, NuNER, Flan, T0. There’s not a lot, but still at least a few good zero shot models in both architectures.

visarga 2y ago

It's because you need to mess with embeddings or even train new heads on top of a network to use it. LLMs just use tokens-in tokens-out, they don't classify with softmax over classes, they softmax over vocabulary tokens. LLMs are more convenient