8 ms·

LLMs are significantly slower than traditional ML, typically costlier and, I have been told, tend to be less accurate than a traditional model trained on a larg

LLMs are significantly slower than traditional ML, typically costlier and, I have been told, tend to be less accurate than a traditional model trained on a large dataset. But, they are zero/few shot classifiers. Meaning that you can get your classification running and reasonably accurate now, collect data and switch to a fine-tuned very efficient traditional ML model later.

hellovai 3y ago
That's a great summary and insight. We should likely use that verbiage to help make it more crystal clear :)
famouswaffles 3y ago
Current State of the art (GPT-4) is not going to be less accurate than whatever bespoke option you can cook up. https://news.ycombinator.com/item?id=36685921 https://news.ycombinator.com/item?id=36685921
- withinboredom 3y ago
  I wouldn't be so sure of that.
- golergka 3y ago
  It will be much slower and costlier.
- mplewis 3y ago
  This is absolutely untrue.
  - famouswaffles 3y ago
    Feel free to show otherwise
    - chaxor 3y ago
      I think they mean that it is likely to heavily depend upon the task. Decoder models are good at generation of language, and of course they're going to do well where that counts. But if you want to do typical NER+Relation extraction and then normalize to an in-house dictionary of 10 million IDs? You can't do that as effectively with GPT-4. You need a local model (right now). There are a lot of things that GPT-4 doesn't touch in terms of data domain, so for many projects, yes local (typically encoder) models are 1) way faster 2) way" cheaper and 3) have better metrics. Of course, the capability could* be there with a large decoder model, but if it's a task that needs your specific (large amount of) data, you have to make something locally.
    - mistymountains 3y ago
      No, you need to show the model is better on this narrow task, not just assert it is because it’s a great general LLM. It’s quite possible you’re correct but just saying GPT is the best, prove me wrong, reeks of a VC or AI bandwagoner.
      - famouswaffles 3y ago
        The OP is about classification. I linked to benchmarks showing it is indeed much better on the kind of task outlined in the OP as well as several other NLP tasks. I believe "prove me wrong" is more than appropriate here.
- potatoman22 3y ago
  I'd bet good money that well crafted XGBoost models will outperform GPT-4 on almost any classification problem that uses numerical or tabular data. Which, historically, is what most classification problems use.
godelski 3y ago
> LLMs are significantly slower than traditional ML, typically costlier Literally point 3 in the article. > But, they are zero/few shot classifiers This is __NOT__ true. Zero-shot means out of domain, and if we're talking about text trained LLMs, there really isn't anything text that is out of domain for them because they are trained on almost anything you can find on the internet. This is not akin to training something on Tiny Shakespeare and then having it perform sentiment analysis (classification) on Sci-Fi novels. Similarly, training a model on JFT or LAION does not give you the ability to perform zero shot classification on datasets like COCO or ImageNet, since the same semantic data exists in both datasets. I don't know why people started using this term to describe the domain adaptation or transfer learning, but it is not okay. Zero-shot requires novel classes, and subsets are not novel.
- PartiallyTyped 3y ago
  > Zero-shot means out of domain, and if we're talking about text trained LLMs, there really isn't anything text that is out of domain for them because they are trained on almost anything you can find on the internet. Respectfully, i disagree. I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. > I don't know why people started using this term to describe the domain adaptation or transfer learning, but it is not okay. Zero-shot requires novel classes, and subsets are not novel. Respectfully, i disagree. Zero-shot is perfectly valid because there is no backpropagation or weight change involved. Causal LLMs are meta-learners due to the attention mechanism and the autoregressive nature of the model. These two change the effective weight of the matrices. For all sequences of inputs and all possible weights; there exists an instantiation of a neural network without attention that produces identical vectors for the current token given only the previous token. Do the math, or read the paper “LLMs are meta learners”. Therefore, for all tasks, giving the model examples of inputs changes its effective weights without actually modifying it, it is perfectly valid for “zero shot learning” because you didn’t do backprop of any kind, you merely did input transformations / preprocessing.
  - godelski 3y ago
    > I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. Can you give an example so that we may better discuss or that I can adequately update my understandings? But I will say that simplifying this down to "just trained to predict the next token" is not accurate as it does not account for the differences in architectures and cost functions which dramatically affect this statement due to the differences in their biases. As a clear example, training an image model on likelihood does note guarantee that the model will produce high fidelity samples[0]. But it will be better at imputation or classification. Some other helpful references[1,2] > Zero-shot is perfectly valid because there is no backpropagation or weight change involved. I disagree with this. What you have described is still within the broader class of fine tuning. Note that zero-shot is also tuning. I can make this perfectly clear with a simple example that is directly related to my previous argument. ``Suppose we train a model on the CIFAR-10 dataset. Then we "zero-shot" evaluate it on CIFAR-5, where we've just removed 5 random classes.`` I think you'll agree that it should be unsurprising that the model performs well on this second task. This is exactly the "Train on LAION then 'zero-shot' classification on ImageNet" task we commonly see. Subsets are not a clear task change. > These two change the effective weight of the matrices. I'm having a difficult time understanding your argument as this directly contradicts your first sentence. I wouldn't even make the lack of weight change a requirement for zero-shot learning as the intent is really that we do not need to directly change. If a model has enough general knowledge and we do not need to modify the parameters explicitly through providing more training (i.e. using a cost function and {back,forward}prop), then this is sufficient (randomly changing parameters, adding non-trainable parameters like activations, or pruning is also acceptable. As well as explicitly what you mentioned). The point comes down to requiring no additional training for __additional domain{,s}__. The training part is not the important part here and not what is in question. My point is explicitly about claiming that subdomains do not constitute zero-shot learning. If you disagree in what I have claimed are subdomains, then that's a different argument. I'm not arguing against the latter points because that's also not arguing against what I claimed. But I will say that "just because you didn't use backprop doesn't mean it isn't zero-shot" and if you disagree, then note that you have to claim that the CIFAR-5 example is "zero-shot." Tldr: A -> B doesn't require that B -> A [0]A note on the evaluation of generative models: https://arxiv.org/abs/1511.01844 https://arxiv.org/abs/1511.01844 (link for also obtaining slides and code: http://theis.io/publications/17/ http://theis.io/publications/17/) Also worth looking at many of the works that cite this one: https://www.semanticscholar.org/paper/A-note-on-the-evaluation-of-generative-models-Theis-Oord/39e0c341351f8f4a39ac890b96217c7f4bde5369#citing-papers https://www.semanticscholar.org/paper/A-note-on-the-evaluati... [1a] Assessing Generative Models via Precision and Recall: https://arxiv.org/abs/1806.00035 https://arxiv.org/abs/1806.00035 [1b] Improved Precision and Recall Metric for Assessing Generative Models: https://arxiv.org/abs/1904.06991 https://arxiv.org/abs/1904.06991 [2] The Role of ImageNet Classes in Fréchet Inception Distance: https://arxiv.org/abs/2203.06026 https://arxiv.org/abs/2203.06026
SkyPuncher 3y ago
To me, LLMs feel like "low-code" tools in most applicable domains. They're very, very good at creating a new, novel solution - but specially trained ML models will rule.

hellovai 3y ago

That's a great summary and insight. We should likely use that verbiage to help make it more crystal clear :)

famouswaffles 3y ago

Current State of the art (GPT-4) is not going to be less accurate than whatever bespoke option you can cook up. https://news.ycombinator.com/item?id=36685921 https://news.ycombinator.com/item?id=36685921

withinboredom 3y ago
I wouldn't be so sure of that.
golergka 3y ago
It will be much slower and costlier.
mplewis 3y ago
This is absolutely untrue.
- famouswaffles 3y ago
  Feel free to show otherwise
  - chaxor 3y ago
    I think they mean that it is likely to heavily depend upon the task. Decoder models are good at generation of language, and of course they're going to do well where that counts. But if you want to do typical NER+Relation extraction and then normalize to an in-house dictionary of 10 million IDs? You can't do that as effectively with GPT-4. You need a local model (right now). There are a lot of things that GPT-4 doesn't touch in terms of data domain, so for many projects, yes local (typically encoder) models are 1) way faster 2) way" cheaper and 3) have better metrics. Of course, the capability could* be there with a large decoder model, but if it's a task that needs your specific (large amount of) data, you have to make something locally.
  - mistymountains 3y ago
    No, you need to show the model is better on this narrow task, not just assert it is because it’s a great general LLM. It’s quite possible you’re correct but just saying GPT is the best, prove me wrong, reeks of a VC or AI bandwagoner.
    - famouswaffles 3y ago
      The OP is about classification. I linked to benchmarks showing it is indeed much better on the kind of task outlined in the OP as well as several other NLP tasks. I believe "prove me wrong" is more than appropriate here.
potatoman22 3y ago
I'd bet good money that well crafted XGBoost models will outperform GPT-4 on almost any classification problem that uses numerical or tabular data. Which, historically, is what most classification problems use.

godelski 3y ago

> LLMs are significantly slower than traditional ML, typically costlier Literally point 3 in the article. > But, they are zero/few shot classifiers This is __NOT__ true. Zero-shot means out of domain, and if we're talking about text trained LLMs, there really isn't anything text that is out of domain for them because they are trained on almost anything you can find on the internet. This is not akin to training something on Tiny Shakespeare and then having it perform sentiment analysis (classification) on Sci-Fi novels. Similarly, training a model on JFT or LAION does not give you the ability to perform zero shot classification on datasets like COCO or ImageNet, since the same semantic data exists in both datasets. I don't know why people started using this term to describe the domain adaptation or transfer learning, but it is not okay. Zero-shot requires novel classes, and subsets are not novel.

PartiallyTyped 3y ago
> Zero-shot means out of domain, and if we're talking about text trained LLMs, there really isn't anything text that is out of domain for them because they are trained on almost anything you can find on the internet. Respectfully, i disagree. I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. > I don't know why people started using this term to describe the domain adaptation or transfer learning, but it is not okay. Zero-shot requires novel classes, and subsets are not novel. Respectfully, i disagree. Zero-shot is perfectly valid because there is no backpropagation or weight change involved. Causal LLMs are meta-learners due to the attention mechanism and the autoregressive nature of the model. These two change the effective weight of the matrices. For all sequences of inputs and all possible weights; there exists an instantiation of a neural network without attention that produces identical vectors for the current token given only the previous token. Do the math, or read the paper “LLMs are meta learners”. Therefore, for all tasks, giving the model examples of inputs changes its effective weights without actually modifying it, it is perfectly valid for “zero shot learning” because you didn’t do backprop of any kind, you merely did input transformations / preprocessing.
- godelski 3y ago
  > I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. Can you give an example so that we may better discuss or that I can adequately update my understandings? But I will say that simplifying this down to "just trained to predict the next token" is not accurate as it does not account for the differences in architectures and cost functions which dramatically affect this statement due to the differences in their biases. As a clear example, training an image model on likelihood does note guarantee that the model will produce high fidelity samples[0]. But it will be better at imputation or classification. Some other helpful references[1,2] > Zero-shot is perfectly valid because there is no backpropagation or weight change involved. I disagree with this. What you have described is still within the broader class of fine tuning. Note that zero-shot is also tuning. I can make this perfectly clear with a simple example that is directly related to my previous argument. ``Suppose we train a model on the CIFAR-10 dataset. Then we "zero-shot" evaluate it on CIFAR-5, where we've just removed 5 random classes.`` I think you'll agree that it should be unsurprising that the model performs well on this second task. This is exactly the "Train on LAION then 'zero-shot' classification on ImageNet" task we commonly see. Subsets are not a clear task change. > These two change the effective weight of the matrices. I'm having a difficult time understanding your argument as this directly contradicts your first sentence. I wouldn't even make the lack of weight change a requirement for zero-shot learning as the intent is really that we do not need to directly change. If a model has enough general knowledge and we do not need to modify the parameters explicitly through providing more training (i.e. using a cost function and {back,forward}prop), then this is sufficient (randomly changing parameters, adding non-trainable parameters like activations, or pruning is also acceptable. As well as explicitly what you mentioned). The point comes down to requiring no additional training for __additional domain{,s}__. The training part is not the important part here and not what is in question. My point is explicitly about claiming that subdomains do not constitute zero-shot learning. If you disagree in what I have claimed are subdomains, then that's a different argument. I'm not arguing against the latter points because that's also not arguing against what I claimed. But I will say that "just because you didn't use backprop doesn't mean it isn't zero-shot" and if you disagree, then note that you have to claim that the CIFAR-5 example is "zero-shot." Tldr: A -> B doesn't require that B -> A [0]A note on the evaluation of generative models: https://arxiv.org/abs/1511.01844 https://arxiv.org/abs/1511.01844 (link for also obtaining slides and code: http://theis.io/publications/17/ http://theis.io/publications/17/) Also worth looking at many of the works that cite this one: https://www.semanticscholar.org/paper/A-note-on-the-evaluation-of-generative-models-Theis-Oord/39e0c341351f8f4a39ac890b96217c7f4bde5369#citing-papers https://www.semanticscholar.org/paper/A-note-on-the-evaluati... [1a] Assessing Generative Models via Precision and Recall: https://arxiv.org/abs/1806.00035 https://arxiv.org/abs/1806.00035 [1b] Improved Precision and Recall Metric for Assessing Generative Models: https://arxiv.org/abs/1904.06991 https://arxiv.org/abs/1904.06991 [2] The Role of ImageNet Classes in Fréchet Inception Distance: https://arxiv.org/abs/2203.06026 https://arxiv.org/abs/2203.06026

SkyPuncher 3y ago

To me, LLMs feel like "low-code" tools in most applicable domains. They're very, very good at creating a new, novel solution - but specially trained ML models will rule.