4 ms·

> So basically, GPT-3 is big, but it's not that good in tasks for which there are formal benchmarks It is very good in all those tasks. In the paper all the nu

> So basically, GPT-3 is big, but it's not that good in tasks for which there are formal benchmarks It is very good in all those tasks. In the paper all the numbers comparing GPT-3 with other models on those benchmark are comparing GPT-3 in a zero/one/few shot setting (so no gradient step), against the previous state of the art finetuned for hours or days on the specific task with millions of gradients steps. If you had time and money to finetue GPT-3 on the specific task there is every reason to believe the gap would be huge. This is the big thing about GPT-3, the promise of not needing to finetune anymore. This is huge in term of productivity but also because it allows you to use the model in settings for which there are basically no dataset available to finetune on.

YeGoblynQueenne 6y ago
>> If you had time and money to finetue GPT-3 on the specific task there is every reason to believe the gap would be huge. On the contrary, there is every reason to assume that GPT-3 cannot significantly improve its results on those tasks with extensive fine tuning. Because, if GPT-3 could significantly improve its performance on those tasks with extensive fine-tuning, the OpenAI paper would be reporting those results (edit: OpenAI sure has the time and money to finetune their model). As we all know by now, there is strong bias in never reporting negative results in machine learning as in other research fields, so we can be reasonably certain that if there is an obvious experiment to perform and that experiment is missing from a paper, it was attemtped and the results were poor.
- popcorncolonel 6y ago
  Not necessarily. There have been a number of papers trying to emphasize the generalization power of a network rather than "it got a SOTA number", and so leave off fine tuning results. Because that would distract from the point of the paper.
- Voloskaya 6y ago
  I entirely disagree with your interpretation : > "OpenAI sure has the time and money to finetune their model" This model is absolutly gigantic. In term of training it's a nightmare. I am pretty sure they don't have the capacity to train 10 of them in parallel. So finetuning on all the downstream task needs to be done basically in serial and takes forever. They might have a lot of money, but time is as important for them than for anyone else, their lab isn't at the edge of black hole. If what you are saying is true I am pretty sure they would have reported it because it would be an extremely interesting and important result in my opinion. If finetuning gave you no advantage over a few shot settings, it would basically mean that the model already know everything there is to know about the task just from it's pretraining and any additional training is useless as the model is not learning anything. Finally, given the pre-training curves with various model sizes, we clearly haven't reached saturation there, there is no indication anywhere that we have reached saturation on downstream task. So, for me the far more likely explanation is that fine-tuning on downstream tasks is indeed very costly (time and/or money) even by their standard and isn't even on topic for this paper.
  - YeGoblynQueenne 6y ago
    An important thing that one learns in academia is that nobody gives anyone the benefit of the doubt. This is a lesson learned the hard way: you make an unsupported claim and a murder of angry reviewers pounce upon it like hyenas hungry for flesh. Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". No. This is not how a piece of research work is evaluated, not even a piece of work in deep learning, a field that has abandoned all pretensions to science in recent years. My personal advice (speaking as someone who has been attacked by the hyenas and paid my pound of flesh) is that one should always demand the highest standard of proof for any claim in a research paper. That, if one really wishes to know what's going on. Intellectual curiosity and scientific wonderment should not result in gullibility.
    - Voloskaya 6y ago
      > Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". Do you not see the cognitive dissonance? You were precisely claiming that because it should be easy to do they most likely have done it, and if they didn't report results it's because it failed. You are making this unsupported claim with 0 evidence to back you up.
      - YeGoblynQueenne 6y ago
        I did not say it's easy or hard. I said they can do it because they have the time and money. In any case, it's an obvious thing to do and there's no obvious explanation of why they didn't, given that they could.

YeGoblynQueenne 6y ago

>> If you had time and money to finetue GPT-3 on the specific task there is every reason to believe the gap would be huge. On the contrary, there is every reason to assume that GPT-3 cannot significantly improve its results on those tasks with extensive fine tuning. Because, if GPT-3 could significantly improve its performance on those tasks with extensive fine-tuning, the OpenAI paper would be reporting those results (edit: OpenAI sure has the time and money to finetune their model). As we all know by now, there is strong bias in never reporting negative results in machine learning as in other research fields, so we can be reasonably certain that if there is an obvious experiment to perform and that experiment is missing from a paper, it was attemtped and the results were poor.

popcorncolonel 6y ago
Not necessarily. There have been a number of papers trying to emphasize the generalization power of a network rather than "it got a SOTA number", and so leave off fine tuning results. Because that would distract from the point of the paper.
Voloskaya 6y ago
I entirely disagree with your interpretation : > "OpenAI sure has the time and money to finetune their model" This model is absolutly gigantic. In term of training it's a nightmare. I am pretty sure they don't have the capacity to train 10 of them in parallel. So finetuning on all the downstream task needs to be done basically in serial and takes forever. They might have a lot of money, but time is as important for them than for anyone else, their lab isn't at the edge of black hole. If what you are saying is true I am pretty sure they would have reported it because it would be an extremely interesting and important result in my opinion. If finetuning gave you no advantage over a few shot settings, it would basically mean that the model already know everything there is to know about the task just from it's pretraining and any additional training is useless as the model is not learning anything. Finally, given the pre-training curves with various model sizes, we clearly haven't reached saturation there, there is no indication anywhere that we have reached saturation on downstream task. So, for me the far more likely explanation is that fine-tuning on downstream tasks is indeed very costly (time and/or money) even by their standard and isn't even on topic for this paper.
- YeGoblynQueenne 6y ago
  An important thing that one learns in academia is that nobody gives anyone the benefit of the doubt. This is a lesson learned the hard way: you make an unsupported claim and a murder of angry reviewers pounce upon it like hyenas hungry for flesh. Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". No. This is not how a piece of research work is evaluated, not even a piece of work in deep learning, a field that has abandoned all pretensions to science in recent years. My personal advice (speaking as someone who has been attacked by the hyenas and paid my pound of flesh) is that one should always demand the highest standard of proof for any claim in a research paper. That, if one really wishes to know what's going on. Intellectual curiosity and scientific wonderment should not result in gullibility.
  - Voloskaya 6y ago
    > Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". Do you not see the cognitive dissonance? You were precisely claiming that because it should be easy to do they most likely have done it, and if they didn't report results it's because it failed. You are making this unsupported claim with 0 evidence to back you up.
    - YeGoblynQueenne 6y ago
      I did not say it's easy or hard. I said they can do it because they have the time and money. In any case, it's an obvious thing to do and there's no obvious explanation of why they didn't, given that they could.