5 ms·

I entirely disagree with your interpretation : > "OpenAI sure has the time and money to finetune their model" This model is absolutly gigantic. In term of tra

I entirely disagree with your interpretation : > "OpenAI sure has the time and money to finetune their model" This model is absolutly gigantic. In term of training it's a nightmare. I am pretty sure they don't have the capacity to train 10 of them in parallel. So finetuning on all the downstream task needs to be done basically in serial and takes forever. They might have a lot of money, but time is as important for them than for anyone else, their lab isn't at the edge of black hole. If what you are saying is true I am pretty sure they would have reported it because it would be an extremely interesting and important result in my opinion. If finetuning gave you no advantage over a few shot settings, it would basically mean that the model already know everything there is to know about the task just from it's pretraining and any additional training is useless as the model is not learning anything. Finally, given the pre-training curves with various model sizes, we clearly haven't reached saturation there, there is no indication anywhere that we have reached saturation on downstream task. So, for me the far more likely explanation is that fine-tuning on downstream tasks is indeed very costly (time and/or money) even by their standard and isn't even on topic for this paper.

YeGoblynQueenne 6y ago
An important thing that one learns in academia is that nobody gives anyone the benefit of the doubt. This is a lesson learned the hard way: you make an unsupported claim and a murder of angry reviewers pounce upon it like hyenas hungry for flesh. Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". No. This is not how a piece of research work is evaluated, not even a piece of work in deep learning, a field that has abandoned all pretensions to science in recent years. My personal advice (speaking as someone who has been attacked by the hyenas and paid my pound of flesh) is that one should always demand the highest standard of proof for any claim in a research paper. That, if one really wishes to know what's going on. Intellectual curiosity and scientific wonderment should not result in gullibility.
- Voloskaya 6y ago
  > Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". Do you not see the cognitive dissonance? You were precisely claiming that because it should be easy to do they most likely have done it, and if they didn't report results it's because it failed. You are making this unsupported claim with 0 evidence to back you up.
  - YeGoblynQueenne 6y ago
    I did not say it's easy or hard. I said they can do it because they have the time and money. In any case, it's an obvious thing to do and there's no obvious explanation of why they didn't, given that they could.

YeGoblynQueenne 6y ago

An important thing that one learns in academia is that nobody gives anyone the benefit of the doubt. This is a lesson learned the hard way: you make an unsupported claim and a murder of angry reviewers pounce upon it like hyenas hungry for flesh. Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". No. This is not how a piece of research work is evaluated, not even a piece of work in deep learning, a field that has abandoned all pretensions to science in recent years. My personal advice (speaking as someone who has been attacked by the hyenas and paid my pound of flesh) is that one should always demand the highest standard of proof for any claim in a research paper. That, if one really wishes to know what's going on. Intellectual curiosity and scientific wonderment should not result in gullibility.

Voloskaya 6y ago
> Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". Do you not see the cognitive dissonance? You were precisely claiming that because it should be easy to do they most likely have done it, and if they didn't report results it's because it failed. You are making this unsupported claim with 0 evidence to back you up.
- YeGoblynQueenne 6y ago
  I did not say it's easy or hard. I said they can do it because they have the time and money. In any case, it's an obvious thing to do and there's no obvious explanation of why they didn't, given that they could.