4 ms·

Recent Advances in Language Model Fine-Tuning (2021)

3 points by eternalban 4y ago

PaulHoule 4y ago
I am not so sure if fine-tuning is always a slam dunk over just training a classifier on the embedding, particularly when you consider training time and reliability (the two stage classifier never forgets catastrophically for one thing.)
- eternalban 4y ago
  I can see OpenAI et al offering 2-stage solutions to enterprise customers, a custom built model. My impression is fine tuning opens up the (end product) architectural possibilities. Is that wrong?
  - PaulHoule 4y ago
    In the case of seq-to-seq (mark proper names red) I think fine tuning is necessary but for many classification problems it might not be. I look it from the viewpoint of the end user who might not really have enough training data for fine tuning to work, might not want to fight with GPU drivers to train a network, etc. Personally I feel like using an API like the GPT-3 API that could be cut off at any time is like putting my hand in a toilet but a lot of people would feel that way over doing intensive training. Big picture I see a lot of people who express strong opinions which might not really be justified by the evidence. My favorite kind of paper in arXiv these days are run-of-the-mill papers where somebody solves a new text classification problem with a variety of techniques. I'd like to know what kind of results average people get on average problems more rather than see the world's experts compete to get 0.89 accuracy as opposed to 0.88 on some Kaggle dataset that's been done to death. (I used to ask "Are you sure any of this would work on a dataset that isn't the MNIST digits?") (For that matter it is a pet peeve of mine that deep learning papers frequently quote a number of accuracy, F1 or ROC numbers and set the best one in bold but rarely do any real cross-validation to know if the 0.89 is significantly better than the 0.88) I'd like to see some meta-analysis in this area in fact.

4 ms·

Recent Advances in Language Model Fine-Tuning (2021)

3 points by eternalban 4y ago

PaulHoule 4y ago
I am not so sure if fine-tuning is always a slam dunk over just training a classifier on the embedding, particularly when you consider training time and reliability (the two stage classifier never forgets catastrophically for one thing.)
- eternalban 4y ago
  I can see OpenAI et al offering 2-stage solutions to enterprise customers, a custom built model. My impression is fine tuning opens up the (end product) architectural possibilities. Is that wrong?
  - PaulHoule 4y ago
    In the case of seq-to-seq (mark proper names red) I think fine tuning is necessary but for many classification problems it might not be. I look it from the viewpoint of the end user who might not really have enough training data for fine tuning to work, might not want to fight with GPU drivers to train a network, etc. Personally I feel like using an API like the GPT-3 API that could be cut off at any time is like putting my hand in a toilet but a lot of people would feel that way over doing intensive training. Big picture I see a lot of people who express strong opinions which might not really be justified by the evidence. My favorite kind of paper in arXiv these days are run-of-the-mill papers where somebody solves a new text classification problem with a variety of techniques. I'd like to know what kind of results average people get on average problems more rather than see the world's experts compete to get 0.89 accuracy as opposed to 0.88 on some Kaggle dataset that's been done to death. (I used to ask "Are you sure any of this would work on a dataset that isn't the MNIST digits?") (For that matter it is a pet peeve of mine that deep learning papers frequently quote a number of accuracy, F1 or ROC numbers and set the best one in bold but rarely do any real cross-validation to know if the 0.89 is significantly better than the 0.88) I'd like to see some meta-analysis in this area in fact.