3 ms·
Good question - one followup question there is value for who? If it is to train the LLM that is labeling, then I agree. If it is to train a smaller downstream m
by nihit-desai 3y ago
Good question - one followup question there is value for who?
If it is to train the LLM that is labeling, then I agree.
If it is to train a smaller downstream model (e.g. finetune a pretrained BERT model) then the value is as good as coming from any human annotator and only a function of label quality
- voz_ 3y agoWhy retrain that smaller model from scratch tho? Just do a little transfer learning, or get creative and see if you can prune down to a smaller model algorithmically instead of doing the whole label and train rigamarole from scratch on what is effectively regurgitation. I’m not sold this has directional value.
- nihit-desai 3y agoHmm, I'm not suggesting training a smaller model from scratch - in most cases you'd want to finetune a pretrained model (aka, transfer learning) for your specific usecase/problem domain. The need for labeled data for any kind of training is a constant though :)