4 ms·

Tune PaLM 2 with your own RLHF training data

17 points by panarky 3y ago

nico 3y ago
Didn’t see any benchmarks or comparisons Is it worth it to do it? There’s been plenty of comments lately indicating that fine-tuning is not that effective and that in a lot of cases reduces the generalization capabilities of the model Some say it’s better to just do some type of augmented search/retrieval on top of the model instead Does anyone have any numbers for reference?
- panarky 3y ago
  PaLM 2 is the underlying model for more than two dozen Google products, such as MedPaLM 2. Each product fine-tunes their own weights on top of the underlying model, but it seems like that fine-tuning is more about style than substance. For example, I listened to a 90-minute interview about the MedPaLM 2 product. It achieved an accuracy of 85% on the MedQA dataset (median score for human medical students is 67.4%) and 72.3% on the MedMCQA dataset (median score for human medical students is 58.4%). But it didn't get its medical knowledge and reasoning capabilities from the fine-tuning, since the underlying PaLM 2 model already had that. The fine-tuning was focused on interaction style, more professional and objective, avoiding informal language or expressing personal opinions. MedPaLM 2 was also fine-tuned to answer questions in a comprehensive, deliberate and informative way. For example, if a doctor asks MedPaLM 2 about the causes of a medical condition, MedPaLM 2 provides a list of possible causes, as well as information about the symptoms, diagnosis, treatment alternatives and side effects. The fine-tuning also helped it to express its degree of certainty or uncertainty. The underlying PaLM 2 already had the knowledge and reasoning capability to perform well on standardized medical tests, and the fine-tuning helped it to express it more effectively.
tulip4attoo 3y ago
I don't see the reward model that they are using. I think that is an important factor when doing rlhf.
- aldarisbm 3y ago
  What does this impact? how aligned to the base model does the output model stays?

17 points by panarky 3y ago

nico 3y ago

Didn’t see any benchmarks or comparisons Is it worth it to do it? There’s been plenty of comments lately indicating that fine-tuning is not that effective and that in a lot of cases reduces the generalization capabilities of the model Some say it’s better to just do some type of augmented search/retrieval on top of the model instead Does anyone have any numbers for reference?

panarky 3y ago
PaLM 2 is the underlying model for more than two dozen Google products, such as MedPaLM 2. Each product fine-tunes their own weights on top of the underlying model, but it seems like that fine-tuning is more about style than substance. For example, I listened to a 90-minute interview about the MedPaLM 2 product. It achieved an accuracy of 85% on the MedQA dataset (median score for human medical students is 67.4%) and 72.3% on the MedMCQA dataset (median score for human medical students is 58.4%). But it didn't get its medical knowledge and reasoning capabilities from the fine-tuning, since the underlying PaLM 2 model already had that. The fine-tuning was focused on interaction style, more professional and objective, avoiding informal language or expressing personal opinions. MedPaLM 2 was also fine-tuned to answer questions in a comprehensive, deliberate and informative way. For example, if a doctor asks MedPaLM 2 about the causes of a medical condition, MedPaLM 2 provides a list of possible causes, as well as information about the symptoms, diagnosis, treatment alternatives and side effects. The fine-tuning also helped it to express its degree of certainty or uncertainty. The underlying PaLM 2 already had the knowledge and reasoning capability to perform well on standardized medical tests, and the fine-tuning helped it to express it more effectively.

tulip4attoo 3y ago

I don't see the reward model that they are using. I think that is an important factor when doing rlhf.

aldarisbm 3y ago
What does this impact? how aligned to the base model does the output model stays?