2 ms·

Didn’t see any benchmarks or comparisons Is it worth it to do it? There’s been plenty of comments lately indicating that fine-tuning is not that effective and

by nico 3y ago

Didn’t see any benchmarks or comparisons Is it worth it to do it? There’s been plenty of comments lately indicating that fine-tuning is not that effective and that in a lot of cases reduces the generalization capabilities of the model Some say it’s better to just do some type of augmented search/retrieval on top of the model instead Does anyone have any numbers for reference?

panarky 3y ago
PaLM 2 is the underlying model for more than two dozen Google products, such as MedPaLM 2. Each product fine-tunes their own weights on top of the underlying model, but it seems like that fine-tuning is more about style than substance. For example, I listened to a 90-minute interview about the MedPaLM 2 product. It achieved an accuracy of 85% on the MedQA dataset (median score for human medical students is 67.4%) and 72.3% on the MedMCQA dataset (median score for human medical students is 58.4%). But it didn't get its medical knowledge and reasoning capabilities from the fine-tuning, since the underlying PaLM 2 model already had that. The fine-tuning was focused on interaction style, more professional and objective, avoiding informal language or expressing personal opinions. MedPaLM 2 was also fine-tuned to answer questions in a comprehensive, deliberate and informative way. For example, if a doctor asks MedPaLM 2 about the causes of a medical condition, MedPaLM 2 provides a list of possible causes, as well as information about the symptoms, diagnosis, treatment alternatives and side effects. The fine-tuning also helped it to express its degree of certainty or uncertainty. The underlying PaLM 2 already had the knowledge and reasoning capability to perform well on standardized medical tests, and the fine-tuning helped it to express it more effectively.

2 ms·

Didn’t see any benchmarks or comparisons Is it worth it to do it? There’s been plenty of comments lately indicating that fine-tuning is not that effective and

by nico 3y ago

panarky 3y ago
PaLM 2 is the underlying model for more than two dozen Google products, such as MedPaLM 2. Each product fine-tunes their own weights on top of the underlying model, but it seems like that fine-tuning is more about style than substance. For example, I listened to a 90-minute interview about the MedPaLM 2 product. It achieved an accuracy of 85% on the MedQA dataset (median score for human medical students is 67.4%) and 72.3% on the MedMCQA dataset (median score for human medical students is 58.4%). But it didn't get its medical knowledge and reasoning capabilities from the fine-tuning, since the underlying PaLM 2 model already had that. The fine-tuning was focused on interaction style, more professional and objective, avoiding informal language or expressing personal opinions. MedPaLM 2 was also fine-tuned to answer questions in a comprehensive, deliberate and informative way. For example, if a doctor asks MedPaLM 2 about the causes of a medical condition, MedPaLM 2 provides a list of possible causes, as well as information about the symptoms, diagnosis, treatment alternatives and side effects. The fine-tuning also helped it to express its degree of certainty or uncertainty. The underlying PaLM 2 already had the knowledge and reasoning capability to perform well on standardized medical tests, and the fine-tuning helped it to express it more effectively.