7 ms·
PaLM is ridiculously inefficient in the number of parameters. It's about as good on quantitative benchmarks as LLaMa 65B
by VHRanger 3y ago
PaLM is ridiculously inefficient in the number of parameters. It's about as good on quantitative benchmarks as LLaMa 65B
- nl 3y agoThis is true (and sort of the point of PaLM - it was an experiment on truly huge model sizes). But also Llama 65B is very efficient for its size: it generally outperforms models twice as large on most benchmarks.
- VHRanger 3y agoYes, because it turns out if you have a more reasonable number of parameters, but train for longer, the outcome is a more efficient model