4 ms·
Abstract: Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reacti
by verdverm 4y ago
Abstract:
Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for small models within a multi-task training framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our 770M T5 model outperforms the 540B PaLM model using only 80% of available data on a benchmark task.
- bjornsing 4y agoThat last bit sounds impressive…
- verdverm 4y agoIt does, they are largely using a full sized LLM to guide a smaller one, but it does mean we can pool resources for the large LLM and then build a company smaller LLM for wide distribution.
- dpflan 4y agoYes, these huge models are the expert systems for training specialized systems that will run often and need to run cheaply.
- VHRanger 4y agoPaLM is ridiculously inefficient in the number of parameters. It's about as good on quantitative benchmarks as LLaMa 65B
- nl 4y agoThis is true (and sort of the point of PaLM - it was an experiment on truly huge model sizes). But also Llama 65B is very efficient for its size: it generally outperforms models twice as large on most benchmarks.
- VHRanger 4y agoYes, because it turns out if you have a more reasonable number of parameters, but train for longer, the outcome is a more efficient model
- deleted 4y ago[deleted]