3 ms·
The wording makes it seem that the small one outperforms the large one, all else equal. But PaLM is tested in the few-shot setting, specifically chain-of-though
by svantana 3y ago
The wording makes it seem that the small one outperforms the large one, all else equal. But PaLM is tested in the few-shot setting, specifically chain-of-thought prompting, which means:
770M model uses 80% of dataset
540B model uses 0.01% of dataset
- verdverm 3y agoPart of what is going on here is that this paradigm is to train a smaller, task-specific model from a large, generalized model. When comparing to other methods for doing this, it outperforms. So one of these smaller, specialized models can outperform PaLM on some tasks, but not all. The other benefit comes at inference time, where you can get much faster and cheaper outputs.