3 ms·
We're talking about different things. You're talking about finetuning models to tasks known in advance. I'm talking about the ability to generalize to new tasks
by cs702 3y ago
We're talking about different things. You're talking about finetuning models to tasks known in advance. I'm talking about the ability to generalize to new tasks: https://arxiv.org/pdf/2206.07682 https://arxiv.org/pdf/2206.07682. Please don't argue against a straw-man.
- lolinder 3y agoI'm not attacking a straw man, it seems that I just don't understand the distinction you're drawing. As I understand it we're contrasting two opposite approaches to ML: fine tuning small models for specific applications versus training a single large model that can generalize to new tasks without preparing them ahead of time. I'm arguing that in fine tuning is far more useful than people are currently giving it credit for, and that generalizing a single massive model to new tasks is overrated. Can you clarify where you're seeing a straw man?
- cs702 3y agoYou're talking about tasks known in advance; I'm not.
- lolinder 3y agoAnd I'm saying that any task that wasn't known in advance immediately becomes a task known in advance once it's done once. I'm not arguing that there is no place for large, general models—they're great for exploration—just that a smaller foundation model shouldn't be dismissed offhand based solely on parameter count.