9 ms·
Are any of the recent NLP advancements due to improvements beyond throwing more data and horsepower at “dumb” models? Will GPT-4 be any different? It seems lik
by freyr 6y ago
Are any of the recent NLP advancements due to improvements beyond throwing more data and horsepower at “dumb” models? Will GPT-4 be any different?
It seems like the current approaches will always fall short of our loftier AI aspirations, but we’re reaching a level of mimicry where we can start to ask, “Does it matter for this task?”
- disgruntledphd2 6y agoNo, almost all the progress is driven by bigger GPUs and datasets. To be fair, things like CNN's and BERT were definitely massive improvements, but a lot of modern AI is just throwing compute at problems and seeing what sticks.
- p1esk 6y agoWill GPT-4 be any different? That's the point - it does not need to be different. If it demonstrates similar improvement to what we saw with GPT-1 --> GPT-2 --> GPT-3, then it will be enough to actually start using it. It's like the progression MNIST --> CIFAR-10 --> ImageNet --> the point where object recognition is good enough for real world applications. But in addition to making it bigger, we can also make it better: smarter attention, external data queries, better word encoding, better data quality, more than one data type as input, etc. There's plenty of room for improvement.