3 ms·

It'd be ironic if your comment was generated by GPT-3. But forget GPT-3. In 10 years, looking back at AI history, the year 2020 will probably be viewed as the p

by p1esk 6y ago

It'd be ironic if your comment was generated by GPT-3. But forget GPT-3. In 10 years, looking back at AI history, the year 2020 will probably be viewed as the point separating pre GPT-4 and post GPT-4 epochs. GPT-4 is the model I expect to make things interesting again, not just in NLP, but in AI.

freyr 6y ago
Are any of the recent NLP advancements due to improvements beyond throwing more data and horsepower at “dumb” models? Will GPT-4 be any different? It seems like the current approaches will always fall short of our loftier AI aspirations, but we’re reaching a level of mimicry where we can start to ask, “Does it matter for this task?”
- disgruntledphd2 6y ago
  No, almost all the progress is driven by bigger GPUs and datasets. To be fair, things like CNN's and BERT were definitely massive improvements, but a lot of modern AI is just throwing compute at problems and seeing what sticks.
- p1esk 6y ago
  Will GPT-4 be any different? That's the point - it does not need to be different. If it demonstrates similar improvement to what we saw with GPT-1 --> GPT-2 --> GPT-3, then it will be enough to actually start using it. It's like the progression MNIST --> CIFAR-10 --> ImageNet --> the point where object recognition is good enough for real world applications. But in addition to making it bigger, we can also make it better: smarter attention, external data queries, better word encoding, better data quality, more than one data type as input, etc. There's plenty of room for improvement.
The_rationalist 6y ago
It's not because something seems smart that it actually is. GPT - 3 is ridiculous in that it doesn't have semantic understanding. Text generation is the wrong task, it's cool to watch and actually fuel hype on the ML train but what you're really looking for is semantic parsing, which GPT or openAI has nothing to do with and is mostly underfunded.

by p1esk 6y ago

freyr 6y ago

Are any of the recent NLP advancements due to improvements beyond throwing more data and horsepower at “dumb” models? Will GPT-4 be any different? It seems like the current approaches will always fall short of our loftier AI aspirations, but we’re reaching a level of mimicry where we can start to ask, “Does it matter for this task?”

disgruntledphd2 6y ago
No, almost all the progress is driven by bigger GPUs and datasets. To be fair, things like CNN's and BERT were definitely massive improvements, but a lot of modern AI is just throwing compute at problems and seeing what sticks.
p1esk 6y ago
Will GPT-4 be any different? That's the point - it does not need to be different. If it demonstrates similar improvement to what we saw with GPT-1 --> GPT-2 --> GPT-3, then it will be enough to actually start using it. It's like the progression MNIST --> CIFAR-10 --> ImageNet --> the point where object recognition is good enough for real world applications. But in addition to making it bigger, we can also make it better: smarter attention, external data queries, better word encoding, better data quality, more than one data type as input, etc. There's plenty of room for improvement.

The_rationalist 6y ago

It's not because something seems smart that it actually is. GPT - 3 is ridiculous in that it doesn't have semantic understanding. Text generation is the wrong task, it's cool to watch and actually fuel hype on the ML train but what you're really looking for is semantic parsing, which GPT or openAI has nothing to do with and is mostly underfunded.