4 ms·
As long as we haven't reached AGI I feel like this is true of any new benchmark. BERT wasn't trained or designed for the task, give some smart folks a few mont
by g82918 7y ago
As long as we haven't reached AGI I feel like this is true of any new benchmark. BERT wasn't trained or designed for the task, give some smart folks a few months and they can now beat the task. The bigger question is what would we like an AI to be able to do. Is this benchmark a good one? Is there maybe a better choice of questions to get the type of NLP we want?
- nl 7y agoIs this benchmark a good one? Any benchmark which reflects a task that humans do is a good one, unless it has specific weaknesses that a computer exploits. I'd use models that are written for this is my work, so I find it useful. I feel like this is true of any new benchmark.... give some smart folks a few months and they can now beat the task. In NLP work this use not to be the case. 5 years ago we were stuck at a local maximum. And this undervalues this task - this bridges the gap between unstructured and structured data. In many ways it is the holy grail for many tasks.