3 ms·
Well, that's a nice tnetennba. Their results are actually more impressive than what's reported because the human baseline is in the 80% ish, so they're getting
by YeGoblynQueenne 10y ago
Well, that's a nice tnetennba. Their results are actually more impressive than what's reported because the human baseline is in the 80% ish, so they're getting close (but only on this test so don't expect Her just yet).
On the flip side, the complete test includes different categories of word: principal noun ("named entity"), noun, verb and preposition. This team -like others- only report their results on the two noun categories because LSTM RNNs already get human-like results (that's in the paper). Facebook's original paper noted that nouns and principal nouns are hard for statistical methods. I say probably because there's not enough of them to go around in text, Facebook says it's because of local-only context.
The point is that this is a state of the art result only on named entities and nouns. If you want to process verbs and preposition, good old LSTM are your friend. If you want both, it's going to hurt.
Also, Facebook used Memory Networks and reported they do much better on nouns than LSTM. This paper uses... er, it's complicated. There's GRU and bidrectional RNNs, and Convnets, besides that it's a bit fuzzy. In any case it looks like the complex architectures have the edge now.
But, I don't want to see too much into this because this is not some competition, like ImageNet, the teams all have the entire data and there's nothing protecting them from overfitting to the test set.
- akirajimbo 10y agoAFAIK some split of the children's book test dataset are actually not that well-formed. For example the Verb split, about 1/4 of the correct answers are 'said', which means you can get ~25% accuracy by just choosing 'said'. Then it may make less sense to make too much effort on those ones?