3 ms·
RACE test accuracy of 89.4. The latter appears to be a particularly strong improvement,a jump of +17.4% absolute points over BERT (Devlin et al., 2019), +7.6% o
by nl 7y ago
RACE test accuracy of 89.4. The latter appears to be a particularly strong improvement,a jump of +17.4% absolute points over BERT (Devlin et al., 2019), +7.6% over XLNet (Yang et al.,2019), +6.2% over RoBERTa (Liu et al., 2019), and 5.3% over DCMI+ (Zhang et al., 2019), an ensemble of multiple models specifically designed for reading comprehension tasks. Our single model achieves an accuracy of86.5%, which is still2.4%better than the state-of-the-art ensemble model
This is an amazing result.
RACE has a performance ceiling of 94.5% set by inaccuracies in the data. Mechanical Turk performance is 73.3%.
It's a hard, reading comprehension test where you can't just extract spans from the text and match against answers. Section 2 of https://www.aclweb.org/anthology/D17-1082.pdf https://www.aclweb.org/anthology/D17-1082.pdf has a sample.