In the HANS paper I'd point out this bit: "we retrained each model on the MNLI training set augmented with a dataset structured exactly like HANS (i.e. using t

In the HANS paper I'd point out this bit: "we retrained each model on the MNLI training set augmented with a dataset structured exactly like HANS (i.e. using the same thirty subcases) but containing no specific examples that appeared in HANS.... In general, the models trained on the augmented MNLI performed very well on HANS" I think it's great to find examples where language models break. But it doesn't look like they have found a fundamental problem here - just a weakness that can be fixed with "just" engineering. I do agree there are things we don't know how to do yet. I think Chollet's "On the Measure of Intelligence" paper[1] is the best writing I've seen on this, which shows specific things that are currently hard for computers to do and give a route towards general intelligence. [1] https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547