It’s an interesting paper, but my worry is that this is cherry-picking a surprising negative result. (The paper does not seem to discuss how many evals were con

It’s an interesting paper, but my worry is that this is cherry-picking a surprising negative result. (The paper does not seem to discuss how many evals were considered to find this case.) The general expectation should be that some AI failure modes will be for things that seem obvious to humans. (Because they have a different architecture to humans and different strengths.) The important question is whether this is a single weird cognitive bug on the family size formulation, or representative of a broad family of cognitive errors? It’s difficult to say from this paper. It’s well known that the framing of a logic problem can dramatically influence its difficulty for humans, even when the underlying comparisons are isomorphic. I think this line of research is important, but we need a larger battery of evals (and training on the evals will always be a confounder). The broader family of reasoning here is relational problems (as noted in the paper) and so as a follow-up it would be interesting to explore reasoning performance across a set of isomorphic problems. My hunch is this will generalize somewhat (as LLMs are still not great at relational reasoning), but that there is something about the family formulation in particular that is confusing for LLMs.