Good article. I think winograd/winogrande are super clever (and also kind of a “fun” idea). My personal take is that blanket understanding is too hard of a tas

Good article. I think winograd/winogrande are super clever (and also kind of a “fun” idea). My personal take is that blanket understanding is too hard of a task to define, so we ought to cheat and talk about types of understanding. In my mind, understanding a thing means not only that you can answer, but also that you can justify your answer. So different kinds of understanding point to different kinds of justifications. In math classes, you’ll be asked not only to state whether a thing is true or false, but also to show that it’s true or false, giving an answer as well as a proof. In a literature class, you don’t just make a point, you also have to support it in natural language. Same with science classes, supporting things with data and logic. The closest we have right now in ML (widely) is statistical measurements on holdout data. It’s 99% accurate on other things, so it’s 99% likely to be right on this too, if this was sampled from something just like the holdout. We also (less widely) have a little cottage industry of post-hoc explanations that try to explain models predictions. I’d love to see models that can do better than post-hoc explanations. I want a model that “understands in terms of predicate logic” that can spit out an answer with a checkable proof. Or “understands in terms of a knowledge graph” or “understands in terms of a set of human anatomy facts” or “understands in terms of natural language arguments” or any number of things that can spit out an answer as well as a justification. Just asking for blanket understanding means we have to define blanket understanding, but there’s a lot of limited understanding that’s still better than what we have today.