I read that as perform not get the right answer. Performing arithmetic is one kind of reasoning process. But getting the answers right is not necessarily the s

I read that as perform not get the right answer. Performing arithmetic is one kind of reasoning process. But getting the answers right is not necessarily the same as performing. If you go on to read, what he's trying to test is the system's ability to even attempt to plan out a problem solving "route". Which it doesn't really do. If it could, it could defer to another system (fancy calculators or solvers) to do the work. But its lack of ability to reason means it can't even be made to do that. (EDIT: I do think the paper would be stronger if he put the math and formal logic etc problems later. E.g. the problem he puts forward in 3.14, 3.15 etc is more immediately damning as it reflects the "kind" of daily life reasoning that people would expect these systems to be able to perform.)