How would you measure the intelligence of an AGI? Say you've got HAL in one corner of the room and The Terminator in the other, and you can ask them questions a

How would you measure the intelligence of an AGI? Say you've got HAL in one corner of the room and The Terminator in the other, and you can ask them questions and ask them to perform computations, and they'll obey you. How can you make sense of which one is smarter? A naive attempt would be to ask them to name the biggest (natural) number, but that's no good: HAL says 1000, Terminator says 1001, but of course HAL also knows 1001 is a number so Terminator only wins by dumb luck. We can salvage the idea though by switching natural number for computable ordinal number, and instead of having them name a single number, have them enumerate computable ordinal numbers indefinitely. The set of codes of computable ordinals is Turing non-computable, so neither machine will succeed in enumerating all of them, and by comparing the size of the ordinals enumerated by the two machines, you get an elegant, parsimonious, not-too-contrived notion of which machine is (mathematically) smarter. Read the slides I linked!