Oh that was a brilliant video clip. I hadn't seen that before, thank you!! > The important question in the OpenAI work that you haven't touched on is how to ev

Oh that was a brilliant video clip. I hadn't seen that before, thank you!! > The important question in the OpenAI work that you haven't touched on is how to evaluate superintelligence. I guess I would frame the problem like this: > Let's say there is a very esoteric but important branch of abstract mathematics that only a few people claim to understand. Is there a way for us to determine which mathematicians are actually intelligent, and which are bluffing? How? This is a tricky one. To my dog, I am revered as a super-being of intelligence and capability. But if he watches me play grandmaster-level chess, or writing a paper on abstract mathematics -- it must look like insanity. In sci-fi, I rather like the image of super-intelligence from one of my favorite short-stories: "When the Yogurt Took Over" [1] > No one argues with the yogurt. No one tweaks its formulas. The rest of the time it rests there in its factory, thinking about whatever intelligent fermented milk thinks about. It just sits there in its vat -- and its actions seem largely incomprehensible to us -- as incomprehensible as me playing Magic: The Gathering is to my dog. It must look like lunacy. (given what I spend on the game, I'm not sure it's not) So if we're going to evaluate superintelligence, then I feel that -- for starters -- it must be on somewhat of a clear playing-field. We can clearly evaluate super-ability in Chess, in Go, and in Starcraft 2 because there are clearly defined rules. The only true test of whether one is superior to another will be because "it works". Until we can test abstract mathematics objectively, then I'm not sure we could ever judge. In so far as questions of particle physics and whatnot could actually be tested -- those feel like the sorts of areas where we might be able to evaluate superintelligence. But SPAG is much smaller than that. The hope that SPAG offers is that -- as long as the game rules leverage things like language and semantics -- then (assuming the model is able to generalize), then the increased mastery of language will transfer to other tasks. And the SPAG results seem to bear that out. [1] https://whatever.scalzi.com/2010/10/02/when-the-yogurt-took-over-a-short-story/ https://whatever.scalzi.com/2010/10/02/when-the-yogurt-took-...