4 ms·

This is just as silly as claiming that people "moved the goalposts" when a computer beat Kasparov at chess to claim that it wasn't AGI: it wasn't a good test an

This is just as silly as claiming that people "moved the goalposts" when a computer beat Kasparov at chess to claim that it wasn't AGI: it wasn't a good test and some people only realize this after the computer beat Kasparov but couldn't do much else. In this case the ARC maintainers specifically have stated that this is a necessary but not sufficient test of AGI (I personally think it is neither).

famouswaffles 2y ago
It's not silly. The computer that could beat Kasparov couldn't do anything else so of course it wasn't Artificial General Intelligence. o3 can do much much more. There is nothing narrow about SOTA LLMs. They are already General. It doesn't matter what ARC Maintainers have said. There is no common definition of General that LLMs fail to meet. It's not a binary thing. By the time a single machine covers every little test humanity can devise, what comes out of that is not 'AGI' as the words themselves mean but a General Super Intelligence.
- foobarqux 2y ago
  It is silly, the logic is the same: "Only a (world-altering) 'AGI' could do [test]" -> test is passed -> no (world-altering) 'AGI' -> conclude that [test] is not a sufficient test for (world-altering) 'AGI' -> chase new benchmark. If you want to play games about how to define AGI go ahead. People have been claiming for years that we've already reached AGI and with every improvement they have to bizarrely claim anew that now we've really achieved AGI. But after a few months people realize it still doesn't do what you would expect of an AGI and so you chase some new benchmark ("just one more eval"). The fact is that there really hasn't been the type of world-altering impact that people generally associate with AGI and no reason to expect one.
  - famouswaffles 2y ago
    >It is silly, the logic is the same: "Only a (world-altering) 'AGI' could do [test]" -> test is passed -> no (world-altering) 'AGI' -> conclude that [test] is not a sufficient test for (world-altering) 'AGI' -> chase new benchmark. Basically nobody today thinks beating a single benchmark and nothing else will make you a General Intelligence. As you've already pointed out out, even the maintainers of ARC-AGI do not think this. >If you want to play games about how to define AGI go ahead. I'm not playing any games. ENIAC cannot do 99% of the things people use computers to do today and yet barely anybody will tell you it wasn't the first general purpose computer. On the contrary, it is people who seem to think "General" is a moniker for everything under the sun (and then some) that are playing games with definitions. >People have been claiming for years that we've already reached AGI and with every improvement they have to bizarrely claim anew that now we've really achieved AGI. Who are these people ? Do you have any examples at all. Genuine question >But after a few months people realize it still doesn't do what you would expect of an AGI and so you chase some new benchmark ("just one more eval"). What do you expect from 'AGI'? Everybody seems to have different expectations, much of it rooted in science fiction and not even reality, so this is a moot point. What exactly is World Altering to you ? Genuinely, do you even have anything other than a "I'll know it when i see it ?" If you introduce technology most people adopt, is that world altering or are you waiting for Skynet ?
    - foobarqux 2y ago
      > Basically nobody today thinks beating a single benchmark and nothing else will make you a General Intelligence. People's comments, including in this very thread, seem to suggest otherwise (c.f. comments about "goal post moving"). Are you saying that a widespread belief wasn't that a chess playing computer would require AGI? Or that Go was at some point the new test for AGI? Or the Turing test? > I'm not playing any games... "General" is a moniker for everything under the sun that are playing games with definitions. People have a colloquial understanding of AGI whose consequence is a significant change to daily life, not the tortured technical definition that you are using. Again your definition isn't something anyone cares about (except maybe in the legal contract between OpenAI and Microsoft). > Who are these people ? Do you have any examples at all. Genuine question How about you? I get the impression that you think AGI was achieved some time ago. It's a bit difficult to simultaneously argue both that we achieved AGI in GPT-N and also that GPT-(N+X) is now the real breakthrough AGI while claiming that your definition of AGI is useful. > What do you expect from 'AGI'? I think everyone's definition of AGI includes, as a component, significant changes to the world, which probably would be something like rapid GDP growth or unemployment (though you could have either of those without AGI). The fact that you have to argue about what the word "general" technically means is proof that we don't have AGI in a sense that anyone cares about.
      - famouswaffles 2y ago
        >People's comments, including in this very thread, seem to suggest otherwise (c.f. comments about "goal post moving"). But you don't see this kind of discussion on the narrow models/techniques that made strides on this benchmark, do you ? >People have a colloquial understanding of AGI whose consequence is a significant change to daily life, not the tortured technical definition that you are using And ChatGPT has represented a significant change to the daily lives of many. It's the fastest adopted software product in history. In just 2 years, it's one of the top ten most visited sites on the planet worldwide. A lot of people have had the work they do significant change since its release. This is why I ask, what is world altering ? >How about you? I get the impression that you think AGI was achieved some time ago. Sure >It's a bit difficult to simultaneously argue both that we achieved AGI in GPT-N and also that GPT-(N+X) is now the real breakthrough AGI I have never claimed GPT-N+X is the "new breakthrough AGI". As far as I'm concerned, we hit AGI sometime ago and are making strides in competence and/or enabling even more capabilities. You can recognize ENIAC as a general purpose computer and also recognize the breakthroughs in computing since then. They're not mutually exclusive. And personally, I'm more impressed with o3's Frontier Math score than ARC. >I think everyone's definition of AGI includes, as a component, significant changes to the world Sure >which probably would be something like rapid GDP growth or unemployment What people imagine as "significant change" is definitely not in any broad agreement. Even in science fiction, the existence of general intelligences more competent than today's LLMs does not necessarily precursor massive unemployment or GDP growth. And for a lot of people, the clincher stopping them from calling a machine AGI is not even any of these things. For some, that it is "sentient" or "cannot lie" is far more important than any spike of unemployment.

famouswaffles 2y ago

It's not silly. The computer that could beat Kasparov couldn't do anything else so of course it wasn't Artificial General Intelligence. o3 can do much much more. There is nothing narrow about SOTA LLMs. They are already General. It doesn't matter what ARC Maintainers have said. There is no common definition of General that LLMs fail to meet. It's not a binary thing. By the time a single machine covers every little test humanity can devise, what comes out of that is not 'AGI' as the words themselves mean but a General Super Intelligence.

foobarqux 2y ago
It is silly, the logic is the same: "Only a (world-altering) 'AGI' could do [test]" -> test is passed -> no (world-altering) 'AGI' -> conclude that [test] is not a sufficient test for (world-altering) 'AGI' -> chase new benchmark. If you want to play games about how to define AGI go ahead. People have been claiming for years that we've already reached AGI and with every improvement they have to bizarrely claim anew that now we've really achieved AGI. But after a few months people realize it still doesn't do what you would expect of an AGI and so you chase some new benchmark ("just one more eval"). The fact is that there really hasn't been the type of world-altering impact that people generally associate with AGI and no reason to expect one.
- famouswaffles 2y ago
  >It is silly, the logic is the same: "Only a (world-altering) 'AGI' could do [test]" -> test is passed -> no (world-altering) 'AGI' -> conclude that [test] is not a sufficient test for (world-altering) 'AGI' -> chase new benchmark. Basically nobody today thinks beating a single benchmark and nothing else will make you a General Intelligence. As you've already pointed out out, even the maintainers of ARC-AGI do not think this. >If you want to play games about how to define AGI go ahead. I'm not playing any games. ENIAC cannot do 99% of the things people use computers to do today and yet barely anybody will tell you it wasn't the first general purpose computer. On the contrary, it is people who seem to think "General" is a moniker for everything under the sun (and then some) that are playing games with definitions. >People have been claiming for years that we've already reached AGI and with every improvement they have to bizarrely claim anew that now we've really achieved AGI. Who are these people ? Do you have any examples at all. Genuine question >But after a few months people realize it still doesn't do what you would expect of an AGI and so you chase some new benchmark ("just one more eval"). What do you expect from 'AGI'? Everybody seems to have different expectations, much of it rooted in science fiction and not even reality, so this is a moot point. What exactly is World Altering to you ? Genuinely, do you even have anything other than a "I'll know it when i see it ?" If you introduce technology most people adopt, is that world altering or are you waiting for Skynet ?
  - foobarqux 2y ago
    > Basically nobody today thinks beating a single benchmark and nothing else will make you a General Intelligence. People's comments, including in this very thread, seem to suggest otherwise (c.f. comments about "goal post moving"). Are you saying that a widespread belief wasn't that a chess playing computer would require AGI? Or that Go was at some point the new test for AGI? Or the Turing test? > I'm not playing any games... "General" is a moniker for everything under the sun that are playing games with definitions. People have a colloquial understanding of AGI whose consequence is a significant change to daily life, not the tortured technical definition that you are using. Again your definition isn't something anyone cares about (except maybe in the legal contract between OpenAI and Microsoft). > Who are these people ? Do you have any examples at all. Genuine question How about you? I get the impression that you think AGI was achieved some time ago. It's a bit difficult to simultaneously argue both that we achieved AGI in GPT-N and also that GPT-(N+X) is now the real breakthrough AGI while claiming that your definition of AGI is useful. > What do you expect from 'AGI'? I think everyone's definition of AGI includes, as a component, significant changes to the world, which probably would be something like rapid GDP growth or unemployment (though you could have either of those without AGI). The fact that you have to argue about what the word "general" technically means is proof that we don't have AGI in a sense that anyone cares about.
    - famouswaffles 2y ago
      >People's comments, including in this very thread, seem to suggest otherwise (c.f. comments about "goal post moving"). But you don't see this kind of discussion on the narrow models/techniques that made strides on this benchmark, do you ? >People have a colloquial understanding of AGI whose consequence is a significant change to daily life, not the tortured technical definition that you are using And ChatGPT has represented a significant change to the daily lives of many. It's the fastest adopted software product in history. In just 2 years, it's one of the top ten most visited sites on the planet worldwide. A lot of people have had the work they do significant change since its release. This is why I ask, what is world altering ? >How about you? I get the impression that you think AGI was achieved some time ago. Sure >It's a bit difficult to simultaneously argue both that we achieved AGI in GPT-N and also that GPT-(N+X) is now the real breakthrough AGI I have never claimed GPT-N+X is the "new breakthrough AGI". As far as I'm concerned, we hit AGI sometime ago and are making strides in competence and/or enabling even more capabilities. You can recognize ENIAC as a general purpose computer and also recognize the breakthroughs in computing since then. They're not mutually exclusive. And personally, I'm more impressed with o3's Frontier Math score than ARC. >I think everyone's definition of AGI includes, as a component, significant changes to the world Sure >which probably would be something like rapid GDP growth or unemployment What people imagine as "significant change" is definitely not in any broad agreement. Even in science fiction, the existence of general intelligences more competent than today's LLMs does not necessarily precursor massive unemployment or GDP growth. And for a lot of people, the clincher stopping them from calling a machine AGI is not even any of these things. For some, that it is "sentient" or "cannot lie" is far more important than any spike of unemployment.