12 ms·

Except OpenAI hasn’t yet finished discovery on its true goal: AGI. I wonder if they risk plateauing at a local maximum.

by fprog 2y ago

Zambyte 2y ago
I'm genuinely curious: what do you expect an "AGI" system to be able to do that we can't do with today's technology?
- jagrsw 2y ago
  Some first ideas coming to mind: Engineering Level: Solve CO2 Levels End sickness/death Enhance cognition by integrating with willing minds. Safe and efficient interplanetary travel. Harness vastly higher levels of energy (solar, nuclear) for global benefit. Science: Uncover deeper insights into the laws of nature. Explore fundamental mysteries like the simulation hypothesis, Riemann hypothesis, multiverse theory, and the existence of white holes. Effective SETI Misc: End of violent conflicts Fair yet liberal resource allocation (if still needed), "from scarcity to abundance"
  - Zambyte 2y ago
    Do you believe the average human has general intelligence, and do you believe the average human can intellectually achieve these things in ways existing technology cannot?
    - jagrsw 2y ago
      Yes, considering that AI operates differently from human minds, there are several advantages: AI does not experience fatigue or distractions => consistent performance. AI can scale its processing power significantly, despite the challenges associated with it (I understand the challenges) AI can ingest and process new information at an extraordinary speed. AIs can rewrite themselves AIs can be multiplicated (solving scarcity of intelligence in manufacturing) Once achieving AGI, progress could compound rapidly, for better or worse, due to the above points.
      - Jensson 2y ago
        The first AGI will probably take way too much compute to have a significant effect, unless there is a revolution in architecture that gets us fast and cheap AGI at once the AGI revolution will be very slow and gradual. A model that is as good as an average human but costs $10 000 per effective manhour to run is not very useful, but it is still an AGI.
        jagrsw 2y ago
        > A model that is as good as an average human but costs $10 000 per effective manhour to run is not very useful, but it is still an AGI. Geohot (https://geohot.github.io/blog/ https://geohot.github.io/blog/) estimates that a human brain equivalent requires 20 PFLOPS. Current top-of-the-line GPUs are around 2 PFLOPS and consume up to 500W. Scaling that linearly results in 5kW, which translates to approximately 3 EUR per hour if I calculate correctly.
        Jensson 2y ago
        That is if the first model we make is as efficient as a human.
  - jprete 2y ago
    The incentives aren't structured properly for these things to happen, it has always been a sci-fi fairy tale that AGI would achieve these things.
  - TimPC 2y ago
    The problem with CO2 levels is that no one likes the solution not that we don't have one. I highly doubt adding AGI to the mix is going to magically make things better. If anything we'll just burn more CO2 providing all the compute resources it needs. People want their suburban lifestyle with their red meat and their pick-up truck or SUV. They drive fuel inefficient vehicles long-distances to urban work environments and they seem to have very limited interest in changing that. People who like detached homes aren't suddenly affording the rare instances of that closer to their work. We burn lots of oil because we drive fuel inefficient vehicles long distances. This is a problem of changing human preferences which you just aren't going to solve with an AGI.
    - jagrsw 2y ago
      Assuming embedded AI in every piece of robotics - sometimes directly, sometimes connected to a central server (this is doable even today) - it'll revolutionize industries: human-less mining, processing, manufacturing, services, and transportation. These factories would eventually produce and install enough solar power or build sufficient nuclear plants and energy infrastructure, making energy clean and free. With abundant electric cars (at this future point in time) and clean electricity powering heating, transportation, and manufacturing, some AIs could be repurposed for CO2 capture. It sounds deceptively easy, but from an engineering standpoint, it likely holds up. With free energy and AGI handling labor and thinking, we can achieve what a civilization could do and more (cause no individual incentives come into play). However, human factors could be a problem: protests (luddites), wireheading, misuse of AI, and AI-induced catastrophes (alignment).
      - breuleux 2y ago
        Having more energy is intrinsically dangerous, though, because it's indiscriminate: more energy cannot enable bigger solutions without also enabling bigger problems. Energy is the limiting factor to how much damage we can do. If we have way more of it, all bets are off. For instance, the current issue may be that we are indirectly cooking the planet through CO2 emissions, so capturing that sounds like a good idea. But even with clean energy, there is a point where we would cook the planet directly via waste heat of AI and gizmos and factories and whatever unforeseen crap we'll conjure just because we can. And given our track record I'm far from confident that we wouldn't do precisely that.
  - davidgerard 2y ago
    now do magical flying unicorn ponies, which I understand there is also considerable demand for
- hskalin 2y ago
  self consciousness?
  - Zambyte 2y ago
    And how do you test for that?
    - datameta 2y ago
      Probably the best thing we can do at the moment is compile a list of ways in which we shouldn't test for intelligence.
- tsimionescu 2y ago
  The simplest answer, without adding any extraordinary capabilities to the AGI that veer into magical intelligence, is to have AI assistants that can seemlessly interact with technology the way a human assistant would. So, if you want to meet with someone, instead of opening you calendar app and looking for an opening, you'd ask your AGI assistant to talk to their AGI assistant and set up a 1h meeting soon. Or, instead of going on Google to find plane tickets, you'd ask you AGI assistant to find the most reasonable tickets for a certain date range. This would not require any special intelligence more advanced than a human's, but it does require a very general understanding of the human world that is miles beyond what LLMs can achieve today. Going only slightly further with assumptions about how smart an AGI would be, it could revolutionize education, at any level, by acting as a true personalized tutor for a single student, or even for a small group of students. The single biggest problem in education is that it's impossible to scale the highest quality education - and an AGI with capabilities similar to a college professor would entirely solve that.
  - Zambyte 2y ago
    This is definitely an interesting way to look at it. My initial reaction is to consider that I can enhance the capabilities of a system without increasing its inteligence. For example, if I give a monkey a hammer, it can do more than it could do when it didn't have the hammer, but it is not more intelligent (though it could probably learn things by interacting with the world with the hammer). That leads me to think: can we enhance the capabilities of what we call "AI systems" to do these things, without increasing their intelligence? It seems like you can glue GPT-4o to some calendar APIs to do exactly this. This seems more like an issue of tooling rather than an issue of intelligence to me. I guess the issue here is: can a system be "generally intelligent" if it doesn't have access to general tools to act on that intelligence? I think so, but I also can see how the line is very fuzzy between an AI system and the tools it can leverage, as really they both do information processing of some sort. Thanks for the insight.
    - tsimionescu 2y ago
      I'm sure some aspects of this can be achieved by manually programming GPT-4 links to other specific services. And obviously, some interaction tools would have to be written manually even for an AGI. The difference though is the amount of work. Today if you wanted GPT-4 to work as I describe, you would have to write an integration for Gmail, another one for Office365, another one for Proton etc. You would probably have to create a management interface to give access to your auth tokens for each of these to OpenAI so they can activate these interactions. The person you want to sync with would have to do the same. In contrast, an AGI that only has average human intelligence, or even below, would just need access to, say, Firefox APIs, and should easily be able to achieve all of this. And it would work regardless if the other side is a different AGI using a different provider, or even if they are just a regular human assistant.
- Jensson 2y ago
  An AGI could replace human experts at tasks that doesn't require physical embodiment, like diagnosing patients, drafting contracts, doing your taxes etc. If you still do those manually and not just offload all of it to ChatGPT then you would greatly benefit from a real AGI that could do those tasks on their own. And no, using ChatGPT like you use a search engine isn't ChatGPT solving your problem, that is you solving your problem. ChatGPT solving your problem would mean it drives you, not you driving it like it works today. When I hired people to help me do taxes they told me what papers they needed and then they did my taxes correctly without me having to look it through and correct them, an AGI would work like that for most tasks, it means you no longer need to think or learn to solve problems since the AGI solves them for you.
  - Zambyte 2y ago
    Let's take a step back from LLMs. Could you accept the network of all interconnected computers as a generally intelligent system? The key part here that drives me to ask this is: > ChatGPT solving your problem would mean it drives you, not you driving it like it works today. I had a very bad Reddit addiction in the past. It took me years of consciously trying to quit in order to break the habit. I think I could make a reasonable argument that Reddit was using me to solve its problems, rather than myself using it to solve mine. I think this is also true of a lot of systems - Facebook, TikTok, YouTube, etc. It's hard to pin down all computers as an "agent" in the way we like to think about that word and assign some degree of intelligence to, but I think it is at least an interesting exercise to try.
    - Jensson 2y ago
      Companies are general intelligences and they use people, yes. But that depends on humans interpreting that data reddit users generates and updating their models, code and algorithms to adapt to that data, the computer systems alone aren't general intelligences if you remove the humans. An AGI could run such a company without humans anywhere in the loop, just like humans can run such a company without an AGI helping them. I'd say a strong signal that AGI has happened are large fully automated companies without a single human decisionmaker in the company, no CEO etc. Until that has happened I'd say AGI isn't here, if that happens it could be AGI but I can also imagine a good enough script to do it for some simple thing.
- Symmetry 2y ago
  A working memory that can preserve information indefinitely outside a particular context window and which can engage in multi-step reasoning that doesn't show up in its outputs. GPT4o's context window is 128k tokens which is somewhere on the order of 128kB. Your brain's context window, all the subliminal activations from the nerves in your gut and the parts of your visual field you aren't necessarily paying attention to is on the order of 2MB. So a similar order of magnitude though GPT has a sliding window and your brain has more of an exponential decay in activations. That LLMs can accomplish everything they do just with what seems analogous to human reflex rather than human reasoning is astounding and more than a bit scary.
  - datameta 2y ago
    I'm curious what resources led you to calculate a 2MB context window, I'd like to learn more.
    - Symmetry 2y ago
      Looking up an estimate of the brain's input bandwidth at 10 million bits per second and multiplying by the second or two a subliminal stimuli can continue to affect a person's behavior. This is a very crude estimate and probably an order of magnitude off, but I don't think many orders of magnitude off.
- amirhirsch 2y ago
  Prove the Riemann Hypothesis
  - Zambyte 2y ago
    Can you do that, are you not generally intelligent, or is that a bad metric for general intelligence? At least one of these is true.
insane_dreamer 2y ago
That's probably a good thing
anthonypasq 2y ago
how do you know that is actually currently their true goal. it appears to me that the goal has shifted, and the people that cared about the original goal are leaving.
- shwaj 2y ago
  I’m pretty sure Altman still cares about full AGI, if only because of the power inherent in achieving that goal.
JohnFen 2y ago
I don't think that their goal is achievable (at least not within any of our lifetimes), so I think that "plateauing" is inevitable.

Except OpenAI hasn’t yet finished discovery on its true goal: AGI. I wonder if they risk plateauing at a local maximum.

by fprog 2y ago

Except OpenAI hasn’t yet finished discovery on its true goal: AGI. I wonder if they risk plateauing at a local maximum.

Zambyte 2y ago

I'm genuinely curious: what do you expect an "AGI" system to be able to do that we can't do with today's technology?

jagrsw 2y ago
Some first ideas coming to mind: Engineering Level: Solve CO2 Levels End sickness/death Enhance cognition by integrating with willing minds. Safe and efficient interplanetary travel. Harness vastly higher levels of energy (solar, nuclear) for global benefit. Science: Uncover deeper insights into the laws of nature. Explore fundamental mysteries like the simulation hypothesis, Riemann hypothesis, multiverse theory, and the existence of white holes. Effective SETI Misc: End of violent conflicts Fair yet liberal resource allocation (if still needed), "from scarcity to abundance"
- Zambyte 2y ago
  Do you believe the average human has general intelligence, and do you believe the average human can intellectually achieve these things in ways existing technology cannot?
  - jagrsw 2y ago
    Yes, considering that AI operates differently from human minds, there are several advantages: AI does not experience fatigue or distractions => consistent performance. AI can scale its processing power significantly, despite the challenges associated with it (I understand the challenges) AI can ingest and process new information at an extraordinary speed. AIs can rewrite themselves AIs can be multiplicated (solving scarcity of intelligence in manufacturing) Once achieving AGI, progress could compound rapidly, for better or worse, due to the above points.
    - Jensson 2y ago
      The first AGI will probably take way too much compute to have a significant effect, unless there is a revolution in architecture that gets us fast and cheap AGI at once the AGI revolution will be very slow and gradual. A model that is as good as an average human but costs $10 000 per effective manhour to run is not very useful, but it is still an AGI.
      - jagrsw 2y ago
        > A model that is as good as an average human but costs $10 000 per effective manhour to run is not very useful, but it is still an AGI. Geohot (https://geohot.github.io/blog/ https://geohot.github.io/blog/) estimates that a human brain equivalent requires 20 PFLOPS. Current top-of-the-line GPUs are around 2 PFLOPS and consume up to 500W. Scaling that linearly results in 5kW, which translates to approximately 3 EUR per hour if I calculate correctly.
        Jensson 2y ago
        That is if the first model we make is as efficient as a human.
- jprete 2y ago
  The incentives aren't structured properly for these things to happen, it has always been a sci-fi fairy tale that AGI would achieve these things.
- TimPC 2y ago
  The problem with CO2 levels is that no one likes the solution not that we don't have one. I highly doubt adding AGI to the mix is going to magically make things better. If anything we'll just burn more CO2 providing all the compute resources it needs. People want their suburban lifestyle with their red meat and their pick-up truck or SUV. They drive fuel inefficient vehicles long-distances to urban work environments and they seem to have very limited interest in changing that. People who like detached homes aren't suddenly affording the rare instances of that closer to their work. We burn lots of oil because we drive fuel inefficient vehicles long distances. This is a problem of changing human preferences which you just aren't going to solve with an AGI.
  - jagrsw 2y ago
    Assuming embedded AI in every piece of robotics - sometimes directly, sometimes connected to a central server (this is doable even today) - it'll revolutionize industries: human-less mining, processing, manufacturing, services, and transportation. These factories would eventually produce and install enough solar power or build sufficient nuclear plants and energy infrastructure, making energy clean and free. With abundant electric cars (at this future point in time) and clean electricity powering heating, transportation, and manufacturing, some AIs could be repurposed for CO2 capture. It sounds deceptively easy, but from an engineering standpoint, it likely holds up. With free energy and AGI handling labor and thinking, we can achieve what a civilization could do and more (cause no individual incentives come into play). However, human factors could be a problem: protests (luddites), wireheading, misuse of AI, and AI-induced catastrophes (alignment).
    - breuleux 2y ago
      Having more energy is intrinsically dangerous, though, because it's indiscriminate: more energy cannot enable bigger solutions without also enabling bigger problems. Energy is the limiting factor to how much damage we can do. If we have way more of it, all bets are off. For instance, the current issue may be that we are indirectly cooking the planet through CO2 emissions, so capturing that sounds like a good idea. But even with clean energy, there is a point where we would cook the planet directly via waste heat of AI and gizmos and factories and whatever unforeseen crap we'll conjure just because we can. And given our track record I'm far from confident that we wouldn't do precisely that.
- davidgerard 2y ago
  now do magical flying unicorn ponies, which I understand there is also considerable demand for
hskalin 2y ago
self consciousness?
- Zambyte 2y ago
  And how do you test for that?
  - datameta 2y ago
    Probably the best thing we can do at the moment is compile a list of ways in which we shouldn't test for intelligence.
tsimionescu 2y ago
The simplest answer, without adding any extraordinary capabilities to the AGI that veer into magical intelligence, is to have AI assistants that can seemlessly interact with technology the way a human assistant would. So, if you want to meet with someone, instead of opening you calendar app and looking for an opening, you'd ask your AGI assistant to talk to their AGI assistant and set up a 1h meeting soon. Or, instead of going on Google to find plane tickets, you'd ask you AGI assistant to find the most reasonable tickets for a certain date range. This would not require any special intelligence more advanced than a human's, but it does require a very general understanding of the human world that is miles beyond what LLMs can achieve today. Going only slightly further with assumptions about how smart an AGI would be, it could revolutionize education, at any level, by acting as a true personalized tutor for a single student, or even for a small group of students. The single biggest problem in education is that it's impossible to scale the highest quality education - and an AGI with capabilities similar to a college professor would entirely solve that.
- Zambyte 2y ago
  This is definitely an interesting way to look at it. My initial reaction is to consider that I can enhance the capabilities of a system without increasing its inteligence. For example, if I give a monkey a hammer, it can do more than it could do when it didn't have the hammer, but it is not more intelligent (though it could probably learn things by interacting with the world with the hammer). That leads me to think: can we enhance the capabilities of what we call "AI systems" to do these things, without increasing their intelligence? It seems like you can glue GPT-4o to some calendar APIs to do exactly this. This seems more like an issue of tooling rather than an issue of intelligence to me. I guess the issue here is: can a system be "generally intelligent" if it doesn't have access to general tools to act on that intelligence? I think so, but I also can see how the line is very fuzzy between an AI system and the tools it can leverage, as really they both do information processing of some sort. Thanks for the insight.
  - tsimionescu 2y ago
    I'm sure some aspects of this can be achieved by manually programming GPT-4 links to other specific services. And obviously, some interaction tools would have to be written manually even for an AGI. The difference though is the amount of work. Today if you wanted GPT-4 to work as I describe, you would have to write an integration for Gmail, another one for Office365, another one for Proton etc. You would probably have to create a management interface to give access to your auth tokens for each of these to OpenAI so they can activate these interactions. The person you want to sync with would have to do the same. In contrast, an AGI that only has average human intelligence, or even below, would just need access to, say, Firefox APIs, and should easily be able to achieve all of this. And it would work regardless if the other side is a different AGI using a different provider, or even if they are just a regular human assistant.
Jensson 2y ago
An AGI could replace human experts at tasks that doesn't require physical embodiment, like diagnosing patients, drafting contracts, doing your taxes etc. If you still do those manually and not just offload all of it to ChatGPT then you would greatly benefit from a real AGI that could do those tasks on their own. And no, using ChatGPT like you use a search engine isn't ChatGPT solving your problem, that is you solving your problem. ChatGPT solving your problem would mean it drives you, not you driving it like it works today. When I hired people to help me do taxes they told me what papers they needed and then they did my taxes correctly without me having to look it through and correct them, an AGI would work like that for most tasks, it means you no longer need to think or learn to solve problems since the AGI solves them for you.
- Zambyte 2y ago
  Let's take a step back from LLMs. Could you accept the network of all interconnected computers as a generally intelligent system? The key part here that drives me to ask this is: > ChatGPT solving your problem would mean it drives you, not you driving it like it works today. I had a very bad Reddit addiction in the past. It took me years of consciously trying to quit in order to break the habit. I think I could make a reasonable argument that Reddit was using me to solve its problems, rather than myself using it to solve mine. I think this is also true of a lot of systems - Facebook, TikTok, YouTube, etc. It's hard to pin down all computers as an "agent" in the way we like to think about that word and assign some degree of intelligence to, but I think it is at least an interesting exercise to try.
  - Jensson 2y ago
    Companies are general intelligences and they use people, yes. But that depends on humans interpreting that data reddit users generates and updating their models, code and algorithms to adapt to that data, the computer systems alone aren't general intelligences if you remove the humans. An AGI could run such a company without humans anywhere in the loop, just like humans can run such a company without an AGI helping them. I'd say a strong signal that AGI has happened are large fully automated companies without a single human decisionmaker in the company, no CEO etc. Until that has happened I'd say AGI isn't here, if that happens it could be AGI but I can also imagine a good enough script to do it for some simple thing.
Symmetry 2y ago
A working memory that can preserve information indefinitely outside a particular context window and which can engage in multi-step reasoning that doesn't show up in its outputs. GPT4o's context window is 128k tokens which is somewhere on the order of 128kB. Your brain's context window, all the subliminal activations from the nerves in your gut and the parts of your visual field you aren't necessarily paying attention to is on the order of 2MB. So a similar order of magnitude though GPT has a sliding window and your brain has more of an exponential decay in activations. That LLMs can accomplish everything they do just with what seems analogous to human reflex rather than human reasoning is astounding and more than a bit scary.
- datameta 2y ago
  I'm curious what resources led you to calculate a 2MB context window, I'd like to learn more.
  - Symmetry 2y ago
    Looking up an estimate of the brain's input bandwidth at 10 million bits per second and multiplying by the second or two a subliminal stimuli can continue to affect a person's behavior. This is a very crude estimate and probably an order of magnitude off, but I don't think many orders of magnitude off.
amirhirsch 2y ago
Prove the Riemann Hypothesis
- Zambyte 2y ago
  Can you do that, are you not generally intelligent, or is that a bad metric for general intelligence? At least one of these is true.

insane_dreamer 2y ago

That's probably a good thing

anthonypasq 2y ago

how do you know that is actually currently their true goal. it appears to me that the goal has shifted, and the people that cared about the original goal are leaving.

shwaj 2y ago
I’m pretty sure Altman still cares about full AGI, if only because of the power inherent in achieving that goal.

JohnFen 2y ago

I don't think that their goal is achievable (at least not within any of our lifetimes), so I think that "plateauing" is inevitable.