5 ms·

This is a great point. There’s a lot of wooly thinking about what it means for a system to be intelligent, particularly “super” intelligent. People seem to thin

This is a great point. There’s a lot of wooly thinking about what it means for a system to be intelligent, particularly “super” intelligent. People seem to think we’ll create a machine that is almost literally infallible — able to predict both physical systems and human behavior with perfect foresight many steps in advance. I’m not sure that’s even possible, let alone likely.

logicchains 3y ago
>able to predict both physical systems and human behavior with perfect foresight many steps in advance It's literally impossible. That's a key part of https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory ; in formally chaotic systems (of which there are many), the uncertainty in a forecast increases exponentially with elapsed time, which means it's impossible for an AI, no matter how smart, to predict very far ahead. People who believe otherwise are engaging in magical thinking; like "intelligence" is some magical quality that allows one to break the laws of the universe, do the equivalent of sorting a list in O(1) time.
- ethanbond 3y ago
  Yet we have pretty good weather forecasts, pretty good traffic forecasts, pretty good election forecasts, and so on, and they continue to get better with more data and more compute. While it may never be able to “perfectly predict” at Resolution X does not mean it can’t perfectly predict at Resolution <X. We already have good predictions of human behavior at some resolutions.
benlivengood 3y ago
Part of the existential risk is that intelligent goal-directed agents will recognize the unpredictability of their environment and do what humans have done to our environment; simplify and systematize it so that it is more predictable (forestry, mining, pavement, antibiotics, pest-control, industry). Biological life is unpredictable and so replacing it with simpler biological organisms or well-understood machines and systems will be a valid strategy for artificial agents to obtain and retain power to achieve their goals. To put it another way; there are two ways to approach infallibility: be uncomputably intelligent (AIXI) or bulldoze/terraform/grey-goo the Universe into something very predictable. An agent smart enough to understand both options and how reality works is likely to choose the physically realizable option. Only careful choices of goals that value human flourishing (AI alignment) will avoid optimizing us out of the picture.
- c_crank 3y ago
  By what physical means do these AI 'goal directed agents' actually manipulate the physical world?
  - ChatGTP 3y ago
    Have you ever heard of robots ? Have you ever heard of Boston Dynamics and Tesla ? I’m being snarky but you keep commenting on these posts with imo intentionally naïve comments. People will make super smart robots if they have the tech and there is no regulation stopping it. Why people are pretending this is some far out inconceivable idea ? Denial ?
    - c_crank 3y ago
      I have heard of these robots and auto driving cars. Both of these have pretty strong limitations regarding their ability to affect the environment as much as a human can. Typically these robots are heavily limited by battery usage and need to be close by to a source of electrical power. They also lack the ability to repair themselves, or construct new robots. A full fledged robot ecosystem with robots building other robots building other robots without any human intervention is a long way away, and is essential for these kind of scenarios that AI Doomsayers naively propose. The assumption seems to be that, if one builds the brain of the robot, the body and the hands, the heart and the lungs, will just all spontaneously fall into place.
      - ChatGTP 3y ago
        I don’t think it’s inconceivable that if humans can build robots a super intelligence at of the future would be able to work it out. They would also workout how to build more efficient systems , charging infra and batteries. But I agree there would be limits to things like batteries. I don’t entirely disagree with your points either. I just don’t imagine any of this is insurmountable if super intelligence becomes reality. Personally I think we’ll create this reality through military application before any AI does.
ethanbond 3y ago
What is "intelligence" other than proximity to that destination?

logicchains 3y ago

>able to predict both physical systems and human behavior with perfect foresight many steps in advance It's literally impossible. That's a key part of https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory ; in formally chaotic systems (of which there are many), the uncertainty in a forecast increases exponentially with elapsed time, which means it's impossible for an AI, no matter how smart, to predict very far ahead. People who believe otherwise are engaging in magical thinking; like "intelligence" is some magical quality that allows one to break the laws of the universe, do the equivalent of sorting a list in O(1) time.

ethanbond 3y ago
Yet we have pretty good weather forecasts, pretty good traffic forecasts, pretty good election forecasts, and so on, and they continue to get better with more data and more compute. While it may never be able to “perfectly predict” at Resolution X does not mean it can’t perfectly predict at Resolution <X. We already have good predictions of human behavior at some resolutions.

benlivengood 3y ago

Part of the existential risk is that intelligent goal-directed agents will recognize the unpredictability of their environment and do what humans have done to our environment; simplify and systematize it so that it is more predictable (forestry, mining, pavement, antibiotics, pest-control, industry). Biological life is unpredictable and so replacing it with simpler biological organisms or well-understood machines and systems will be a valid strategy for artificial agents to obtain and retain power to achieve their goals. To put it another way; there are two ways to approach infallibility: be uncomputably intelligent (AIXI) or bulldoze/terraform/grey-goo the Universe into something very predictable. An agent smart enough to understand both options and how reality works is likely to choose the physically realizable option. Only careful choices of goals that value human flourishing (AI alignment) will avoid optimizing us out of the picture.

c_crank 3y ago
By what physical means do these AI 'goal directed agents' actually manipulate the physical world?
- ChatGTP 3y ago
  Have you ever heard of robots ? Have you ever heard of Boston Dynamics and Tesla ? I’m being snarky but you keep commenting on these posts with imo intentionally naïve comments. People will make super smart robots if they have the tech and there is no regulation stopping it. Why people are pretending this is some far out inconceivable idea ? Denial ?
  - c_crank 3y ago
    I have heard of these robots and auto driving cars. Both of these have pretty strong limitations regarding their ability to affect the environment as much as a human can. Typically these robots are heavily limited by battery usage and need to be close by to a source of electrical power. They also lack the ability to repair themselves, or construct new robots. A full fledged robot ecosystem with robots building other robots building other robots without any human intervention is a long way away, and is essential for these kind of scenarios that AI Doomsayers naively propose. The assumption seems to be that, if one builds the brain of the robot, the body and the hands, the heart and the lungs, will just all spontaneously fall into place.
    - ChatGTP 3y ago
      I don’t think it’s inconceivable that if humans can build robots a super intelligence at of the future would be able to work it out. They would also workout how to build more efficient systems , charging infra and batteries. But I agree there would be limits to things like batteries. I don’t entirely disagree with your points either. I just don’t imagine any of this is insurmountable if super intelligence becomes reality. Personally I think we’ll create this reality through military application before any AI does.

ethanbond 3y ago

What is "intelligence" other than proximity to that destination?