7 ms·

> Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect. All of the focus on AGI is a distraction. I think it's important fo

> Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect. All of the focus on AGI is a distraction. I think it's important for a state to declare it's intent with a technology. The alternative is arguing the idea that technology advances autonomously, independent of human interactions, values, or ideas, which is, in my opinion, an incredibly naïve notion. I would rather have a state say "we won't use this technology for evil" than a state that says nothing at all and simply allows the businesses to develop in any direction their greed leads them. It's entirely valid to critique the uses of a technology, because "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly) is a technology like any other, like a landmine, like a synthetic virus, etc. In the same way, it's valid to criticize an actor for purposely hiding their intentions with a technology.

circuit10 2y ago
The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. In fact we humans are doing the same thing, we can't really improve our intelligence directly but we are trying to create AI to achieve our goals, and there's no reason that the AI itself wouldn't do so assuming it's capable and we don't attempt to stop it, and currently we don't really know how to reliably control it. We have absolutely no idea how to specify human values in a robust way which is what we would need to figure out to build this safely
- slg 2y ago
  I think that is missing the point. The AI's goals are what are determined by its human masters. Those human masters can already have nefarious and selfish goals that don't align with "human values". We don't need to invent hypothetical sentient AI boogeymen turning the universe into paperclips in order to be fearful of the future that ubiquitous AI creates. Humans would happily do that too if they get to preside over that paperclip empire.
  - Filligree 2y ago
    “Yes, X would be catastrophic. But have you considered Y, which is also catastrophic?” We need to avoid both, otherwise it’s a disaster either way.
    - slg 2y ago
      I agree, but that is removing the nuance that in this specific case Y is a prerequisite of X so focusing solely on X is a mistake. And for sake of clarity: X = sentient AI can do something dangerous Y = humans can use non-sentient AI to do something dangerous
      - circuit10 2y ago
        "sentient" (meaning "able to perceive or feel things") isn't a useful term here, it's impossible to measure objectively, it's an interesting philosophical question but we don't know if AI needs to be sentient to be powerful or what sentient even really means Humans will not be able to use AI do something selfish if we can't get it to do what we want at all, so we need to solve that (larger) problem before we come to that one
        wombatpm 2y ago
        Ok self flying drones that size if a deck of cards carrying a single bullet and enough processing power to fly around looking for faces, navigate to said face, fire when in range. Produce them by the thousands and release on the battlefield. Existing AI is more than capable.
        dgfitz 2y ago
        You can do that without AI. Been able to do it for probably 7-10 years.
        20after4 2y ago
        You can do that now, for sure, but I think it qualifies to call it AI. If you don't want to call it AI, that's fine too. It is indeed dangerous and already here. Making the autonomous programmed behavior of said tech more powerful (and more complex), along with more ubiquitous, just makes it even more dangerous.
- mr_toad 2y ago
  > The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. I’ve heard this argument before, and I don’t entirely accept it. It presumes that AI will be capable of playing 4D chess and thinking logically 10 moves ahead. It’s an interesting plot as a SF novel (literally the plot of the movie “I Robot”), but neural networks just don’t behave that way. They act, like us, on instinct (or training), not in some hyper-logical fashion. The idea that AI will behave like Star Trek’s Data (or Lore), has proven to be completely wrong.
  - bbor 2y ago
    Despite what Sam Altman (a high-school graduate) might want to be true, human cognition is not just a massive pile of intuition; there are critical deliberative and intentional aspects to cognition, which is something we've seen come to the fore with the hubbub around "reasoning" in LLMs. Any AGI design will necessarily take these facts into account--hardcoded or no--and will absolutely be capable of forming plans and executing them over time, as Simon & Newell described the best back in '71: The problem solver’s search for a solution is an odyssey through the problem space, from one knowledge state to another, until… [they] know the answer. With this in mind, I really don't see any basis to attack the intelligence explosion hypothesis. I linked a Yudkowsky paper above examining how empirically feasible it might be, which is absolutely an unsolved question at some level. But the utility of the effort itself is just downright obvious, even if we didn't have reams of internet discussions like this one to nudge any nascent agent in that direction. [1] Simon & Newell, 1971: Human Problem Solving https://psycnet.apa.org/record/1971-24266-001 https://psycnet.apa.org/record/1971-24266-001
    - marcus0x62 2y ago
      > Despite what Sam Altman (a high-school graduate) might want to be true > I linked a Yudkowsky paper above examining how empirically feasible it might be ...
      - bbor 2y ago
roenxi 2y ago
But if the state approaches a technology with intent it is usually for the purposes of a military offence. I don't think that is a good idea in the context of AI! Although I also don't think there is any stopping it. The US has things like DARPA for example and a lot of Chinese investment seems to be done with the intent of providing capabilities to their army. The list of things states have attempted to deploy offensively is nearly endless. Modern operations research arguably came out of the British empire attempting (succeeding) to weaponise mathematics. If you give a state fertiliser it makes bombs, if you give it nuclear power it makes bombs, if you give it drones it makes bombs, if you give it advanced science or engineering of any form it makes bombs. States are the most ingenious system for turning things into bombs that we've ever invented; in the grand old days of siege warfare they even managed to weaponise corpses, refuse and junk because it turned out lobbing that stuff at the enemy was effective. The entire spectrum of technology from nothing to nanotech, hurled at enemies to kill them. We'd all love if states commit to not doing evil but the state is the entity most active at figuring out how to use new tech X for evil.
- Agentus 2y ago
  quick devil’s advocate on a tangential point. is designer better killing tools necessarily evil? seems like the nature of the world is eat or be eaten and on the empire-scale, conquer or be conquered. that latter point seems to be the historical norm. Even with democracy, reasoning doesnt prevail but force of numbers seems to be the end determiner. Point is, humans arent easy to reason with or negotiate, coercion is the dominant force through out history especially when dealing with groups of different values. if one groups gives up the arms race of ultimate coercion tools or loses a conflict then they become subservient to the winners terms and norms (japan, germany, even Britain and France plus all the smaller states in between are subservient to the US)
  - musicale 2y ago
    > is design[ing] better killing tools necessarily evil? Who could possibly have predicted that the autonomous, invincible doomsday weapon we created for the good of humanity might one day be used against us?
bbor 2y ago
I usually don't engage on A[GS]I on here, but I feel like this is a decent time for an exception -- you're certainly well spoken and clear, which helps! Three things: (I) All of the focus on AGI is a distraction. I strongly disagree on that, at least if you're implying some intentionality. I think it's just provably true that many experts are honestly worried, even if you don't include the people who have dedicated a good portion of their lives to the cause. For example: OpenAI has certainly been corrupted through the loss of its nonprofit board, but I think their founding charter[1] was pretty clearly earnest -- and dire. (II) "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly) To be fair, this uncertainty in the term has been there since the dawn of the field, a fact made clear by perrenial rephrasings of the sentiment "AI is whatever hasn't been done yet" (~Larry Tesler 1979, see [2]). I'd love to get into the weeds on the different kinds of intelligence and why being to absolutist about the term can get real Faustian real quick, but these quotes bring up a more convincing, fundamental point: these chatbots are damn impressive. They do something--intuitive inference+fluent language use--that was impossible yesterday, and many experts would've guessed was decades away at least, if not centuries. Truly intelligent or not on their own, that's a more important development than you imply here. Finally, that brings me to the crux: (III) AI... is a technology like any other There's a famous Sundar Pichai (Google CEO) quote that he's been paraphrasing since 2018 -- soon after ChatGPT broke, he phrased it as such: I’ve always thought of A.I. as the most profound technology humanity is working on-more profound than fire or electricity or anything that we’ve done in the past. It gets to the essence of what intelligence is, what humanity is. We are developing technology which, for sure, one day will be far more capable than anything we’ve ever seen before. [3] When skeptics hear this, they understandably tend to write this off as capitalist bias from someone trying to pump Google's stock. However, I'd retort: 1) This kind of talk is so grandiose that it seems like a questionable move if that's the goal, 2) it's a sentiment echoed by many scientists (as I mentioned at the start of this rant) and 3) the unprecedented investments made across the world into the DL boom speak for themselves, sincerity-wise. Yes, this is because AI will create uber-efficient factories, upset labor relations, produce terrifying autonomous weapons, and all that stuff we're used to hearing about from the likes of Bostrom[4], Yudkowsky[5], and my personal fave, Huw Price[6]. But Pichai's raising something even more fundamental: the prospect of artificial people. Even if we ignore the I-Robot-style concerns about their potential moral standing, that is just a fundamentally spooky prospect, bringing very fundamental questions of A) individual worth and B) the nature of human cognition to the fore. And, to circle back: distinct from anything we've seen before. To close this long anxiety-driven manuscript, I'll end with a quote from an underappreciated philosopher of technology named Lewis Mumford on what he called "neotechnics": The scientific method, whose chief advances had been in mathematics and the physical sciences, took possession of other domains of experience: the living organism and human society also became the objects of systematic investigation... instead of mechanism forming a pattern for life, living organisms began to form a pattern for mechanism. In short, the concepts of science--hitherto associated largely with the cosmic, the inorganic, the "mechanical"--were now applied to every phase of human experience and every manifestation of life... men sought for an underlying order and logic of events which would embrace more complex manifestations.[7] TL;DR: IMHO, the US & UK refusing to cooperate at this critical moment is the most important event of your lifetime so far. [1] OpenAI's Charter https://web.archive.org/web/20230714043611/https://openai.com/charter https://web.archive.org/web/20230714043611/https://openai.co... [2] Investigation of a famous AI quote https://quoteinvestigator.com/2024/06/20/not-ai/ https://quoteinvestigator.com/2024/06/20/not-ai/ [3] Pichai, 2023: "AI is more profound than fire or electricity" https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profound-than-fire-electricity/ https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profou... [4] Bostrom, 2014: Superintelligence https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... [5] Yudkowsky, 2013: Intelligence Explosion Microeconomics https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [6] Huw Price's bio @ The Center for Existential Risk https://www.cser.ac.uk/team/huw-price/ https://www.cser.ac.uk/team/huw-price/ [7] Mumford, 1934: Technics and Civilization https://archive.org/details/in.ernet.dli.2015.49974 https://archive.org/details/in.ernet.dli.2015.49974
jstanley 2y ago
> I think it's important for a state to declare it's intent with a technology. The alternative is arguing the idea that technology advances autonomously, independent of human interactions, values, or ideas The sleight of hand here is the implication that human interactions, values, and ideas are only expressed through the state.
- forgetfreeman 2y ago
  The sleight of hand here is implying that there are any forces smaller than nation states that can credibly reign in problematic technology. Relying on good intentions to win out against market forces isn't even naive, it's just stupid.
  - TeMPOraL 2y ago
    So many sleights here. Another sleight of hand in this subthread is suggesting that "the idea that technology advances autonomously, independent of human interactions, values, or ideas" is merely an idea, and not an actual observable fact at scale. Society and culture are downstream of economics, and economics is mostly downstream of technological progress. Of course, the progress isn't autonomous in the sense of having a sentient mind of its own - it's "merely" gradient descent down the economic landscape. Just like the market itself. There's no reining in of problematic technology unless, like you say, nation states get involved directly. And they don't stand much chance either unless they get serious. People still laugh at Eliezer's comments from that news article of yesteryear, but he was and is spot-on: being serious about restricting technology actually does mean threatening to drop bombs on facilities developing it in violation of restrictions - if we're not ready to have our representatives make such threats, and then actually follow through and drop the bombs if someone decides to test our resolve, then we're not serious.
    - forgetfreeman 2y ago
      People laugh at all kinds of common sense declarations that they ought not find funny in the slightest. Its one of our species glaring failures.

circuit10 2y ago

The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. In fact we humans are doing the same thing, we can't really improve our intelligence directly but we are trying to create AI to achieve our goals, and there's no reason that the AI itself wouldn't do so assuming it's capable and we don't attempt to stop it, and currently we don't really know how to reliably control it. We have absolutely no idea how to specify human values in a robust way which is what we would need to figure out to build this safely

slg 2y ago
I think that is missing the point. The AI's goals are what are determined by its human masters. Those human masters can already have nefarious and selfish goals that don't align with "human values". We don't need to invent hypothetical sentient AI boogeymen turning the universe into paperclips in order to be fearful of the future that ubiquitous AI creates. Humans would happily do that too if they get to preside over that paperclip empire.
- Filligree 2y ago
  “Yes, X would be catastrophic. But have you considered Y, which is also catastrophic?” We need to avoid both, otherwise it’s a disaster either way.
  - slg 2y ago
    I agree, but that is removing the nuance that in this specific case Y is a prerequisite of X so focusing solely on X is a mistake. And for sake of clarity: X = sentient AI can do something dangerous Y = humans can use non-sentient AI to do something dangerous
    - circuit10 2y ago
      "sentient" (meaning "able to perceive or feel things") isn't a useful term here, it's impossible to measure objectively, it's an interesting philosophical question but we don't know if AI needs to be sentient to be powerful or what sentient even really means Humans will not be able to use AI do something selfish if we can't get it to do what we want at all, so we need to solve that (larger) problem before we come to that one
      - wombatpm 2y ago
        Ok self flying drones that size if a deck of cards carrying a single bullet and enough processing power to fly around looking for faces, navigate to said face, fire when in range. Produce them by the thousands and release on the battlefield. Existing AI is more than capable.
        dgfitz 2y ago
        You can do that without AI. Been able to do it for probably 7-10 years.
        20after4 2y ago
        You can do that now, for sure, but I think it qualifies to call it AI. If you don't want to call it AI, that's fine too. It is indeed dangerous and already here. Making the autonomous programmed behavior of said tech more powerful (and more complex), along with more ubiquitous, just makes it even more dangerous.
mr_toad 2y ago
> The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. I’ve heard this argument before, and I don’t entirely accept it. It presumes that AI will be capable of playing 4D chess and thinking logically 10 moves ahead. It’s an interesting plot as a SF novel (literally the plot of the movie “I Robot”), but neural networks just don’t behave that way. They act, like us, on instinct (or training), not in some hyper-logical fashion. The idea that AI will behave like Star Trek’s Data (or Lore), has proven to be completely wrong.
- bbor 2y ago
  Despite what Sam Altman (a high-school graduate) might want to be true, human cognition is not just a massive pile of intuition; there are critical deliberative and intentional aspects to cognition, which is something we've seen come to the fore with the hubbub around "reasoning" in LLMs. Any AGI design will necessarily take these facts into account--hardcoded or no--and will absolutely be capable of forming plans and executing them over time, as Simon & Newell described the best back in '71: The problem solver’s search for a solution is an odyssey through the problem space, from one knowledge state to another, until… [they] know the answer. With this in mind, I really don't see any basis to attack the intelligence explosion hypothesis. I linked a Yudkowsky paper above examining how empirically feasible it might be, which is absolutely an unsolved question at some level. But the utility of the effort itself is just downright obvious, even if we didn't have reams of internet discussions like this one to nudge any nascent agent in that direction. [1] Simon & Newell, 1971: Human Problem Solving https://psycnet.apa.org/record/1971-24266-001 https://psycnet.apa.org/record/1971-24266-001
  - marcus0x62 2y ago
    > Despite what Sam Altman (a high-school graduate) might want to be true > I linked a Yudkowsky paper above examining how empirically feasible it might be ...
    - bbor 2y ago

roenxi 2y ago

But if the state approaches a technology with intent it is usually for the purposes of a military offence. I don't think that is a good idea in the context of AI! Although I also don't think there is any stopping it. The US has things like DARPA for example and a lot of Chinese investment seems to be done with the intent of providing capabilities to their army. The list of things states have attempted to deploy offensively is nearly endless. Modern operations research arguably came out of the British empire attempting (succeeding) to weaponise mathematics. If you give a state fertiliser it makes bombs, if you give it nuclear power it makes bombs, if you give it drones it makes bombs, if you give it advanced science or engineering of any form it makes bombs. States are the most ingenious system for turning things into bombs that we've ever invented; in the grand old days of siege warfare they even managed to weaponise corpses, refuse and junk because it turned out lobbing that stuff at the enemy was effective. The entire spectrum of technology from nothing to nanotech, hurled at enemies to kill them. We'd all love if states commit to not doing evil but the state is the entity most active at figuring out how to use new tech X for evil.

Agentus 2y ago
quick devil’s advocate on a tangential point. is designer better killing tools necessarily evil? seems like the nature of the world is eat or be eaten and on the empire-scale, conquer or be conquered. that latter point seems to be the historical norm. Even with democracy, reasoning doesnt prevail but force of numbers seems to be the end determiner. Point is, humans arent easy to reason with or negotiate, coercion is the dominant force through out history especially when dealing with groups of different values. if one groups gives up the arms race of ultimate coercion tools or loses a conflict then they become subservient to the winners terms and norms (japan, germany, even Britain and France plus all the smaller states in between are subservient to the US)
- musicale 2y ago
  > is design[ing] better killing tools necessarily evil? Who could possibly have predicted that the autonomous, invincible doomsday weapon we created for the good of humanity might one day be used against us?

bbor 2y ago

I usually don't engage on A[GS]I on here, but I feel like this is a decent time for an exception -- you're certainly well spoken and clear, which helps! Three things: (I) All of the focus on AGI is a distraction. I strongly disagree on that, at least if you're implying some intentionality. I think it's just provably true that many experts are honestly worried, even if you don't include the people who have dedicated a good portion of their lives to the cause. For example: OpenAI has certainly been corrupted through the loss of its nonprofit board, but I think their founding charter[1] was pretty clearly earnest -- and dire. (II) "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly) To be fair, this uncertainty in the term has been there since the dawn of the field, a fact made clear by perrenial rephrasings of the sentiment "AI is whatever hasn't been done yet" (~Larry Tesler 1979, see [2]). I'd love to get into the weeds on the different kinds of intelligence and why being to absolutist about the term can get real Faustian real quick, but these quotes bring up a more convincing, fundamental point: these chatbots are damn impressive. They do something--intuitive inference+fluent language use--that was impossible yesterday, and many experts would've guessed was decades away at least, if not centuries. Truly intelligent or not on their own, that's a more important development than you imply here. Finally, that brings me to the crux: (III) AI... is a technology like any other There's a famous Sundar Pichai (Google CEO) quote that he's been paraphrasing since 2018 -- soon after ChatGPT broke, he phrased it as such: I’ve always thought of A.I. as the most profound technology humanity is working on-more profound than fire or electricity or anything that we’ve done in the past. It gets to the essence of what intelligence is, what humanity is. We are developing technology which, for sure, one day will be far more capable than anything we’ve ever seen before. [3] When skeptics hear this, they understandably tend to write this off as capitalist bias from someone trying to pump Google's stock. However, I'd retort: 1) This kind of talk is so grandiose that it seems like a questionable move if that's the goal, 2) it's a sentiment echoed by many scientists (as I mentioned at the start of this rant) and 3) the unprecedented investments made across the world into the DL boom speak for themselves, sincerity-wise. Yes, this is because AI will create uber-efficient factories, upset labor relations, produce terrifying autonomous weapons, and all that stuff we're used to hearing about from the likes of Bostrom[4], Yudkowsky[5], and my personal fave, Huw Price[6]. But Pichai's raising something even more fundamental: the prospect of artificial people. Even if we ignore the I-Robot-style concerns about their potential moral standing, that is just a fundamentally spooky prospect, bringing very fundamental questions of A) individual worth and B) the nature of human cognition to the fore. And, to circle back: distinct from anything we've seen before. To close this long anxiety-driven manuscript, I'll end with a quote from an underappreciated philosopher of technology named Lewis Mumford on what he called "neotechnics": The scientific method, whose chief advances had been in mathematics and the physical sciences, took possession of other domains of experience: the living organism and human society also became the objects of systematic investigation... instead of mechanism forming a pattern for life, living organisms began to form a pattern for mechanism. In short, the concepts of science--hitherto associated largely with the cosmic, the inorganic, the "mechanical"--were now applied to every phase of human experience and every manifestation of life... men sought for an underlying order and logic of events which would embrace more complex manifestations.[7] TL;DR: IMHO, the US & UK refusing to cooperate at this critical moment is the most important event of your lifetime so far. [1] OpenAI's Charter https://web.archive.org/web/20230714043611/https://openai.com/charter https://web.archive.org/web/20230714043611/https://openai.co... [2] Investigation of a famous AI quote https://quoteinvestigator.com/2024/06/20/not-ai/ https://quoteinvestigator.com/2024/06/20/not-ai/ [3] Pichai, 2023: "AI is more profound than fire or electricity" https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profound-than-fire-electricity/ https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profou... [4] Bostrom, 2014: Superintelligence https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... [5] Yudkowsky, 2013: Intelligence Explosion Microeconomics https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [6] Huw Price's bio @ The Center for Existential Risk https://www.cser.ac.uk/team/huw-price/ https://www.cser.ac.uk/team/huw-price/ [7] Mumford, 1934: Technics and Civilization https://archive.org/details/in.ernet.dli.2015.49974 https://archive.org/details/in.ernet.dli.2015.49974

jstanley 2y ago

> I think it's important for a state to declare it's intent with a technology. The alternative is arguing the idea that technology advances autonomously, independent of human interactions, values, or ideas The sleight of hand here is the implication that human interactions, values, and ideas are only expressed through the state.

forgetfreeman 2y ago
The sleight of hand here is implying that there are any forces smaller than nation states that can credibly reign in problematic technology. Relying on good intentions to win out against market forces isn't even naive, it's just stupid.
- TeMPOraL 2y ago
  So many sleights here. Another sleight of hand in this subthread is suggesting that "the idea that technology advances autonomously, independent of human interactions, values, or ideas" is merely an idea, and not an actual observable fact at scale. Society and culture are downstream of economics, and economics is mostly downstream of technological progress. Of course, the progress isn't autonomous in the sense of having a sentient mind of its own - it's "merely" gradient descent down the economic landscape. Just like the market itself. There's no reining in of problematic technology unless, like you say, nation states get involved directly. And they don't stand much chance either unless they get serious. People still laugh at Eliezer's comments from that news article of yesteryear, but he was and is spot-on: being serious about restricting technology actually does mean threatening to drop bombs on facilities developing it in violation of restrictions - if we're not ready to have our representatives make such threats, and then actually follow through and drop the bombs if someone decides to test our resolve, then we're not serious.
  - forgetfreeman 2y ago
    People laugh at all kinds of common sense declarations that they ought not find funny in the slightest. Its one of our species glaring failures.