4 ms·

> Superintelligence will also be bound by fundamental information-theoretic limits …is mainly why this has not been worrying me much. All issues with modern in

> Superintelligence will also be bound by fundamental information-theoretic limits …is mainly why this has not been worrying me much. All issues with modern incarnation of generative ML aside, AGI doomism really does strike me as profitable deity-worshipping death cult. The biggest threat in the equation will always be humans who deploy tech, not the tech itself.

Filligree 2y ago
Well, sure, those fundamental limits always apply. But why do you assume humanity is anywhere close to the limits?
ben_w 2y ago
Saying that humans can be a threat and AI can't, seems to me to be a claim that humans are exempt from information-theoretic limits. I see general-AI (given that robots already exist) as having the potential to do all that humans can do, including be harmful. Like most, I also model superhuman-AI in a trivial way, as being as capable as the best human in every field — Einstein's grasp of physics, Sun Tzu's of military strategy, Churchill's of rhetoric, etc. > The biggest threat in the equation will always be humans who deploy tech, not the tech itself. Naturally; but does that include humans who unwittingly set the AI's utility function with a bug in it and pressed "run", in a way that ends with us saying "the situation has developed not necessarily to [Humanity's] advantage"?
- anileated 2y ago
  Any tech can be harmful, cf. cryptocurrency or nuclear bombs. My comment mostly referred to taking it to such extreme with a particular tech. Some people also tend, while demonizing (the yet to be confirmed to be possible even in theory) AGI and its capabilities, to completely sideline humans deploying it, as if they would be guilt-free.
  - ben_w 2y ago
    > the yet to be confirmed to be possible even in theory It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together, and simulate their combined brain activity. It's all ultimately physics, so it's possible. The hard part isn't the theory, but actually doing that — and not just because we don't know how to scan living brains to measure synaptic weights. There's a lot we don't know about how minds work in general nor how our brains work in particular. But purely in theory, that method ultimately boils down to known physics. (And as for why even that may lead to bad outcomes depending on who asks this simulated amalgam for advice: https://en.wikipedia.org/wiki/Milgram_experiment https://en.wikipedia.org/wiki/Milgram_experiment, https://en.wikipedia.org/wiki/Groupthink https://en.wikipedia.org/wiki/Groupthink, and https://en.wikipedia.org/wiki/Peer_pressure https://en.wikipedia.org/wiki/Peer_pressure) > as if they would be guilt-free. You can assign guilt and blame as you see fit. In these kinds of scenario, the human who makes the fatal error, who many might call responsible, would likely be going "but it works on my machine!" as they die from some error that may or may not be obvious even in hindsight. I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people — even back 20 years ago when I did my degree, we had case studies from Therac-25 and the London Ambulance Service: https://en.wikipedia.org/wiki/LASCAD https://en.wikipedia.org/wiki/LASCAD Even back then our profession knew we needed to be held to higher standards, but didn't want to do what that entailed.
    - anileated 2y ago
      > It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together That would be a far cry from what AGI paperclip doomists imagine. > and simulate their combined brain activity. If simulating their brain activity is the same as what would happen if they were actually in the room, what we would get is probably 80% arguing over what to do, posturing, etc. It’s not clear to me how we would cause some intelligence to arise that would be immune to that while being superior in other ways and be sufficiently human to appear as intelligence and not, say, some calculation being performed like computers did for decades. > I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people I absolutely agree that us programmers should generally be more aware how “neutral” tech they develop is used to not only to benefit but also to hurt people—be it cryptocurrency, E2EE communications, or indeed very much ML—and work towards alleviating that in practical and ethical ways. I just think in case of ML the threat is more prosaic. (I can be wrong, though I’d hate to be proven so, of course.)

Filligree 2y ago

Well, sure, those fundamental limits always apply. But why do you assume humanity is anywhere close to the limits?

ben_w 2y ago

Saying that humans can be a threat and AI can't, seems to me to be a claim that humans are exempt from information-theoretic limits. I see general-AI (given that robots already exist) as having the potential to do all that humans can do, including be harmful. Like most, I also model superhuman-AI in a trivial way, as being as capable as the best human in every field — Einstein's grasp of physics, Sun Tzu's of military strategy, Churchill's of rhetoric, etc. > The biggest threat in the equation will always be humans who deploy tech, not the tech itself. Naturally; but does that include humans who unwittingly set the AI's utility function with a bug in it and pressed "run", in a way that ends with us saying "the situation has developed not necessarily to [Humanity's] advantage"?

anileated 2y ago
Any tech can be harmful, cf. cryptocurrency or nuclear bombs. My comment mostly referred to taking it to such extreme with a particular tech. Some people also tend, while demonizing (the yet to be confirmed to be possible even in theory) AGI and its capabilities, to completely sideline humans deploying it, as if they would be guilt-free.
- ben_w 2y ago
  > the yet to be confirmed to be possible even in theory It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together, and simulate their combined brain activity. It's all ultimately physics, so it's possible. The hard part isn't the theory, but actually doing that — and not just because we don't know how to scan living brains to measure synaptic weights. There's a lot we don't know about how minds work in general nor how our brains work in particular. But purely in theory, that method ultimately boils down to known physics. (And as for why even that may lead to bad outcomes depending on who asks this simulated amalgam for advice: https://en.wikipedia.org/wiki/Milgram_experiment https://en.wikipedia.org/wiki/Milgram_experiment, https://en.wikipedia.org/wiki/Groupthink https://en.wikipedia.org/wiki/Groupthink, and https://en.wikipedia.org/wiki/Peer_pressure https://en.wikipedia.org/wiki/Peer_pressure) > as if they would be guilt-free. You can assign guilt and blame as you see fit. In these kinds of scenario, the human who makes the fatal error, who many might call responsible, would likely be going "but it works on my machine!" as they die from some error that may or may not be obvious even in hindsight. I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people — even back 20 years ago when I did my degree, we had case studies from Therac-25 and the London Ambulance Service: https://en.wikipedia.org/wiki/LASCAD https://en.wikipedia.org/wiki/LASCAD Even back then our profession knew we needed to be held to higher standards, but didn't want to do what that entailed.
  - anileated 2y ago
    > It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together That would be a far cry from what AGI paperclip doomists imagine. > and simulate their combined brain activity. If simulating their brain activity is the same as what would happen if they were actually in the room, what we would get is probably 80% arguing over what to do, posturing, etc. It’s not clear to me how we would cause some intelligence to arise that would be immune to that while being superior in other ways and be sufficiently human to appear as intelligence and not, say, some calculation being performed like computers did for decades. > I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people I absolutely agree that us programmers should generally be more aware how “neutral” tech they develop is used to not only to benefit but also to hurt people—be it cryptocurrency, E2EE communications, or indeed very much ML—and work towards alleviating that in practical and ethical ways. I just think in case of ML the threat is more prosaic. (I can be wrong, though I’d hate to be proven so, of course.)