10 ms·

The Myth of a Superhuman AI

30 points by conanxin 5y ago

totally_not_ai 5y ago
> Hyper-intelligent algorithms are not going to take over the world for these five reasons. I agree. Just like a nuclear power plant will never meltdown, because they are extremely well designed with large safety buffers. And just like a pandemic will never be started by lab experiments, because they use extreme safety measures when they work with viruses. We can always just unplug the AI or disconnect it from the Internet, or just not give it a survival instinct, right?
- gilch 5y ago
  Pretty sure this is sarcasm, as in, the lies are so transparent they're humorous, and this was on purpose. For those who didn't notice, nuclear plants have, in fact, melted down, more than once. Biosafety level 4 labs have, in fact, had leaks of pathogens before, more than once. An agent AI will tend to develop a survival instinct by default, even if we don't give it one, because it is a convergent instrumental goal. Thus, it would be motivated to not simply let you yank it, would fight you if you tried, and may even take preemptive precautions.
tech-historian 5y ago
Who is the author Kevin Kelly and why does he think he has a greater hold on this issue than luminaries like Ray Kurzweil?
- mindcrime 5y ago
  To the first half of your question: https://en.wikipedia.org/wiki/Kevin_Kelly_(editor) https://en.wikipedia.org/wiki/Kevin_Kelly_(editor) To the second half: Dunno. Good question.
- cowpig 5y ago
  Appeals to authority are not convincing
  - mindcrime 5y ago
    This isn't a formal debate, and authority is useful especially as a "first pass" filter, or in terms of establishing Bayesian priors. The key, IMO, is not to assume that the mere existence of authority guarantees that someone is right. But in either case, it's certainly valid to ask the question "why should I take Kevin Kelly seriously on this topic?"
    - cowpig 5y ago
      I don't agree in this context. The author wrote an entire article about his position, and dismissing it entirely because some "luminary" disagrees is not an interesting response. OP didn't even bother to point out how or why Kurzweil purportedly disagrees.
      - tech-historian 5y ago
        I thought it was obvious. Clearly it was not. Ray Kurzweil is a futurist who has spent multiple decades researching this topic and has written multiple books on it. His bio speaks for itself. https://en.wikipedia.org/wiki/Ray_Kurzweil https://en.wikipedia.org/wiki/Ray_Kurzweil
      - mindcrime 5y ago
        I guess I interpreted tech-historian's post a little differently than you. Likewise, I don't support dismissing Kelly's article out-of-hand, based on Kurzweil's position. But I do think it's fair to raise the issue in terms of "why should one take Kevin Kelly seriously on this?"
- AnimalMuppet 5y ago
  Why is Ray Kurzweil considered a luminary in this context? Sure, he talks a lot. He makes grand statements. Is there any reason to take him seriously in this context? He sure seems to believe, and to want to believe. But actual evidence that he's going to be right any time soon? That seems to be missing.
  - enord 5y ago
    Everybody wants to be the Fox Mulder of their story.
  - tech-historian 5y ago
    It's clear you're not familiar with Kurzweil's work in AI (or his background in general). He's written multiple books on the topic over multiple decades, including the seminal "The Singularity is Near". He's devoted his life to the study of this field. https://en.wikipedia.org/wiki/Ray_Kurzweil https://en.wikipedia.org/wiki/Ray_Kurzweil https://en.wikipedia.org/wiki/The_Singularity_Is_Near https://en.wikipedia.org/wiki/The_Singularity_Is_Near
    - zaybqp 5y ago
      Thank you for enlightening us with the new knowledge of this wonderful luminary. I have learned so much from these URLs. Do you have any more recommendations on what we should read next? I find myself deeply enthralled by his fascinating claims and I want to seek out more of his teachings.
    - 6gvONxR4sf7o 5y ago
      He's also written books on how much of your diet should be fat, and similar topics. I've never been able to tell if he's worth taking seriously. My instincts say no. https://en.wikipedia.org/wiki/The_10%25_Solution_for_a_Healthy_Life https://en.wikipedia.org/wiki/The_10%25_Solution_for_a_Healt... https://en.wikipedia.org/wiki/Fantastic_Voyage:_Live_Long_Enough_to_Live_Forever https://en.wikipedia.org/wiki/Fantastic_Voyage:_Live_Long_En...
    - AnimalMuppet 5y ago
- aaaaaaaaaaab 5y ago
  >luminaries like Ray Kurzweil This is satire, right?
deleted 5y ago
[deleted]
sxp 5y ago
tl;dr: Humans have souls which machines will never have. The core of the article is this quote "I will extend that further to claim that the only way to get a very human-like thought process is to run the computation on very human-like wet tissue. That also means that very big, complex artificial intelligences run on dry silicon will produce big, complex, unhuman-like minds." which is just the boring old argument against replicating human thinking. Similarly, there is an extreme amount of pro-human arrogance in the article "At the core of the notion of a superhuman intelligence — particularly the view that this intelligence will keep improving itself — is the essential belief that intelligence has an infinite scale. I find no evidence for this.... So the question is, where is the limit of intelligence? We tend to believe that the limit is way beyond us, way “above” us, as we are “above” an ant. Setting aside the recurring problem of a single dimension, what evidence do we have that the limit is not us? Why can’t we be at the maximum?" Humans are just a bunch of atoms that randomly learned to think over a few billion years. Given how fast we can improve tech via directed means, we should be able to replicate what nature has done in thousands, hundreds, or tens of years.
- cowpig 5y ago
  The word "soul" does not appear once in the article
- DuskStar 5y ago
  > That also means that very big, complex artificial intelligences run on dry silicon will produce big, complex, unhuman-like minds. Incidentally, this is exactly what the AI superintelligence people are terrified of, and what the paperclip problem is meant to illustrate. https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html
mindcrime 5y ago
Ugh. This is a pretty bad article. And don't get me wrong... I'm no "AI Alarmist." But to me, this article is equally as useless as the various pronouncements it seems to be trying to refute. There's a lot of verbiage here, but no cohesive, coherent argument to support the thesis that "Superhuman AI is just a myth" (which is to say, "Superhuman AI is not possible"). The reality is, there are a lot of unknowns in this space, and to dismiss the possibility of an "evil super-intelligence" out-of-hand is, IMO, every bit as silly as buying into the "OMG, AI is unleashing the demon" fear-mongering. I just think it's too early to know for sure the answers to some of these questions, like: 1. Is AGI possible at all (for whatever definition of "AGI" you want to use)? 2. If the answer to (1) is "yes", then what about ASI (Artificial Super Intelligence) - is that possible? 3. If (1) and (2) are both "yes", then is it possible to create an ASI that will either be benevolent, or controllable, or both? And so on... we just don't know yet. And that in and of itself arguably suggests the exercise of a certain measure of caution.
- enord 5y ago
  You're making an argument that is already in TFA. It's in addition -and only partially related to- the main point which is: The arguments that are commonly used to demonstrate the possibility or threat of "ASI" are made from claims presented _without evidence_. This will invalidate fantastical accelerationist woo until such evidence is presented, and -in the current state of the art research within relevant fields- it's not even certain we can know what would constitute such evidence _in principle_.
  - mindcrime 5y ago
    I assume you're referring to this part: If the expectation of a superhuman AI takeover is built on five key assumptions that have no basis in evidence, then this idea is more akin to a religious belief — a myth My problem is, even this is wrong. I do not find the "five key assumptions" he cites to be the foundation of beliefs that a dangerous "Superhuman AI" could emerge. So to me, this article simply fails right from the jump. The thing he is arguing for and/or against just don't, collectively, constitute a well-developed argument for the overall thesis. Not in my opinion anyway. To go through them one by one: Artificial intelligence is already getting smarter than us, at an exponential rate. I don't know anybody making this claim, nor is it apparent that belief in this is necessary to think that dangerous "Superhuman AI" could be a problem in the future. I guess it comes down to what you think Kelly meant by "soon" in the preceding sentence. We’ll make AIs into a general purpose intelligence, like our own. It's the "like our own" part of this that I have a problem with. Yes, I think many people believe that we will create some manner of "general intelligence", but I also believe that a lot of researchers acknowledge that it may not be 100% "human like". But this may be my own bias showing, as I personally harp a lot on the importance of the distinction between "human level" intelligence and "human like" intelligence. We can make human intelligence in silicon. This is probably the most sound one of all, but here's the thing... saying "there is no evidence for this" is simply wrong. To hold to this is to completely ignore all progress made building AI in silicon to date. And while we don't yet have "general intelligence", I don't see how one can just ignore incremental progress completely and say "there's nothing there". I would argue that the null hypothesis should probably be that general intelligence on silicon is possible that that the burden of proof is on the people arguing the opposite. Also, not everybody thinking about / working on AI is focused on the "in silicon" approach. The majority? Yes. But I'm sure there are a handful of people working on ideas related to DNA computing, chemical computing, growing synthetic brain matter, etc. Intelligence can be expanded without limit. To me, this may or not not be true, but it's completely irrelevant to the thesis at hand. The question isn't "can machine intelligence be unbounded?", but rather "can machine intelligence exceed human intelligence?" Once we have exploding superintelligence it can solve most of our problems. Again, this strikes me as mostly irrelevant to the question at hand. So yes, this article raises some interesting points, but I don't see it accomplishing what it seems to purport to do.
blamestross 5y ago
The bottleneck for intelligence is experience. Running on silicon doesn't allow an intelligence to engage with reality significantly faster than a human can. There are some benefits due to access to data and processing power, but the data isn't useful without the context of experience and all processing power can do it overfit model without experience. Even if we put AI into an army of robots running around and experiencing things, there are still scaling limits to encoding and communicating knowledge and understanding. Human organizations are a great example of the scaling limits of intelligence.
- skybrian 5y ago
  Actually no, there is almost no chance that it will be harder to transfer skills between machines than it is to raise and educate a child. Something very weird would have to happen for it to be that hard.
  - shkkmo 5y ago
    Is it? Humans have been optimizing for information transmission between individuals for a long time. Not just our hardware and neural nets have been optimized, but the structure of the information itself. It is quite possible that you need such a large set of compex interactions to develop intelligence that intelligence can only develop in reality. It is possible that the process for doing this inherently inolves a chaotic system that makes duplicating or paralleling that development impossible. I don't see any reason why these artificial intelligences with divergent chaotically emergent capabilities would have to be able to teach each other things as easily as we do. I've long been of the opinion culture / "social software" is vital for producing knowledge for humans and that this social software has been fine tuned and iterated upon for a very long time (and possibly co-developed with the evolution of our brain hardware so that both are co-specialized for each other). This software is likely very specialized to our brain structures and might not generalize well to other intelligence substrates. Thus even if we do develop neural network structres that can run on silicone and match human level intelligence, those entities would still need a lot of interaction with reality to develop their own culture that would allow them understand the world at the level we do. Edit: Is it even possible to efficiently run the necessary theories of human mind on non-human brains to operate effectively in our epistemic landscape? Communication requires huge amounts of inference using theories of mind to resolve ambiguity and add implicit information. It is possible that we can "recompile" human knowledge to run on other types of intelligences, but I suspect this will be far more difficult that commonly expected. Just look at how hard (nigh impossible) it is today to even boostrap a new general purpose computer without depending on the output other general purpose computers. We simply do not understand cognition or semantics well enough to know the answers to these questions adequately or with any degree of certainty.
choeger 5y ago
> Intelligence is not a single dimension, so “smarter than humans” is a meaningless concept. That's not the point. We can map any multi-dimensional form of intelligence into a single dimension with any suitable norm. It just matters that for any dimension we can quantify intelligence in the sense that something more intelligent is imaginable. > Humans do not have general purpose minds, and neither will AIs. That does not matter at all. Then a superhuman intelligence is not "general", so what? It's still wastly more capable then any human. > Emulation of human thinking in other media will be constrained by cost. But what is the cost constraints? Human brains are constrained by mechanical considerations. How efficient could a computerized AI become? > Dimensions of intelligence are not infinite. Again, what's the argument? The number of atoms in the universe is not infinite, either (probably). > Intelligences are only one factor in progress. That's not the problem of the super AI.
- mindcrime 5y ago
  FWIW, I agree with your sentiments pretty much 100%. I was going to say more or less the same thing(s) but didn't feel motivated enough to do more of a "point by point" response. Glad to see somebody chipping in with very similar viewpoints.
- jbay808 5y ago
  Reminds me of the brilliant "On the Impossibility of Supersized Machines" [1]. We show that it is not only implausible that machines will ever exceed human size, but in fact impossible. [...] The term “supersized machine” implies a machine that has crossed some threshold, which is often denoted “human-level largeness.” However, it is not clear what “human-level largeness” could refer to. Has a machine achieved human-level largeness if it has the same height as the average human? If it has the same volume? The same weight? [...] When one begins to consider these questions, one quickly concludes that there are an infinite number of metrics that could be used to measure largeness, and that people who speak of “supersized machines” do not have a particular metric in mind. Surely, then, any future machine will be larger than humans on some metrics and smaller than humans on others, just as they are today. [1] https://arxiv.org/abs/1703.10987 https://arxiv.org/abs/1703.10987
- deleted 5y ago
croes 5y ago
Bold claim. At the moment we don't fully understand the brain and intelligence. Even what makes the difference a living and a non living accumulation of atoms is unknown. But this doesn't not mean we couldn't create a superhuman AI in the future or by accident. I think we don't have the capabilities to do so but this isn't a 100% certaincy more like 99%
- arisAlexis 5y ago
  "the problem with the world is that the intelligent are full of doubt and the idiots cocksure". Applies directly to the bold title. Doesn't matter if we are all annihilated of course nobody will remember.
lern_too_spel 5y ago
Old article. Previous discussion: https://news.ycombinator.com/item?id=14205042 https://news.ycombinator.com/item?id=14205042
- mcswell 5y ago
  Looks like you're being ignored, or at least no one (until me) has replied. I guess people would rather hear themselves think than look at what other people said a few years ago. But thanks for bringing up this link!
YetAnotherNick 5y ago
There is no reason we know that brain emulation is not possible. Even if one second takes 100 years in processing now, there is no reason to think that there is fundamental limit in how many processor you could connect. And for superhuman it just needs to do one thing well(which computers already does) than human and match humans in everything. I am not saying it would be done but I haven't seen any argument on why this exact argument is "impossible".
- otabdeveloper4 5y ago
  > There is no reason we know that brain emulation is not possible. Well, neither is there any reason we know that it's possible. You're just demonstrating your apriori biases ("the brain is just a computer"), nothing more.
  - YetAnotherNick 5y ago
    We have reason to believe it's possible because we know we can simulate a single neuron. It's just the sheer complexity of full brain we can't scan and efficiently simulate.
    - otabdeveloper4 5y ago
      There's absolutely no reason to believe that the brain can be reduced to a collection of neurons. It's what we want to believe, because it's simpler and makes us feel in control; but in real nature there's intelligence without neurons, and the brain needs much more than neurons to do its thing. P.S. If we presume that human intelligence is a random processes, then there's no reason to think that this random process can create another that has more information complexity than the original. It makes no sense mathematically.
- qsort 5y ago
  It's not impossible in principle, but: - we don't understand enough about the brain to emulate it. - we don't know how to model the brain. - even assuming we had the required scientific knowledge, there would likely be major engineering problems. So, in other words, it's not impossible in the same sense FTL travel is not impossible, our scientific understanding does not categorically exclude it, but making predictions as to when it will happen is kind of silly, because it will likely require several fundamentally unpredictable scientific breakthroughs.
habitue 5y ago
Other commenters have mentioned that these particular points have been refuted elsewhere, so I won't rehash that. But I do think it's interesting that the author frames the "orthodox" position as the one taking AI alignment seriously. That's pretty big progress and I think the folks who have been sounding the alarm[0] about AGI should give themselves a pat on the back that the zeitgeist has shifted so much. The arguments from the anti-alarmism camp have been so consistently weak, I think the best explanation for their motivation is still something like "All this AGI stuff sounds so sci-fi. It can't be a real concern". It just goes to show how strongly we bias our expectations of the future on our what we've personally experienced. You really have to try hard to escape that default mode of thinking. [0] By sounding the alarm I mean saying things like "This is at least a possible scenario with huge error bars on when it might happen, maybe at least some people should be thinking hard about this, rather than nobody"
- batavian 5y ago
  It doesn't really even sound sci-fi, or rather, it would make for some bad sci-fi. What we are at risk of, is having deterministic model artifacts trigger complex unintended consequences from being fetishized as decision-making agents. Considering how few CS and AI people I've met who even know that e.g. pragmatics are a critical component of sentence meaning, I sense the most proximate AI dystopia is a lot closer to this latter scenario. Conceptually the possibility of an AGI does present a real challenge (& one worth grappling); in practice, we haven't made any progress really toward realizing that problem. We've become good at feeding "large" datasets into algorithms that are many orders of magnitude less advanced than a lizard brain. Then we get to the issue that our large datasets are only large when thinking of classical statistical sample sizes. They nonetheless are lacking in redundancy, interactivity, stakes or any centered perspective such that there is no chance of anything resembling intelligence, agency, or plain old intentions arising.
dennisblue 5y ago
Why was this garbage even published?
Analemma_ 5y ago
It’s really frightening that “AI risk isn’t real” is rapidly congealing into a tribe-signaling _political_ stance that generates reliable terrible articles like this one and the one in the Washington Post a few weeks ago. You’d think with Covid still killing thousands of people a day the media would more able to recognize that fault mode and not repeat the “Covid isn’t a problem, because saying that it’s a problem would be racist” error of February 2020, but apparently not.
bjornsing 5y ago
Only skimmed the bullet points, but one that’s missing and seems important to me is: * The marginal utility of intelligence may not be so great. If you are already really smart, is there any point to being smarter? Most interesting problems are NP-complete. If you are really smart you can solve a big NP-complete problem. But how much smarter do you need to be to solve a slightly bigger (and therefore more interesting) problem? Well, twice as smart or something. Being just a little bit smarter gets you nothing. So (in this case) the marginal utility of intelligence approaches zero as your AI gets more intelligent.
- kbelder 5y ago
  Being 10% smarter puts you just marginally ahead of other people; and a competitive marginal advantage is often major. So I'd say there's great utility in an AI being smarter than any human... and then the returns diminish.
a_square_peg 5y ago
Interesting discussions. Perhaps it may be easier to look at it from another angle. Current blend of ML/DL algorithms are not AI. I think the tendency of giving aspirational names to software algorithms have created a lot of confusion. Perhaps AI will be one day possible but if so, it's probably safe to say that it won't be anything based on the current ML/DL paradigm. Fun example, Google DeepMind is yet able to learn math: https://www.zdnet.com/article/ai-aint-no-a-student-deepmind-flunks-high-school-math/ https://www.zdnet.com/article/ai-aint-no-a-student-deepmind-...
- blamestross 5y ago
  We keep making glorified optic nerves and spinal reflexes and calling it a brain.
arisAlexis 5y ago
Someone must have written at some point "the myth of a pandemic". Its incomprehensibly stupid to dismiss any kind of threat as a myth unless they prove that it's impossible by laws of physics. And still then there would be room for error. Sad thst a respectable magazine posts such naive articles. I dismiss it from the title before even reading it.
- Barrin92 5y ago
  nobody who is sane would have made that claim about a pandemic because we have empirical evidence for the existence of pandemics. In fact in contrast to superintelligence we have a pretty clear model and definition of what a pandemic entails. The notion that we ought to take everything seriously that is 'not impossible by the laws of physics' is entirely stupid. Because virtually anything is possible, the question is how likely it is, and what evidence we have for it that is quantifiable and meaningful. Or are you currently prepping for an alien invasion? Myth is the absolutely correct label for 'superhuman AI' fantasies, because the difference between a real threat and a myth is that the former is grounded in empiricism while the latter is a product of psychology and storytelling. And superhuman AI is a very old story, it's simply the Golem myth. The religious idea that humans, in their hubris, create a tool beyond their control. It's Goethe's The Sorcerer's Apprentice and it's The Terminator. Also, Kevin Kelly the author of the article is the founder of the magazine.
  - arisAlexis 5y ago
    Even worse that Kelly wrote it. Have you even read the statistics that majority of AI experts believe it is possible? Are you an AI expert? Shouldn't we listen to the scientists on the subject instead of doing our own research like in Covid? The question was rhetorical. Yes we should listen to the scientists and not dismiss what they say is a probable scenario that could lead to annihilation. I also suggest to you and anyone that hasn't read the book "Superintelligence" to read it because the arguments there are solid.
m0zg 5y ago
It's all fun and games until you lose your job to a robot. Likewise superhuman AI already exists in some specific tasks. Machines, for instance, have been superior to humans in facial recognition and matching for 15 years now, if not longer. The speed at which things are improving in other tasks would be considered ludicrous in any other field. Yet people still persist in thinking that it has to be some sentient robot in order to have a huge societal impact. We are on the precipice of many labor intensive but relatively low skill professions going away entirely, and yet people still stress out about synthetic intelligence they can have a conversation with, because it sounds "scarier" to the readership of Wired, which never was anywhere near these imperiled jobs and whose livelihood does not depend on them. But a program that can write crappy-yet-plausible Python code? Now you got their clicks and attention.

30 points by conanxin 5y ago

> Hyper-intelligent algorithms are not going to take over the world for these five reasons. I agree. Just like a nuclear power plant will never meltdown, because they are extremely well designed with large safety buffers. And just like a pandemic will never be started by lab experiments, because they use extreme safety measures when they work with viruses. We can always just unplug the AI or disconnect it from the Internet, or just not give it a survival instinct, right?

gilch 5y ago
Pretty sure this is sarcasm, as in, the lies are so transparent they're humorous, and this was on purpose. For those who didn't notice, nuclear plants have, in fact, melted down, more than once. Biosafety level 4 labs have, in fact, had leaks of pathogens before, more than once. An agent AI will tend to develop a survival instinct by default, even if we don't give it one, because it is a convergent instrumental goal. Thus, it would be motivated to not simply let you yank it, would fight you if you tried, and may even take preemptive precautions.

tech-historian 5y ago

Who is the author Kevin Kelly and why does he think he has a greater hold on this issue than luminaries like Ray Kurzweil?

mindcrime 5y ago
To the first half of your question: https://en.wikipedia.org/wiki/Kevin_Kelly_(editor) https://en.wikipedia.org/wiki/Kevin_Kelly_(editor) To the second half: Dunno. Good question.
cowpig 5y ago
Appeals to authority are not convincing
- mindcrime 5y ago
  This isn't a formal debate, and authority is useful especially as a "first pass" filter, or in terms of establishing Bayesian priors. The key, IMO, is not to assume that the mere existence of authority guarantees that someone is right. But in either case, it's certainly valid to ask the question "why should I take Kevin Kelly seriously on this topic?"
  - cowpig 5y ago
    I don't agree in this context. The author wrote an entire article about his position, and dismissing it entirely because some "luminary" disagrees is not an interesting response. OP didn't even bother to point out how or why Kurzweil purportedly disagrees.
    - tech-historian 5y ago
      I thought it was obvious. Clearly it was not. Ray Kurzweil is a futurist who has spent multiple decades researching this topic and has written multiple books on it. His bio speaks for itself. https://en.wikipedia.org/wiki/Ray_Kurzweil https://en.wikipedia.org/wiki/Ray_Kurzweil
    - mindcrime 5y ago
      I guess I interpreted tech-historian's post a little differently than you. Likewise, I don't support dismissing Kelly's article out-of-hand, based on Kurzweil's position. But I do think it's fair to raise the issue in terms of "why should one take Kevin Kelly seriously on this?"
AnimalMuppet 5y ago
Why is Ray Kurzweil considered a luminary in this context? Sure, he talks a lot. He makes grand statements. Is there any reason to take him seriously in this context? He sure seems to believe, and to want to believe. But actual evidence that he's going to be right any time soon? That seems to be missing.
- enord 5y ago
  Everybody wants to be the Fox Mulder of their story.
- tech-historian 5y ago
  It's clear you're not familiar with Kurzweil's work in AI (or his background in general). He's written multiple books on the topic over multiple decades, including the seminal "The Singularity is Near". He's devoted his life to the study of this field. https://en.wikipedia.org/wiki/Ray_Kurzweil https://en.wikipedia.org/wiki/Ray_Kurzweil https://en.wikipedia.org/wiki/The_Singularity_Is_Near https://en.wikipedia.org/wiki/The_Singularity_Is_Near
  - zaybqp 5y ago
    Thank you for enlightening us with the new knowledge of this wonderful luminary. I have learned so much from these URLs. Do you have any more recommendations on what we should read next? I find myself deeply enthralled by his fascinating claims and I want to seek out more of his teachings.
  - 6gvONxR4sf7o 5y ago
    He's also written books on how much of your diet should be fat, and similar topics. I've never been able to tell if he's worth taking seriously. My instincts say no. https://en.wikipedia.org/wiki/The_10%25_Solution_for_a_Healthy_Life https://en.wikipedia.org/wiki/The_10%25_Solution_for_a_Healt... https://en.wikipedia.org/wiki/Fantastic_Voyage:_Live_Long_Enough_to_Live_Forever https://en.wikipedia.org/wiki/Fantastic_Voyage:_Live_Long_En...
  - AnimalMuppet 5y ago
aaaaaaaaaaab 5y ago
>luminaries like Ray Kurzweil This is satire, right?

deleted 5y ago

[deleted]

sxp 5y ago

tl;dr: Humans have souls which machines will never have. The core of the article is this quote "I will extend that further to claim that the only way to get a very human-like thought process is to run the computation on very human-like wet tissue. That also means that very big, complex artificial intelligences run on dry silicon will produce big, complex, unhuman-like minds." which is just the boring old argument against replicating human thinking. Similarly, there is an extreme amount of pro-human arrogance in the article "At the core of the notion of a superhuman intelligence — particularly the view that this intelligence will keep improving itself — is the essential belief that intelligence has an infinite scale. I find no evidence for this.... So the question is, where is the limit of intelligence? We tend to believe that the limit is way beyond us, way “above” us, as we are “above” an ant. Setting aside the recurring problem of a single dimension, what evidence do we have that the limit is not us? Why can’t we be at the maximum?" Humans are just a bunch of atoms that randomly learned to think over a few billion years. Given how fast we can improve tech via directed means, we should be able to replicate what nature has done in thousands, hundreds, or tens of years.

cowpig 5y ago
The word "soul" does not appear once in the article
DuskStar 5y ago
> That also means that very big, complex artificial intelligences run on dry silicon will produce big, complex, unhuman-like minds. Incidentally, this is exactly what the AI superintelligence people are terrified of, and what the paperclip problem is meant to illustrate. https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html

mindcrime 5y ago

Ugh. This is a pretty bad article. And don't get me wrong... I'm no "AI Alarmist." But to me, this article is equally as useless as the various pronouncements it seems to be trying to refute. There's a lot of verbiage here, but no cohesive, coherent argument to support the thesis that "Superhuman AI is just a myth" (which is to say, "Superhuman AI is not possible"). The reality is, there are a lot of unknowns in this space, and to dismiss the possibility of an "evil super-intelligence" out-of-hand is, IMO, every bit as silly as buying into the "OMG, AI is unleashing the demon" fear-mongering. I just think it's too early to know for sure the answers to some of these questions, like: 1. Is AGI possible at all (for whatever definition of "AGI" you want to use)? 2. If the answer to (1) is "yes", then what about ASI (Artificial Super Intelligence) - is that possible? 3. If (1) and (2) are both "yes", then is it possible to create an ASI that will either be benevolent, or controllable, or both? And so on... we just don't know yet. And that in and of itself arguably suggests the exercise of a certain measure of caution.

enord 5y ago
You're making an argument that is already in TFA. It's in addition -and only partially related to- the main point which is: The arguments that are commonly used to demonstrate the possibility or threat of "ASI" are made from claims presented _without evidence_. This will invalidate fantastical accelerationist woo until such evidence is presented, and -in the current state of the art research within relevant fields- it's not even certain we can know what would constitute such evidence _in principle_.
- mindcrime 5y ago
  I assume you're referring to this part: If the expectation of a superhuman AI takeover is built on five key assumptions that have no basis in evidence, then this idea is more akin to a religious belief — a myth My problem is, even this is wrong. I do not find the "five key assumptions" he cites to be the foundation of beliefs that a dangerous "Superhuman AI" could emerge. So to me, this article simply fails right from the jump. The thing he is arguing for and/or against just don't, collectively, constitute a well-developed argument for the overall thesis. Not in my opinion anyway. To go through them one by one: Artificial intelligence is already getting smarter than us, at an exponential rate. I don't know anybody making this claim, nor is it apparent that belief in this is necessary to think that dangerous "Superhuman AI" could be a problem in the future. I guess it comes down to what you think Kelly meant by "soon" in the preceding sentence. We’ll make AIs into a general purpose intelligence, like our own. It's the "like our own" part of this that I have a problem with. Yes, I think many people believe that we will create some manner of "general intelligence", but I also believe that a lot of researchers acknowledge that it may not be 100% "human like". But this may be my own bias showing, as I personally harp a lot on the importance of the distinction between "human level" intelligence and "human like" intelligence. We can make human intelligence in silicon. This is probably the most sound one of all, but here's the thing... saying "there is no evidence for this" is simply wrong. To hold to this is to completely ignore all progress made building AI in silicon to date. And while we don't yet have "general intelligence", I don't see how one can just ignore incremental progress completely and say "there's nothing there". I would argue that the null hypothesis should probably be that general intelligence on silicon is possible that that the burden of proof is on the people arguing the opposite. Also, not everybody thinking about / working on AI is focused on the "in silicon" approach. The majority? Yes. But I'm sure there are a handful of people working on ideas related to DNA computing, chemical computing, growing synthetic brain matter, etc. Intelligence can be expanded without limit. To me, this may or not not be true, but it's completely irrelevant to the thesis at hand. The question isn't "can machine intelligence be unbounded?", but rather "can machine intelligence exceed human intelligence?" Once we have exploding superintelligence it can solve most of our problems. Again, this strikes me as mostly irrelevant to the question at hand. So yes, this article raises some interesting points, but I don't see it accomplishing what it seems to purport to do.

blamestross 5y ago

The bottleneck for intelligence is experience. Running on silicon doesn't allow an intelligence to engage with reality significantly faster than a human can. There are some benefits due to access to data and processing power, but the data isn't useful without the context of experience and all processing power can do it overfit model without experience. Even if we put AI into an army of robots running around and experiencing things, there are still scaling limits to encoding and communicating knowledge and understanding. Human organizations are a great example of the scaling limits of intelligence.

skybrian 5y ago
Actually no, there is almost no chance that it will be harder to transfer skills between machines than it is to raise and educate a child. Something very weird would have to happen for it to be that hard.
- shkkmo 5y ago
  Is it? Humans have been optimizing for information transmission between individuals for a long time. Not just our hardware and neural nets have been optimized, but the structure of the information itself. It is quite possible that you need such a large set of compex interactions to develop intelligence that intelligence can only develop in reality. It is possible that the process for doing this inherently inolves a chaotic system that makes duplicating or paralleling that development impossible. I don't see any reason why these artificial intelligences with divergent chaotically emergent capabilities would have to be able to teach each other things as easily as we do. I've long been of the opinion culture / "social software" is vital for producing knowledge for humans and that this social software has been fine tuned and iterated upon for a very long time (and possibly co-developed with the evolution of our brain hardware so that both are co-specialized for each other). This software is likely very specialized to our brain structures and might not generalize well to other intelligence substrates. Thus even if we do develop neural network structres that can run on silicone and match human level intelligence, those entities would still need a lot of interaction with reality to develop their own culture that would allow them understand the world at the level we do. Edit: Is it even possible to efficiently run the necessary theories of human mind on non-human brains to operate effectively in our epistemic landscape? Communication requires huge amounts of inference using theories of mind to resolve ambiguity and add implicit information. It is possible that we can "recompile" human knowledge to run on other types of intelligences, but I suspect this will be far more difficult that commonly expected. Just look at how hard (nigh impossible) it is today to even boostrap a new general purpose computer without depending on the output other general purpose computers. We simply do not understand cognition or semantics well enough to know the answers to these questions adequately or with any degree of certainty.

choeger 5y ago

> Intelligence is not a single dimension, so “smarter than humans” is a meaningless concept. That's not the point. We can map any multi-dimensional form of intelligence into a single dimension with any suitable norm. It just matters that for any dimension we can quantify intelligence in the sense that something more intelligent is imaginable. > Humans do not have general purpose minds, and neither will AIs. That does not matter at all. Then a superhuman intelligence is not "general", so what? It's still wastly more capable then any human. > Emulation of human thinking in other media will be constrained by cost. But what is the cost constraints? Human brains are constrained by mechanical considerations. How efficient could a computerized AI become? > Dimensions of intelligence are not infinite. Again, what's the argument? The number of atoms in the universe is not infinite, either (probably). > Intelligences are only one factor in progress. That's not the problem of the super AI.

mindcrime 5y ago
FWIW, I agree with your sentiments pretty much 100%. I was going to say more or less the same thing(s) but didn't feel motivated enough to do more of a "point by point" response. Glad to see somebody chipping in with very similar viewpoints.
jbay808 5y ago
Reminds me of the brilliant "On the Impossibility of Supersized Machines" [1]. We show that it is not only implausible that machines will ever exceed human size, but in fact impossible. [...] The term “supersized machine” implies a machine that has crossed some threshold, which is often denoted “human-level largeness.” However, it is not clear what “human-level largeness” could refer to. Has a machine achieved human-level largeness if it has the same height as the average human? If it has the same volume? The same weight? [...] When one begins to consider these questions, one quickly concludes that there are an infinite number of metrics that could be used to measure largeness, and that people who speak of “supersized machines” do not have a particular metric in mind. Surely, then, any future machine will be larger than humans on some metrics and smaller than humans on others, just as they are today. [1] https://arxiv.org/abs/1703.10987 https://arxiv.org/abs/1703.10987
deleted 5y ago

croes 5y ago

Bold claim. At the moment we don't fully understand the brain and intelligence. Even what makes the difference a living and a non living accumulation of atoms is unknown. But this doesn't not mean we couldn't create a superhuman AI in the future or by accident. I think we don't have the capabilities to do so but this isn't a 100% certaincy more like 99%

arisAlexis 5y ago
"the problem with the world is that the intelligent are full of doubt and the idiots cocksure". Applies directly to the bold title. Doesn't matter if we are all annihilated of course nobody will remember.

lern_too_spel 5y ago

Old article. Previous discussion: https://news.ycombinator.com/item?id=14205042 https://news.ycombinator.com/item?id=14205042

mcswell 5y ago
Looks like you're being ignored, or at least no one (until me) has replied. I guess people would rather hear themselves think than look at what other people said a few years ago. But thanks for bringing up this link!

YetAnotherNick 5y ago

There is no reason we know that brain emulation is not possible. Even if one second takes 100 years in processing now, there is no reason to think that there is fundamental limit in how many processor you could connect. And for superhuman it just needs to do one thing well(which computers already does) than human and match humans in everything. I am not saying it would be done but I haven't seen any argument on why this exact argument is "impossible".

otabdeveloper4 5y ago
> There is no reason we know that brain emulation is not possible. Well, neither is there any reason we know that it's possible. You're just demonstrating your apriori biases ("the brain is just a computer"), nothing more.
- YetAnotherNick 5y ago
  We have reason to believe it's possible because we know we can simulate a single neuron. It's just the sheer complexity of full brain we can't scan and efficiently simulate.
  - otabdeveloper4 5y ago
    There's absolutely no reason to believe that the brain can be reduced to a collection of neurons. It's what we want to believe, because it's simpler and makes us feel in control; but in real nature there's intelligence without neurons, and the brain needs much more than neurons to do its thing. P.S. If we presume that human intelligence is a random processes, then there's no reason to think that this random process can create another that has more information complexity than the original. It makes no sense mathematically.
qsort 5y ago
It's not impossible in principle, but: - we don't understand enough about the brain to emulate it. - we don't know how to model the brain. - even assuming we had the required scientific knowledge, there would likely be major engineering problems. So, in other words, it's not impossible in the same sense FTL travel is not impossible, our scientific understanding does not categorically exclude it, but making predictions as to when it will happen is kind of silly, because it will likely require several fundamentally unpredictable scientific breakthroughs.

habitue 5y ago

Other commenters have mentioned that these particular points have been refuted elsewhere, so I won't rehash that. But I do think it's interesting that the author frames the "orthodox" position as the one taking AI alignment seriously. That's pretty big progress and I think the folks who have been sounding the alarm[0] about AGI should give themselves a pat on the back that the zeitgeist has shifted so much. The arguments from the anti-alarmism camp have been so consistently weak, I think the best explanation for their motivation is still something like "All this AGI stuff sounds so sci-fi. It can't be a real concern". It just goes to show how strongly we bias our expectations of the future on our what we've personally experienced. You really have to try hard to escape that default mode of thinking. [0] By sounding the alarm I mean saying things like "This is at least a possible scenario with huge error bars on when it might happen, maybe at least some people should be thinking hard about this, rather than nobody"

batavian 5y ago
It doesn't really even sound sci-fi, or rather, it would make for some bad sci-fi. What we are at risk of, is having deterministic model artifacts trigger complex unintended consequences from being fetishized as decision-making agents. Considering how few CS and AI people I've met who even know that e.g. pragmatics are a critical component of sentence meaning, I sense the most proximate AI dystopia is a lot closer to this latter scenario. Conceptually the possibility of an AGI does present a real challenge (& one worth grappling); in practice, we haven't made any progress really toward realizing that problem. We've become good at feeding "large" datasets into algorithms that are many orders of magnitude less advanced than a lizard brain. Then we get to the issue that our large datasets are only large when thinking of classical statistical sample sizes. They nonetheless are lacking in redundancy, interactivity, stakes or any centered perspective such that there is no chance of anything resembling intelligence, agency, or plain old intentions arising.

dennisblue 5y ago

Why was this garbage even published?

Analemma_ 5y ago

It’s really frightening that “AI risk isn’t real” is rapidly congealing into a tribe-signaling _political_ stance that generates reliable terrible articles like this one and the one in the Washington Post a few weeks ago. You’d think with Covid still killing thousands of people a day the media would more able to recognize that fault mode and not repeat the “Covid isn’t a problem, because saying that it’s a problem would be racist” error of February 2020, but apparently not.

bjornsing 5y ago

Only skimmed the bullet points, but one that’s missing and seems important to me is: * The marginal utility of intelligence may not be so great. If you are already really smart, is there any point to being smarter? Most interesting problems are NP-complete. If you are really smart you can solve a big NP-complete problem. But how much smarter do you need to be to solve a slightly bigger (and therefore more interesting) problem? Well, twice as smart or something. Being just a little bit smarter gets you nothing. So (in this case) the marginal utility of intelligence approaches zero as your AI gets more intelligent.

kbelder 5y ago
Being 10% smarter puts you just marginally ahead of other people; and a competitive marginal advantage is often major. So I'd say there's great utility in an AI being smarter than any human... and then the returns diminish.

a_square_peg 5y ago

Interesting discussions. Perhaps it may be easier to look at it from another angle. Current blend of ML/DL algorithms are not AI. I think the tendency of giving aspirational names to software algorithms have created a lot of confusion. Perhaps AI will be one day possible but if so, it's probably safe to say that it won't be anything based on the current ML/DL paradigm. Fun example, Google DeepMind is yet able to learn math: https://www.zdnet.com/article/ai-aint-no-a-student-deepmind-flunks-high-school-math/ https://www.zdnet.com/article/ai-aint-no-a-student-deepmind-...

blamestross 5y ago
We keep making glorified optic nerves and spinal reflexes and calling it a brain.

arisAlexis 5y ago

Someone must have written at some point "the myth of a pandemic". Its incomprehensibly stupid to dismiss any kind of threat as a myth unless they prove that it's impossible by laws of physics. And still then there would be room for error. Sad thst a respectable magazine posts such naive articles. I dismiss it from the title before even reading it.

Barrin92 5y ago
nobody who is sane would have made that claim about a pandemic because we have empirical evidence for the existence of pandemics. In fact in contrast to superintelligence we have a pretty clear model and definition of what a pandemic entails. The notion that we ought to take everything seriously that is 'not impossible by the laws of physics' is entirely stupid. Because virtually anything is possible, the question is how likely it is, and what evidence we have for it that is quantifiable and meaningful. Or are you currently prepping for an alien invasion? Myth is the absolutely correct label for 'superhuman AI' fantasies, because the difference between a real threat and a myth is that the former is grounded in empiricism while the latter is a product of psychology and storytelling. And superhuman AI is a very old story, it's simply the Golem myth. The religious idea that humans, in their hubris, create a tool beyond their control. It's Goethe's The Sorcerer's Apprentice and it's The Terminator. Also, Kevin Kelly the author of the article is the founder of the magazine.
- arisAlexis 5y ago
  Even worse that Kelly wrote it. Have you even read the statistics that majority of AI experts believe it is possible? Are you an AI expert? Shouldn't we listen to the scientists on the subject instead of doing our own research like in Covid? The question was rhetorical. Yes we should listen to the scientists and not dismiss what they say is a probable scenario that could lead to annihilation. I also suggest to you and anyone that hasn't read the book "Superintelligence" to read it because the arguments there are solid.

m0zg 5y ago

It's all fun and games until you lose your job to a robot. Likewise superhuman AI already exists in some specific tasks. Machines, for instance, have been superior to humans in facial recognition and matching for 15 years now, if not longer. The speed at which things are improving in other tasks would be considered ludicrous in any other field. Yet people still persist in thinking that it has to be some sentient robot in order to have a huge societal impact. We are on the precipice of many labor intensive but relatively low skill professions going away entirely, and yet people still stress out about synthetic intelligence they can have a conversation with, because it sounds "scarier" to the readership of Wired, which never was anywhere near these imperiled jobs and whose livelihood does not depend on them. But a program that can write crappy-yet-plausible Python code? Now you got their clicks and attention.