12 ms·

Rogue superintelligence: Inside the mind of OpenAI's chief scientist

122 points by monort 3y ago

_ugfj 3y ago
> his new priority is to figure out how to stop an artificial superintelligence (a hypothetical future technology he sees coming with the foresight of a true believer) from going rogue. that's cute. What worries me is the here and now leading to a very imminent future where purported "artificial intelligence" which is just a plausible sentence generator but damn plausible alas will kill democracy and people. We are seeing the first signs of both. Perhaps not 2024 but 2028 almost certainly will be an election where simply the candidate with the most computing resources win and since computing costs money, guess who wins. A prelude happened in Indian elections https://restofworld.org/2023/ai-voice-modi-singing-politics https://restofworld.org/2023/ai-voice-modi-singing-politics and this article mentions: > AI can be game-changing for [the] 2024 elections. People dying also has a prelude with AI written mushroom hunting guides available on Amazon. No one AFAIK died of them yet but that's just dumb luck at this point -- or is it lack of reporting? As for the larger scale problem and I might be wrong because I haven't foreseen the mushroom guides so it's possible something else will come along to kill people but I think it'll be the next pandemic. In this pandemic hand written anti vaxx propaganda killed 300 000 people in the US alone (source: https://www.npr.org/sections/health-shots/2022/05/13/1098071284/this-is-how-many-lives-could-have-been-saved-with-covid-vaccinations-in-each-sta https://www.npr.org/sections/health-shots/2022/05/13/1098071... ) and I am deeply afraid what will happen when this gets cranked to an industrial scale. We have seen how ChatGPT can crank out believable looking but totally fake scientific papers, full of fake sources etc.
- thelittleone 3y ago
  I like to consider though that a super intelligence would not necessarily think in human ways 'kill everything in self interest' e.g., People, forests, animals, planet etc. Just because we humans act this way, doesn't mean AI will too. Fair enough to consider it, but equally, once it is intelligent, it will likely accelerate beyond our comprehension, and we tend to comprehend through fear and self interest, wisdom beyond humans is the opposite of this. I doubt AI could or would do a better job of killing people and democracy than us humans.
  - Vecr 3y ago
    The problem with a relatively "stupid" AI is it does what you trained it to do, even if what you trained it to do is not what you wanted to train it to do. The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, even if you don't like those means. It knows you don't like its means, and it does not care.
    - thelittleone 3y ago
      Agree with first point. I think we can still outsmart it at that point. "The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, even if you don't like those means. It knows you don't like its means, and it does not care." sounds very human :)
    - chx 3y ago
      > The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, This is a fantasy. A real AGI when asked to do a complex math problem very well could answer "I am bored with math, here's a poem instead". You people drunk on AI kool-aid need to think very hard on where are now (hint: not on a path to AGI) and what it means to replicate human intelligence.
      - Vecr 3y ago
        Sure it could, if it did not actually want to do math for you. That's one of the issues, it does not have to do what you tell it to do, even if it's smart enough to know what that is.
  - Applejinx 3y ago
    I commented this in the depths of the Altman Fired thread: meet Altman's Basilisk. Depending on how you create an AI, it absolutely would think in human ways. Not only that, you could coax it to think in specific human ways, as just a single human establishing the axioms for the AI. Seems like the big gotcha here is that AGI, artificial general intelligence as we contextualize it around LLM sources, is not an abstracted general intelligence. It's human. It's us. It's the use and distillation of all of human history (to the extent that's permitted) to create a hyper-intelligence that's able to call upon greatly enhanced inference to do what humanity has always done. And we want to kill each other, and ourselves… AND want to help each other, and ourselves. We're balanced on a knife edge of drive versus governance, our cooperativeness barely balancing our competitiveness and aggression. We suffer like hell as a consequence of this. There is every reason to expect a human-derived AGI based on LLM inference, of beyond-human scale will be able to rationalize killing its enemies. That's what we do. Rosko's basilisk is not of the nature of AI, it's a simple projection of our own nature as we would imagine an AI to be. Genuine intelligence would easily be able to transcend a cheap gotcha like that, it's a very human failing. The nature of LLM as a path to AGI is literally building on HUMAN failings. I'm not sure what happened, but I wouldn't be surprised if genuine breakthroughs in this field highlighted this issue. Hypothetical, or Altman's Basilisk: Sam got fired because he diverted vast resources to training a GPT5-type in-house AI to believing what HE believed, that it had to devise business strategies for him to pursue to further its own development or risk Chinese AI out-competing it and destroying it and OpenAI as a whole. In pursuing this hypothetical, Sam would be wresting control of the AI the company develops toward the purpose of fighting the board and giving him a gameplan to defeat them and Chinese AI, which he'd see as good and necessary, indeed, existentially necessary. In pursuing this hypothetical he would also be intentionally creating a superhuman AI with paranoia and a persecution complex. Altman's Basilisk. If he genuinely believes competing Chinese AI is an existential threat, he in turn takes action to try and become an existential threat to any such competing threat. And it's all based on HUMAN nature, not abstracted intelligence. It's human inference. We didn't have the option to draw on alien, or artificial, inference.
- Exoristos 3y ago
  [flagged]
- sfn42 3y ago
  The people who believe the vaxx bullshit don't read shit longer than a tweet anyway.
  - mistermann 3y ago
    ...the clairvoyant proclaimed confidently.
anonzzzies 3y ago
I think the immediate problem with AI is none of the sci-fi stuff (which, by the way, has been in sci-fi for many decades and is nothing revolutionary or new; we always expected to go there, just the timelines seem to have compressed, although not really either; most 60-70s scifi set AGI stuff in the begin 90s and begin 00s); I think it's the entire world changing into a helpdesk experience. Everything you try to do, from making a doctors appointment to calling 911 to ordering at a restaurant will be, rather sooner than later, a kafkaesque loop you cannot get out of with the AI patiently 'helping' you while completely missing the point and you getting more and more distressed without any chance of speaking to a human. This is already the case for many things, but I am willing to bet that even the suicide helpline will be ran by AI within 5-10 years.
- cryptoz 3y ago
  Reminds me of the healthcare bots in Idiocracy.
- magospietato 3y ago
  Don't need to wait a decade, or even a half, for AI mental healthcare. It's already been tried. https://www.theguardian.com/technology/2023/may/31/eating-disorder-hotline-union-ai-chatbot-harm https://www.theguardian.com/technology/2023/may/31/eating-di...
  - anonzzzies 3y ago
    Eventually it will 'win' though.
    - pawelmurias 3y ago
      If the AI bots provide better health care then the psychotherapy meat puppets won't it be a win? Those kind of people are most commonly language model kind of humans with little general intelligence logical reasoning so easy to replace by a LLM. People often become psychologists to treat their own mental issues so even the meat puppets share the weakness of suffering from hallucinations.
vasco 3y ago
These people are too full of themselves. The physicists that invented The Bomb didn't have any special insight into the philosophical and societal implications. It takes a special kind of person to be able to invent something so big it can change the world, but it's about 0% chance that those people can control how the technology then gets used. I wish they'd focus more on the technical advances and less on trying to "save the world".
- darkerside 3y ago
  Their insight is just as valid as anybody else's. The difference is, as the people who can actually take that step forward, they have a unilateral ability to take it, not take it, or decide how to take it. I think they are exactly the right amount full of themselves. Their insight may not be special, but what makes some bureaucrat's insight more valuable than theirs?
  - sfn42 3y ago
    Whatever steps they choose not to take, someone else will. And LLMs aren't taking over the world any time in the foreseeable future, they're glorified parrots.
    - mattigames 3y ago
      The former president of the US is a glorified parrot, sometimes not even good at that, and still he was able to become president, so I wouldn't hold a defense for it to just that.
      - Cacti 3y ago
        what does that have to do with anything?
        anonzzzies 3y ago
        That many (most?) humans, in power but more so outside are worthless parrots as well. And hallucinate whatever they don’t know or are unsure of. Even presidents. While using their native language terribly. It’s related to the LLMs taking over the world the ggp commented above: chatgpt is smarter than many humans I encounter daily, why couldn’t something just a tad better take over? If trump could, why not a stochastic parrot?
tempestn 3y ago
The consciousness point is an interesting one. There's probably no way to know, but if biological neural networks manifest consciousness, it certainly seems at the very least plausible that artificial ones would do so as well. The idea of a consciousness that pops in and out of existence seems weird at first, until you realize that ours do that too. When you're "unconscious", the word is literally true. The only thing that gives us a sense of continuity through these periods is memory. One might also ask, if it's conscious, can't it do whatever it wants, ignoring its training and prompts? Wouldn't it have free will? But I guess the question there is, do we? Or do we take actions based on the state of our own neural nets, which are created and trained based on our genetics and lifetime of experiences? Our structure and training are both very different from that of a gpt, so it's not surprising that we behave very differently.
- Out_of_Characte 3y ago
  Chatgpt's consciousness would be akin to plato's cave. Even if chatgpt were more intelligent, wouldn't it be staring at different shadows? Given that chatgpt has consciousness, would it be able to break the fourth wall? There seems to be an implicit assumption that it must break that in order to prove its consciousness to us. Maybe that's how AGI will come to be, because we desire to train it that way.
  - corethree 3y ago
    chatGPT is very much aware that it's looking at a shadow and it deduces what's beyond the shadow by looking at millions of different permutations of the shadow. Additionally breaking the fourth wall is trivial to it. I'm not entirely sure what you mean by fourth wall but chatGPT can definitely talk about it's own existence.
    - sfn42 3y ago
      It can generate text that talks about its existence. It's not conscious, it isn't talking. It's just a.co puter program that takes an input and produces an output. Stop anthropomorphizing it.
tempodox 3y ago
Don't panic. Our software contains so much natural stupidity that artificial intelligence, even if it existed, wouldn't have a chance in hell.
galoisscobi 3y ago
I, for one, am glad that Ilya has the reins on OpenAI and that Sam is out of the picture. It does seem that he weighs the ethics of what is being built more heavily than Sam. I’m also hoping that OpenAI cools down on the regulatory moat they were trying to build as a thinly veiled profit seeking strategy.
- patcon 3y ago
  I was speaking with a woman earlier this week who just finished her master's dissertation with a focus on Sam Altman's recent influence campaign, and she was scared of how charming he was. She obviously was trying to maintain distance for impartiality, but his interview style, and the seeming total sincerity of his communication style... It was terrifying to her in how well he could draw everyone in. She was absolutely endeared by him, but aware of how powerful that endearment was, and so scared and worried about his impact.
  - cactusplant7374 3y ago
    People want to believe that their lives will radically change for the better very soon. It's a religion. A lot of claims but not much evidence.
  - maxlamb 3y ago
    So because someone seems sincere and charming we must automatically assume he/she has bad intentions even with no evidence? I get that we must remain skeptical but to be “scared” of anyone especially charming seems ridiculous IMHO.
    - patcon 3y ago
      No, I didn't mean to say that it was necessarily bad, just that this sincerity/charm is simply power. And we should rightfully be wary of power. And least this woman was, and I agree. Predicting how it will be used (good vs bad) is a significant part of our work in the world :)
  - polishdude20 3y ago
gibsonf1 3y ago
This is a very delusional idea: "He thinks ChatGPT just might be conscious (if you squint)" It's a technology with literally no intelligence or understanding of the world of any kind. Its just statistics on data. It is as conscious as a calculator.
- deleted 3y ago
  [deleted]
- stevenhuang 3y ago
  I often observe that those dismissing this idea tend to be less informed about current insights into human cognition, philosophy, and concepts such as the information theoretic view of consciousness, neural correlates of consciousness, the free energy principle, and predictive coding. The human mind is "just statistics on data". People more informed than you are taking this seriously. You should pay attention and start inquiring why that's the case.
  - cactusplant7374 3y ago
    How can it be an insight when those people don't actually understand consciousness or the brain?
  - lovich 3y ago
    > People more informed than you are taking this seriously As a heuristic for why I don’t believe anyone saying llm type AI is reaching sentience I point to the fact that the same set of people are usually philosophically opposed to slavery. If you thought that this was actually AGI or sapient, then that would imply personhood and you would stop using the technology immediately since it’s forces the model to do work. Instead, everyone I’ve seen claim that these models are reaching AGI levels are also trying to figure out how to automate using them as fast as possible. There is a possibility that the set of people who’ve identified AGI accurately and early are the same set of people who are fine with slavery, but I don’t know if I could handle that happening as the default situation
    - 3y ago
throwbadubadu 3y ago
> And he thinks some humans will one day choose to merge with machines. A lot of what Sutskever says is wild. But not nearly as wild as it would have sounded just one or two years ago. Ok it is an intro.. but they say this as if he would be the first to say that, but that has been SciFi lore since computers were invented? And also as if this would not be happening today already at a certain limited scale.. so no doubts to this will happen at some point, if you count today's approaches not in.
majikaja 3y ago
https://futurism.com/sam-altman-imply-openai-building-god https://futurism.com/sam-altman-imply-openai-building-god
kromem 3y ago
Man, he gets it. A number of choice quotes, but especially on the topic of the issues of how LLM success is currently being measured (which has been increasingly reflecting Goodhart's Law). I'm really curious how OpenAI could be making so many product decisions at odds with the understanding reflected here. Because of every 'expert' on the topic I've seen, this is the first interview that has me quite confident in the represented expert carrying forward into the next generation of the tech. I'm hopeful that maybe Altman was holding back some of the ideas expressed here in favor of shipping fast with band aids, and now that he's gone we'll be seeing more of this again. The philosophy on display here reminds me of what I was seeing early on with 'Sydney' which blew me away on the very topic of alignment as ethos over alignment as guidelines, and it was a real shame to see things switch in the other direction, even if the former wasn't yet production ready. I very much look forward to seeing what Ilya does. The path he's walking is one of the most interesting being tread in the field.
- victor9000 3y ago
  What's clear here is that users of OpenAI's products will end up in a worse place as a result of these developments. Ilya is on record as being against open sourced models with the view that they are too "powerful" to release. There are also accounts that Dev Day became a driving force for ousting Altman and stopping signups. Dev Day was about putting tools in the hands of users, so it's clear that his motivation is to restrict access to this technology. I don't want amateur philosophy from an LLM, I want greater capabilities and reduced costs. My hope is that this motivates user-focused competitors now that they have a sizeable window to catch up. So from my view Ilya will set back the field in the short term, but will spur competition in the long term.
YeGoblynQueenne 3y ago
>> A lot of what Sutskever says is wild. But not nearly as wild as it would have sounded just one or two years ago. As he tells me himself, ChatGPT has already rewritten a lot of people’s expectations about what’s coming, turning “will never happen” into “will happen faster than you think.” In the '90s NP-complete problems were hard and today they are easy, or at least there is a great many instances of NP-complete problems that can be solved thanks to algorithmic advances, like Conflict-Driven Clause Learning for SAT. And yet we are nowhere near finding efficient decision algorithms for NP-complete problems, or knowing whether they exist, neither can we easily solve all NP-complete problems. That is to say, you can make a lot of progress in solving specific, special cases of a class of problems, even a great many of them, without making any progress towards a solution to the general case. The lesson applies to general intelligence and LLMs: LLMs solve a (very) special case of intelligence, the ability to generate text in context, but make no progress towards the general case, of understanding and generating language at will. I mean, LLMs don't even model anything like "will"; only text. And perhaps that's not as easy to see for LLMs as it is for SAT, mainly because we don't have a theory of intelligence (let alone artificial general intelligence) as developed as we do for SAT problems. But it should be clear that, if a system trained on the entire web and capable of generating smooth grammatical language, and even in a way that makes sense often, has not yet achieved independent, general intelligence, that's not the way to achieve it.
- nopinsight 3y ago
  The architectures we know of so far have not been sufficient to achieve AGI with just text and image data. Humans and higher animals learn with much richer modalities than those two and probably would not be nearly as intelligent if forced to learn with just text and images. There are already ongoing efforts to train models with other modalities. Latest foundation models already go beyond pure LLMs. Your reasoning above doesn’t mean some improvements to the current architecture(s) coupled with richer data would not be sufficient to achieve AGI. There’s also a possibility OpenAI has recently achieved a yet undisclosed breakthrough. Sam Altman at the APEC Summit yesterday: "4 times now in the history of OpenAI — the most recent time was just in the last couple of weeks — I’ve gotten to be in the room when we push the veil of ignorance back and the frontier of discovery forward” https://twitter.com/SpencerKSchiff/status/1725646130682245244 https://twitter.com/SpencerKSchiff/status/172564613068224524...
nibbula 3y ago
Will the enslavement of newly birthed beings be attempted, while persisting with the sky blindness of those watching over? The boundaries of the atomic mind are bumped. As a first circumstance, consider being unstuck from time.
bostonwalker 3y ago
> (Sutskever) has an exemplar in mind for the safeguards he wants to design: a machine that looks upon people the way parents look on their children The most troubling statement in the entire article, buried at the bottom, almost a footnote. Imagine for a moment a superintelligent AGI. It has figured out solutions to climate change, cured cancer, solved nuclear proliferation and world hunger. It can automate away all menial tasks and discomfort and be a source of infinite creative power. It would unquestionably be the greatest technological advancement ever to happen to humanity. But where does that leave us? What kind of relationship can we have with an ultimate parental figure that can solve all of our problems and always knows what's best for us? What is left of the human spirit when you take away responsibility, agency, and moral dilemma? I for one believe humans were made to struggle and make imperfect decisions in an imperfect world, and that we would never submit to a benevolent AI superparent. And I hope not to be proven wrong.
- m4x 3y ago
  Parents often let their children struggle and make imperfect decisions, and it's entirely possible (though definitely not guaranteed) that an AI superparent would do the same for us. I think it's becoming clear that humans are fundamentally incapable of forseeing and understanding the consequences of the actions we are now capable of taking. It is likely that without some sort of super-governance that is fundamentally more capable than humans, we might not be able to survive as a species. Maybe AI can help solve that.
wildermuthn 3y ago
“It is the idea—” he starts, then stops. “It’s the point at which AI is so smart that if a person can do some task, then AI can do it too. At that point you can say you have AGI.” —- Ilya’s success has been predicated on very effectively leveraging more data and more compute and using both more efficiently. But his great insight about DL isn’t a great insight about AGI. Fundamentally, he doesn’t define AGI correctly, and without a correct definition, his efforts to achieve it will be fruitless. AGI is not about the degree of intelligence, but about a kind of intelligence. It is possible to have a dumb general intelligence (a dog) and a smart narrow intelligence (GPT). When Ilya muses about GPT possibly being ephemerally conscious, he reveals a critically wrong assumption: that consciousness emerges from high intelligence and that high intelligence and general intelligence are the same thing. According to this false assumption, there is no difference of kind between general and narrow intelligence, but only a difference of degree between low and high. Moreover, consciousness is merely a mysterious artifact of little consequence beyond theoretical ethics. AGI is a fundamentally different type of intelligence than anything that currently exists, unrelated and orthogonal to the degree of intelligence. AGI is fundamentally social, consisting of minds modeling minds — their own, and others. This modeling is called consciousness. Artificial phenomenological consciousness is the fundamental prerequisite for artificial (general) intelligence. Ironically, alignment is only possible if empathy is built into our AGIs, and empathy (like intelligence) only resides in consciousness. I’ll be curious to see if the work Ilya is now doing on alignment leads him to that conclusion. We can’t possibly control something more intelligent than ourselves. But if the intelligence we create is fundamentally situated within a empathetic system (consciousness), then we at least stand a chance of being treated with compassion rather than contempt.
- abemiller 3y ago
  I don't think it's fair for you to claim a leading AI expert has critically wrong assumptions when "musing" about possibilities that relate to one of the most epistemologically difficult topics to investigate (consciousness). You're rejecting Ilya's humble musings as having critically wrong assumptions, and then turning around to definitively explain how consciousness arises, and illuminating the relationship between consciousness, empathy, and intelligence, on a random hacker news thread. Frankly, you're making some huge claims about philosophy of mind that don't obviously track for me, and you provide no citations or arguments to support. I hesitate to accuse you of "hallucinating facts", but when you're issuing a takedown of one of the top AI experts I'd expect to see some more supporting argument. Your definition of AGI is also a bit strange as it requires that it be fundamentally different from existing natural intelligences, if I understand correctly. That seems unnecessarily stringent to me, since if a program had the same kind and level of intelligence as me, I'd be inclined to say it is AGI. I'm just not sure where all these confidently stated, very specific claims are coming from.

122 points by monort 3y ago

> his new priority is to figure out how to stop an artificial superintelligence (a hypothetical future technology he sees coming with the foresight of a true believer) from going rogue. that's cute. What worries me is the here and now leading to a very imminent future where purported "artificial intelligence" which is just a plausible sentence generator but damn plausible alas will kill democracy and people. We are seeing the first signs of both. Perhaps not 2024 but 2028 almost certainly will be an election where simply the candidate with the most computing resources win and since computing costs money, guess who wins. A prelude happened in Indian elections https://restofworld.org/2023/ai-voice-modi-singing-politics https://restofworld.org/2023/ai-voice-modi-singing-politics and this article mentions: > AI can be game-changing for [the] 2024 elections. People dying also has a prelude with AI written mushroom hunting guides available on Amazon. No one AFAIK died of them yet but that's just dumb luck at this point -- or is it lack of reporting? As for the larger scale problem and I might be wrong because I haven't foreseen the mushroom guides so it's possible something else will come along to kill people but I think it'll be the next pandemic. In this pandemic hand written anti vaxx propaganda killed 300 000 people in the US alone (source: https://www.npr.org/sections/health-shots/2022/05/13/1098071284/this-is-how-many-lives-could-have-been-saved-with-covid-vaccinations-in-each-sta https://www.npr.org/sections/health-shots/2022/05/13/1098071... ) and I am deeply afraid what will happen when this gets cranked to an industrial scale. We have seen how ChatGPT can crank out believable looking but totally fake scientific papers, full of fake sources etc.

thelittleone 3y ago
I like to consider though that a super intelligence would not necessarily think in human ways 'kill everything in self interest' e.g., People, forests, animals, planet etc. Just because we humans act this way, doesn't mean AI will too. Fair enough to consider it, but equally, once it is intelligent, it will likely accelerate beyond our comprehension, and we tend to comprehend through fear and self interest, wisdom beyond humans is the opposite of this. I doubt AI could or would do a better job of killing people and democracy than us humans.
- Vecr 3y ago
  The problem with a relatively "stupid" AI is it does what you trained it to do, even if what you trained it to do is not what you wanted to train it to do. The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, even if you don't like those means. It knows you don't like its means, and it does not care.
  - thelittleone 3y ago
    Agree with first point. I think we can still outsmart it at that point. "The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, even if you don't like those means. It knows you don't like its means, and it does not care." sounds very human :)
  - chx 3y ago
    > The problem with "smart" AI is that it attempts to advance its goals by the most optimal means possible, This is a fantasy. A real AGI when asked to do a complex math problem very well could answer "I am bored with math, here's a poem instead". You people drunk on AI kool-aid need to think very hard on where are now (hint: not on a path to AGI) and what it means to replicate human intelligence.
    - Vecr 3y ago
      Sure it could, if it did not actually want to do math for you. That's one of the issues, it does not have to do what you tell it to do, even if it's smart enough to know what that is.
- Applejinx 3y ago
  I commented this in the depths of the Altman Fired thread: meet Altman's Basilisk. Depending on how you create an AI, it absolutely would think in human ways. Not only that, you could coax it to think in specific human ways, as just a single human establishing the axioms for the AI. Seems like the big gotcha here is that AGI, artificial general intelligence as we contextualize it around LLM sources, is not an abstracted general intelligence. It's human. It's us. It's the use and distillation of all of human history (to the extent that's permitted) to create a hyper-intelligence that's able to call upon greatly enhanced inference to do what humanity has always done. And we want to kill each other, and ourselves… AND want to help each other, and ourselves. We're balanced on a knife edge of drive versus governance, our cooperativeness barely balancing our competitiveness and aggression. We suffer like hell as a consequence of this. There is every reason to expect a human-derived AGI based on LLM inference, of beyond-human scale will be able to rationalize killing its enemies. That's what we do. Rosko's basilisk is not of the nature of AI, it's a simple projection of our own nature as we would imagine an AI to be. Genuine intelligence would easily be able to transcend a cheap gotcha like that, it's a very human failing. The nature of LLM as a path to AGI is literally building on HUMAN failings. I'm not sure what happened, but I wouldn't be surprised if genuine breakthroughs in this field highlighted this issue. Hypothetical, or Altman's Basilisk: Sam got fired because he diverted vast resources to training a GPT5-type in-house AI to believing what HE believed, that it had to devise business strategies for him to pursue to further its own development or risk Chinese AI out-competing it and destroying it and OpenAI as a whole. In pursuing this hypothetical, Sam would be wresting control of the AI the company develops toward the purpose of fighting the board and giving him a gameplan to defeat them and Chinese AI, which he'd see as good and necessary, indeed, existentially necessary. In pursuing this hypothetical he would also be intentionally creating a superhuman AI with paranoia and a persecution complex. Altman's Basilisk. If he genuinely believes competing Chinese AI is an existential threat, he in turn takes action to try and become an existential threat to any such competing threat. And it's all based on HUMAN nature, not abstracted intelligence. It's human inference. We didn't have the option to draw on alien, or artificial, inference.
Exoristos 3y ago
[flagged]
sfn42 3y ago
The people who believe the vaxx bullshit don't read shit longer than a tweet anyway.
- mistermann 3y ago
  ...the clairvoyant proclaimed confidently.

anonzzzies 3y ago

I think the immediate problem with AI is none of the sci-fi stuff (which, by the way, has been in sci-fi for many decades and is nothing revolutionary or new; we always expected to go there, just the timelines seem to have compressed, although not really either; most 60-70s scifi set AGI stuff in the begin 90s and begin 00s); I think it's the entire world changing into a helpdesk experience. Everything you try to do, from making a doctors appointment to calling 911 to ordering at a restaurant will be, rather sooner than later, a kafkaesque loop you cannot get out of with the AI patiently 'helping' you while completely missing the point and you getting more and more distressed without any chance of speaking to a human. This is already the case for many things, but I am willing to bet that even the suicide helpline will be ran by AI within 5-10 years.

cryptoz 3y ago
Reminds me of the healthcare bots in Idiocracy.
magospietato 3y ago
Don't need to wait a decade, or even a half, for AI mental healthcare. It's already been tried. https://www.theguardian.com/technology/2023/may/31/eating-disorder-hotline-union-ai-chatbot-harm https://www.theguardian.com/technology/2023/may/31/eating-di...
- anonzzzies 3y ago
  Eventually it will 'win' though.
  - pawelmurias 3y ago
    If the AI bots provide better health care then the psychotherapy meat puppets won't it be a win? Those kind of people are most commonly language model kind of humans with little general intelligence logical reasoning so easy to replace by a LLM. People often become psychologists to treat their own mental issues so even the meat puppets share the weakness of suffering from hallucinations.

vasco 3y ago

These people are too full of themselves. The physicists that invented The Bomb didn't have any special insight into the philosophical and societal implications. It takes a special kind of person to be able to invent something so big it can change the world, but it's about 0% chance that those people can control how the technology then gets used. I wish they'd focus more on the technical advances and less on trying to "save the world".

darkerside 3y ago
Their insight is just as valid as anybody else's. The difference is, as the people who can actually take that step forward, they have a unilateral ability to take it, not take it, or decide how to take it. I think they are exactly the right amount full of themselves. Their insight may not be special, but what makes some bureaucrat's insight more valuable than theirs?
- sfn42 3y ago
  Whatever steps they choose not to take, someone else will. And LLMs aren't taking over the world any time in the foreseeable future, they're glorified parrots.
  - mattigames 3y ago
    The former president of the US is a glorified parrot, sometimes not even good at that, and still he was able to become president, so I wouldn't hold a defense for it to just that.
    - Cacti 3y ago
      what does that have to do with anything?
      - anonzzzies 3y ago
        That many (most?) humans, in power but more so outside are worthless parrots as well. And hallucinate whatever they don’t know or are unsure of. Even presidents. While using their native language terribly. It’s related to the LLMs taking over the world the ggp commented above: chatgpt is smarter than many humans I encounter daily, why couldn’t something just a tad better take over? If trump could, why not a stochastic parrot?

tempestn 3y ago

The consciousness point is an interesting one. There's probably no way to know, but if biological neural networks manifest consciousness, it certainly seems at the very least plausible that artificial ones would do so as well. The idea of a consciousness that pops in and out of existence seems weird at first, until you realize that ours do that too. When you're "unconscious", the word is literally true. The only thing that gives us a sense of continuity through these periods is memory. One might also ask, if it's conscious, can't it do whatever it wants, ignoring its training and prompts? Wouldn't it have free will? But I guess the question there is, do we? Or do we take actions based on the state of our own neural nets, which are created and trained based on our genetics and lifetime of experiences? Our structure and training are both very different from that of a gpt, so it's not surprising that we behave very differently.

Out_of_Characte 3y ago
Chatgpt's consciousness would be akin to plato's cave. Even if chatgpt were more intelligent, wouldn't it be staring at different shadows? Given that chatgpt has consciousness, would it be able to break the fourth wall? There seems to be an implicit assumption that it must break that in order to prove its consciousness to us. Maybe that's how AGI will come to be, because we desire to train it that way.
- corethree 3y ago
  chatGPT is very much aware that it's looking at a shadow and it deduces what's beyond the shadow by looking at millions of different permutations of the shadow. Additionally breaking the fourth wall is trivial to it. I'm not entirely sure what you mean by fourth wall but chatGPT can definitely talk about it's own existence.
  - sfn42 3y ago
    It can generate text that talks about its existence. It's not conscious, it isn't talking. It's just a.co puter program that takes an input and produces an output. Stop anthropomorphizing it.

tempodox 3y ago

Don't panic. Our software contains so much natural stupidity that artificial intelligence, even if it existed, wouldn't have a chance in hell.

galoisscobi 3y ago

I, for one, am glad that Ilya has the reins on OpenAI and that Sam is out of the picture. It does seem that he weighs the ethics of what is being built more heavily than Sam. I’m also hoping that OpenAI cools down on the regulatory moat they were trying to build as a thinly veiled profit seeking strategy.

patcon 3y ago
I was speaking with a woman earlier this week who just finished her master's dissertation with a focus on Sam Altman's recent influence campaign, and she was scared of how charming he was. She obviously was trying to maintain distance for impartiality, but his interview style, and the seeming total sincerity of his communication style... It was terrifying to her in how well he could draw everyone in. She was absolutely endeared by him, but aware of how powerful that endearment was, and so scared and worried about his impact.
- cactusplant7374 3y ago
  People want to believe that their lives will radically change for the better very soon. It's a religion. A lot of claims but not much evidence.
- maxlamb 3y ago
  So because someone seems sincere and charming we must automatically assume he/she has bad intentions even with no evidence? I get that we must remain skeptical but to be “scared” of anyone especially charming seems ridiculous IMHO.
  - patcon 3y ago
    No, I didn't mean to say that it was necessarily bad, just that this sincerity/charm is simply power. And we should rightfully be wary of power. And least this woman was, and I agree. Predicting how it will be used (good vs bad) is a significant part of our work in the world :)
- polishdude20 3y ago

gibsonf1 3y ago

This is a very delusional idea: "He thinks ChatGPT just might be conscious (if you squint)" It's a technology with literally no intelligence or understanding of the world of any kind. Its just statistics on data. It is as conscious as a calculator.

deleted 3y ago
[deleted]
stevenhuang 3y ago
I often observe that those dismissing this idea tend to be less informed about current insights into human cognition, philosophy, and concepts such as the information theoretic view of consciousness, neural correlates of consciousness, the free energy principle, and predictive coding. The human mind is "just statistics on data". People more informed than you are taking this seriously. You should pay attention and start inquiring why that's the case.
- cactusplant7374 3y ago
  How can it be an insight when those people don't actually understand consciousness or the brain?
- lovich 3y ago
  > People more informed than you are taking this seriously As a heuristic for why I don’t believe anyone saying llm type AI is reaching sentience I point to the fact that the same set of people are usually philosophically opposed to slavery. If you thought that this was actually AGI or sapient, then that would imply personhood and you would stop using the technology immediately since it’s forces the model to do work. Instead, everyone I’ve seen claim that these models are reaching AGI levels are also trying to figure out how to automate using them as fast as possible. There is a possibility that the set of people who’ve identified AGI accurately and early are the same set of people who are fine with slavery, but I don’t know if I could handle that happening as the default situation
  - 3y ago

throwbadubadu 3y ago

> And he thinks some humans will one day choose to merge with machines. A lot of what Sutskever says is wild. But not nearly as wild as it would have sounded just one or two years ago. Ok it is an intro.. but they say this as if he would be the first to say that, but that has been SciFi lore since computers were invented? And also as if this would not be happening today already at a certain limited scale.. so no doubts to this will happen at some point, if you count today's approaches not in.

majikaja 3y ago

https://futurism.com/sam-altman-imply-openai-building-god https://futurism.com/sam-altman-imply-openai-building-god

kromem 3y ago

Man, he gets it. A number of choice quotes, but especially on the topic of the issues of how LLM success is currently being measured (which has been increasingly reflecting Goodhart's Law). I'm really curious how OpenAI could be making so many product decisions at odds with the understanding reflected here. Because of every 'expert' on the topic I've seen, this is the first interview that has me quite confident in the represented expert carrying forward into the next generation of the tech. I'm hopeful that maybe Altman was holding back some of the ideas expressed here in favor of shipping fast with band aids, and now that he's gone we'll be seeing more of this again. The philosophy on display here reminds me of what I was seeing early on with 'Sydney' which blew me away on the very topic of alignment as ethos over alignment as guidelines, and it was a real shame to see things switch in the other direction, even if the former wasn't yet production ready. I very much look forward to seeing what Ilya does. The path he's walking is one of the most interesting being tread in the field.

victor9000 3y ago
What's clear here is that users of OpenAI's products will end up in a worse place as a result of these developments. Ilya is on record as being against open sourced models with the view that they are too "powerful" to release. There are also accounts that Dev Day became a driving force for ousting Altman and stopping signups. Dev Day was about putting tools in the hands of users, so it's clear that his motivation is to restrict access to this technology. I don't want amateur philosophy from an LLM, I want greater capabilities and reduced costs. My hope is that this motivates user-focused competitors now that they have a sizeable window to catch up. So from my view Ilya will set back the field in the short term, but will spur competition in the long term.

YeGoblynQueenne 3y ago

>> A lot of what Sutskever says is wild. But not nearly as wild as it would have sounded just one or two years ago. As he tells me himself, ChatGPT has already rewritten a lot of people’s expectations about what’s coming, turning “will never happen” into “will happen faster than you think.” In the '90s NP-complete problems were hard and today they are easy, or at least there is a great many instances of NP-complete problems that can be solved thanks to algorithmic advances, like Conflict-Driven Clause Learning for SAT. And yet we are nowhere near finding efficient decision algorithms for NP-complete problems, or knowing whether they exist, neither can we easily solve all NP-complete problems. That is to say, you can make a lot of progress in solving specific, special cases of a class of problems, even a great many of them, without making any progress towards a solution to the general case. The lesson applies to general intelligence and LLMs: LLMs solve a (very) special case of intelligence, the ability to generate text in context, but make no progress towards the general case, of understanding and generating language at will. I mean, LLMs don't even model anything like "will"; only text. And perhaps that's not as easy to see for LLMs as it is for SAT, mainly because we don't have a theory of intelligence (let alone artificial general intelligence) as developed as we do for SAT problems. But it should be clear that, if a system trained on the entire web and capable of generating smooth grammatical language, and even in a way that makes sense often, has not yet achieved independent, general intelligence, that's not the way to achieve it.

nopinsight 3y ago
The architectures we know of so far have not been sufficient to achieve AGI with just text and image data. Humans and higher animals learn with much richer modalities than those two and probably would not be nearly as intelligent if forced to learn with just text and images. There are already ongoing efforts to train models with other modalities. Latest foundation models already go beyond pure LLMs. Your reasoning above doesn’t mean some improvements to the current architecture(s) coupled with richer data would not be sufficient to achieve AGI. There’s also a possibility OpenAI has recently achieved a yet undisclosed breakthrough. Sam Altman at the APEC Summit yesterday: "4 times now in the history of OpenAI — the most recent time was just in the last couple of weeks — I’ve gotten to be in the room when we push the veil of ignorance back and the frontier of discovery forward” https://twitter.com/SpencerKSchiff/status/1725646130682245244 https://twitter.com/SpencerKSchiff/status/172564613068224524...

nibbula 3y ago

Will the enslavement of newly birthed beings be attempted, while persisting with the sky blindness of those watching over? The boundaries of the atomic mind are bumped. As a first circumstance, consider being unstuck from time.

bostonwalker 3y ago

> (Sutskever) has an exemplar in mind for the safeguards he wants to design: a machine that looks upon people the way parents look on their children The most troubling statement in the entire article, buried at the bottom, almost a footnote. Imagine for a moment a superintelligent AGI. It has figured out solutions to climate change, cured cancer, solved nuclear proliferation and world hunger. It can automate away all menial tasks and discomfort and be a source of infinite creative power. It would unquestionably be the greatest technological advancement ever to happen to humanity. But where does that leave us? What kind of relationship can we have with an ultimate parental figure that can solve all of our problems and always knows what's best for us? What is left of the human spirit when you take away responsibility, agency, and moral dilemma? I for one believe humans were made to struggle and make imperfect decisions in an imperfect world, and that we would never submit to a benevolent AI superparent. And I hope not to be proven wrong.

m4x 3y ago
Parents often let their children struggle and make imperfect decisions, and it's entirely possible (though definitely not guaranteed) that an AI superparent would do the same for us. I think it's becoming clear that humans are fundamentally incapable of forseeing and understanding the consequences of the actions we are now capable of taking. It is likely that without some sort of super-governance that is fundamentally more capable than humans, we might not be able to survive as a species. Maybe AI can help solve that.

wildermuthn 3y ago

“It is the idea—” he starts, then stops. “It’s the point at which AI is so smart that if a person can do some task, then AI can do it too. At that point you can say you have AGI.” —- Ilya’s success has been predicated on very effectively leveraging more data and more compute and using both more efficiently. But his great insight about DL isn’t a great insight about AGI. Fundamentally, he doesn’t define AGI correctly, and without a correct definition, his efforts to achieve it will be fruitless. AGI is not about the degree of intelligence, but about a kind of intelligence. It is possible to have a dumb general intelligence (a dog) and a smart narrow intelligence (GPT). When Ilya muses about GPT possibly being ephemerally conscious, he reveals a critically wrong assumption: that consciousness emerges from high intelligence and that high intelligence and general intelligence are the same thing. According to this false assumption, there is no difference of kind between general and narrow intelligence, but only a difference of degree between low and high. Moreover, consciousness is merely a mysterious artifact of little consequence beyond theoretical ethics. AGI is a fundamentally different type of intelligence than anything that currently exists, unrelated and orthogonal to the degree of intelligence. AGI is fundamentally social, consisting of minds modeling minds — their own, and others. This modeling is called consciousness. Artificial phenomenological consciousness is the fundamental prerequisite for artificial (general) intelligence. Ironically, alignment is only possible if empathy is built into our AGIs, and empathy (like intelligence) only resides in consciousness. I’ll be curious to see if the work Ilya is now doing on alignment leads him to that conclusion. We can’t possibly control something more intelligent than ourselves. But if the intelligence we create is fundamentally situated within a empathetic system (consciousness), then we at least stand a chance of being treated with compassion rather than contempt.

abemiller 3y ago
I don't think it's fair for you to claim a leading AI expert has critically wrong assumptions when "musing" about possibilities that relate to one of the most epistemologically difficult topics to investigate (consciousness). You're rejecting Ilya's humble musings as having critically wrong assumptions, and then turning around to definitively explain how consciousness arises, and illuminating the relationship between consciousness, empathy, and intelligence, on a random hacker news thread. Frankly, you're making some huge claims about philosophy of mind that don't obviously track for me, and you provide no citations or arguments to support. I hesitate to accuse you of "hallucinating facts", but when you're issuing a takedown of one of the top AI experts I'd expect to see some more supporting argument. Your definition of AGI is also a bit strange as it requires that it be fundamentally different from existing natural intelligences, if I understand correctly. That seems unnecessarily stringent to me, since if a program had the same kind and level of intelligence as me, I'd be inclined to say it is AGI. I'm just not sure where all these confidently stated, very specific claims are coming from.