7 ms·

Jan Leike's OpenAI departure statement

107 points by jnnnthnn 2y ago

jnnnthnn 2y ago
X.com links are currently broken on HN so I posted a screenshot instead https://x.com/janleike/status/1791498187313963308 https://x.com/janleike/status/1791498187313963308
- ctoth 2y ago
  The funny thing is I got better OCR throwing this at ChatGPT than at the actual OCR engine I use (Windows default.)
  - daniel_iversen 2y ago
    Makes sense since Windows is probably still just using the “latest” Convolutional Neural Network type algorithms for their OCR from 10+ years ago :-)
- flakiness 2y ago
  Posted a thread as a PNG! This is novel, but yeah, necessary unfortunately.
ldayley 2y ago
Thank you Jan for your work and your courage to act and speak candidly about these important topics. Open question to HN: to your knowledge/experience which AGI-building companies or projects have a culture most closely aligned with keeping safety, security, privacy, etc. as high a priority as “winning the race” on this new frontier land-grab? I’d love to find and support those teams over the teams that spend more time focused on getting investment and marketshare.
- phreeza 2y ago
  I can't find the reference right now but I seem to remember hearing Zvi Moskowitz say in a podcast that Anthropic is the most safety-concious of the big players.
- barefeg 2y ago
  From this interview https://youtu.be/Nlkk3glap_U?feature=shared https://youtu.be/Nlkk3glap_U?feature=shared it seemed Anthropic was focusing more on those topics rather than winning the race. He hinted at being “forced” by the competition to release their models.
- Atotalnoob 2y ago
  Anthropic would probably be the closest to this. AFAIK they are still for profit, but they split from OpenAI because they disagreed with the lack of safety culture from OpenAI. This is also seen in their llms which are less capable due to their safety limitations
  - JohnBrookz 2y ago
    Ironic. The product I’m building relies on Haiku. It’s brilliant, cheap and fast. Their models are very capable and much more accessible to small companies.
  - staticman2 2y ago
    Claude Opus doesn't appear to have more safety limitations than ChatGPT. The older Claude 2.1 on the other hand was so ridiculously incapable of functioning due to a safety first design I'm guessing it inspired the goodie 2 parody AI. https://www.goody2.ai/ https://www.goody2.ai/
tptacek 2y ago
So... this is relevant if you think AGI superintelligence is a thing, and not so clearly relevant otherwise?
- llamaimperative 2y ago
  What does "if you think AGI superintelligence is a thing" mean? If you believe it exists currently, could exist, will exist, or something else?
  - tptacek 2y ago
    Any of the above. Maybe, if you really want to put some unneeded extra rigor on the analysis, add "within the lifetime of anyone born in 2024". I don't want to, like, have the argument here about it. Nobody will persuade anybody of anything. But it is not a truth universally acknowledged that AGI superintelligence is actually a thing. There are other reasons to have qualms about OpenAI! They could be misleading the market about their capabilities. They could be abusing IP. They could be ignoring clear and obvious misuse cases, the same way Facebook/Meta did in South Asia. But this exit statement seems to be much more about AGI superintelligence than any of that stuff. OK, so if I don't think that's a thing, I don't have to pay attention, right?
    - llamaimperative 2y ago
      Not going to argue with it, was just trying to understand your statement :)
      - tptacek 2y ago
        Oh, sorry, I didn't think you in particular were going to, but I realized immediately that _someone_ was. :)
    - deleted 2y ago
      [deleted]
    - mitthrowaway2 2y ago
aantix 2y ago
He's convinced that AGI is an eventuality. His call for preparation makes it sound like it's near.
- tempsy 2y ago
  Hard to imagine. But if you actually believe that there's probably a bunch of ways to make millions in public markets betting on that outcome.
  - MVissers 2y ago
    How? Interesting co’s are private (openai, groq, etc). Nvidia already huge. Microsoft and Apple are more users.
    - tempsy 2y ago
      so you think if they came out and said we have created AGI there would be no changes in stock prices to any stock that has anything to do with AI even if they are already "huge"? people said NVDA and FAANG was huge 5 years ago.
    - ZiiS 2y ago
      If it economically better to hire AI then humans for most jobs, and they only need Nvidia hardare and electricity then today's Nvidia are tiny, real tiny.
- MVissers 2y ago
  Probably within 5 years. Compute growing exponentially, algorithm improvement as well, multi modalities allows for different type of data training, etc… Yeah, this shit is near. Also- Quite a dangerous experiment we’re running. And safety-first people are not at the helm anymore.
  - Aperocky 2y ago
    The premise is that LLMs are a path to AGI. I'm not convinced, you can throw all the compute (btw, it's not growing exponentially any more, we have arrived at atom scale) at it and I'm not convinced this will lead to AGI. Our rudimentary, underpowered brain is GI, now you're telling me stacking more GPU bricks will lead to AGI? If it indeed does, it would have came by now.
llamaimperative 2y ago
This won't make a dent in the logical armor of AI optimists: [ ] If you are not intimately familiar with the development of AI, your warnings on safety can be disregarded due to your basic ignorance about the development of AI [x] If you are intimately familiar with the development of AI, your warnings on safety can be disregarded due to potential conflicts of interest and koolaid drinking Unbridled optimism lives another day!
- Aperocky 2y ago
  That's an over complication, how about my naive belief that LLMs (and increasing the size of) don't lead to AGI. Not saying AGI is impossible, just think the large models and the underlying statistic model beneath are not the path.
  - mitthrowaway2 2y ago
    I think they aren't the full answer, no matter how much they're scaled up. But they may be one essential element of a working solution, and perhaps one or two brilliant insights away. I also think that some of the money being invested into the LLM craze will be directed into the search for those other brilliant insights.
  - Footkerchief 2y ago
    LLMS don't have to be smart enough to be AGI. They just have to be smart enough to create AGI. And if creating something smarter than yourself sounds crazy, remember that we were created by simpler ancestors that we now effortlessly dominate.
    - Aperocky 2y ago
      I don't disagree with the general notion, but it seem to me that LLMs being smart enough to create AGI is even more far fetched than if they are just smart enough to be AGI.
  - jononor 2y ago
    Many teams are trying to combine their ideas with LLM. Because despite their weaknesses, it seems LLMs (and related concepts such as RLFH, transformers, self-supervised learning, internet-scale datasets), have made some remarkable gains. Those team are coming from the whole spectrum of ML and AI research. And they wish to use their ideas to overcome some of the weaknesses of current day LLMs. Do you also think that none of these children can lead to AGI? Why not?
daniel_iversen 2y ago
> “Over the past few months […] we were struggling for compute OpenAI literally said they were setting aside 20% of compute to ensure alignment [1] but if you read the fine print, what they said was that they are “dedicating 20% of the compute we’ve secured ‘to date’ to this effort” (emphasis mine). So if their overall compute has increased by 10x then that 20% is suddenly 2%, right? Is OpenAI going to be responsible or is it just a mad race (modelled from the top) to “win” the AI game? [1] https://openai.com/index/introducing-superalignment/?utm_source=tldrai https://openai.com/index/introducing-superalignment/?utm_sou...
- PeterStuer 2y ago
  Alignment with whom?
  - ben_w 2y ago
    Last I heard, the concept of AI-alignment was pre-paradigmatic. This means that we're not ready to answer "with whom", because we don't know what "aligned" even means.
  - jononor 2y ago
    Anyone or anything at all is probably the first steps, which we still don't know how to do - even conceptually. But the first real target would be alignment with the creator of the AI, and the goals an values they set. That is: Alignment with humanity/society as a whole is even further away. And might even be considered out-of-scope for AI research: ensuring that the AI creator (persons/organization) is aligned with society might be considered to be in the political domain.
123yawaworht456 2y ago
"sailing against the wind" is a very apt description of hardlining Yuddite philosophy when your company's models got maybe 20% better in the past two years (original GPT4 is still the best model I've dealt with to this day), while local models got 1000% better. we should all thank G-d these people weren't around during the advent of personal computing and the internet - we'd have word filters in our fucking text processors and publishing something on the internet would require written permission from your local DEI commissar. arrogance, pure fucking hubris brought about by the incomprehensibly stupid assumption that they will get to be the stewards of this technology.
- llamaimperative 2y ago
  we should all thank G-d similar people have been generally in charge of bioengineering and nuclear development labs else we'd have... err nothing. We'd have pretty much nothing at all.
  - 123yawaworht456 2y ago
    nukes are an excellent example, actually! for how many years did the US maintain its monopoly on nuclear weapons, pray tell?
    - llamaimperative 2y ago
      Well nukes continue to be very difficult to download off the Internet or to rent access to via a web browser. I haven't seen anyone advocate that the US government needs to be the only entity able to use AI, but if you have, you knocked their socks off with that one.
      - 123yawaworht456 2y ago
        >Well nukes continue to be very difficult to download off the Internet or to rent access to via a web browser. well, the amount of damage you can inflict with an AI model is a few magnitudes lower than that of a nuclear weapon. >I haven't seen anyone advocate that the US government needs to be the only entity able to use AI, but if you have, you knocked their socks off with that one. did you miss Sam Altman's world tour last year, talking to the politicians about the ooooh so scary AI being developed by their unregulated competitors? the establishment of oversight committees and regulatory bodies in the US, from where corporate sockpuppets will presume to dictate the the terms of development and application of this technology?
ein0p 2y ago
Seems to me the management isn’t quite as high on their own supply (at least privately) and they do not believe anything even remotely resembling “AGI” is possible in the foreseeable future. Which I, and many other people in this field, do agree with.
deleted 2y ago
[deleted]
flakiness 2y ago
> "Learn to feel the AGI." This is flabbergasted statement to me but is probably the necessary attitude to push the AI/ML frontier, I guess. I feel old.
- jononor 2y ago
  This is some internal reference/joke. We on the internet don't really know what it actually means.
  - flakiness 2y ago
    That sounds like a more plausible explanation.
amenhotep 2y ago
"Learn to feel the AGI" shades of > Anticipating the arrival of this all-powerful technology, Sutskever began to behave like a spiritual leader, three employees who worked with him told us. His constant, enthusiastic refrain was “feel the AGI,” a reference to the idea that the company was on the cusp of its ultimate goal. At OpenAI’s 2022 holiday party, held at the California Academy of Sciences, Sutskever led employees in a chant: “Feel the AGI! Feel the AGI!” The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack. From the Atlantic article. He seems a serious person but it's hard to take this circle seriously.
- Aperocky 2y ago
  tbh it smells like they've brought in too much to their own hype. Nothing OpenAI have done more recently suggest to me that they are closer to the AGI. They did create a bunch of other models with different input format that are probably based off pretty much the same thing. And it looks like those other models will be much more expensive to run and burn up their runway sooner.
  - ben_w 2y ago
    > AGI Trouble is, everyone has a different interpretation of those words. Do you mean they're no closer to "highly autonomous systems that outperform humans at most economically valuable work", or do you mean no closer to "a general-purpose machine-learning based system, even if it's only IQ 85", or do you mean no closer to "a conscious and self-aware artificial mind", or do you mean "a super-intelligent system that beats all humans put together"? > And it looks like those other models will be much more expensive to run and burn up their runway sooner I thought "runway" was the metaphor for "before take-off"? They've definitely taken off now. If they were burning more money than they bring in, they could've just not lowered the prices.
    - Aperocky 2y ago
      Until they are turning a profit, I would assume they are on a runway.
nen-nomad 2y ago
It is excellent that he is speaking out. I don't believe we can create something more intelligent than us. Smarter, yes, but not more intelligent. We will always have a deeper and more nuanced understanding of reality than our artificial counterparts. But it does not mean that AI can not be a danger.
- reducesuffering 2y ago
  > We will always have a deeper and more nuanced understanding of reality than our artificial counterparts. Wishful thinking already disproven by collectives (Google and OpenAI) that are already better at understanding and acting upon reality than you or any single human intelligence. These are systems, and they can soon hollow out their biological components.
  - nen-nomad 2y ago
    I agree with you. Those models are smarter than us in specific areas. But my question is, can they understand ideology, religion, etc, and feel them like we do? Can they come up with their traditions and beliefs? Because these are the core and evolving foundations of our reality and one of the most primitive activities since the dawn of humanity, I do not see any of these models even getting close to these concepts. They parrot what we told them. They have no agency of their own. It is also a sign of hubris to think that we can create an artificial construct that is more intelligent than we are.
  - jononor 2y ago
    Do you not consider such collectives part of we (the humans)? I can understand arguing that organizations have some superinteligence. But they are still made of / run on humans - not AI/computers?
    - reducesuffering 2y ago
      They’re currently made up of humans, but more and more will be made of computers. In their human forms, they’re still a subversion of human values, doing things as a whole, because of incentives, that the individual humans wouldn’t do. But in the future, entire corporations will get whittled down to machine processes. An AGI decision maker at the top and each individual task replaced with robot or software agent.
Imnimo 2y ago
I was and am very skeptical of the "superalignment" plan (which was "build an AI that does alignment research for us, and then ask it what to do"). But it's a bad look for OpenAI if they pledged 20% of their compute for this effort and then just didn't actually give them the compute.
azinman2 2y ago
Seems very arrogant at the end to say humanity is waiting on you. It’s not like Covid—19 where the population was literally looking to scientist/gov for an active solution. AGI is something a select set of computer nerds want, but the public isn’t clamoring for. In fact the entire idea of safety/alignment/cultural shift shows the potential downside on society if it actually were achieved. Meanwhile deepmind with alpha fold etc is showing AI can help with pressing problems of humanity without AGI as the necessary first step.
- benatkin 2y ago
  > Seems very arrogant at the end to say humanity is waiting on you. It’s not like Covid—19 where the population was literally looking to scientist/gov for an active solution. AGI is something a select set of computer nerds want, but the public isn’t clamoring for. In fact the entire idea of safety/alignment/cultural shift shows the potential downside on society if it actually were achieved. It's a similar kind of waiting, with varying levels of optimism. When you say "the public isn't clamoring for" I think a lot of the clamoring over Covid—19 was a nonspecific desire to get some normalcy. With AI it's the same, with people worrying about their future careers, or those of loved ones. In either case, people want answers, whether they are enthusiastic about what the companies are doing or not. > Meanwhile deepmind with alpha fold etc is showing AI can help with pressing problems of humanity without AGI as the necessary first step. Humanity is big enough to wait on both of them.
theGnuMe 2y ago
The only sane person in AI is LeCun.
RIMR 2y ago
It really baffles me that even extremely knowledgeable insiders like this end up buying the AGI hype. LLMs are extremely novel and have lots of groundbreaking applications, but this idea that we are headed directly into fantasy sci-fi A.I. is completely bonkers. It's one thing to push this kind of hype to get people talking about A.I. if you're trying to capitalize on this space. It's something else entirely to swallow your own marketing BS as if it were gospel. But let's face it, this guy probably isn't serious, he's just spewing more hype upon departing OpenAI looking for the next tech company to hire him.

107 points by jnnnthnn 2y ago

X.com links are currently broken on HN so I posted a screenshot instead https://x.com/janleike/status/1791498187313963308 https://x.com/janleike/status/1791498187313963308

ctoth 2y ago
The funny thing is I got better OCR throwing this at ChatGPT than at the actual OCR engine I use (Windows default.)
- daniel_iversen 2y ago
  Makes sense since Windows is probably still just using the “latest” Convolutional Neural Network type algorithms for their OCR from 10+ years ago :-)
flakiness 2y ago
Posted a thread as a PNG! This is novel, but yeah, necessary unfortunately.

ldayley 2y ago

Thank you Jan for your work and your courage to act and speak candidly about these important topics. Open question to HN: to your knowledge/experience which AGI-building companies or projects have a culture most closely aligned with keeping safety, security, privacy, etc. as high a priority as “winning the race” on this new frontier land-grab? I’d love to find and support those teams over the teams that spend more time focused on getting investment and marketshare.

phreeza 2y ago
I can't find the reference right now but I seem to remember hearing Zvi Moskowitz say in a podcast that Anthropic is the most safety-concious of the big players.
barefeg 2y ago
From this interview https://youtu.be/Nlkk3glap_U?feature=shared https://youtu.be/Nlkk3glap_U?feature=shared it seemed Anthropic was focusing more on those topics rather than winning the race. He hinted at being “forced” by the competition to release their models.
Atotalnoob 2y ago
Anthropic would probably be the closest to this. AFAIK they are still for profit, but they split from OpenAI because they disagreed with the lack of safety culture from OpenAI. This is also seen in their llms which are less capable due to their safety limitations
- JohnBrookz 2y ago
  Ironic. The product I’m building relies on Haiku. It’s brilliant, cheap and fast. Their models are very capable and much more accessible to small companies.
- staticman2 2y ago
  Claude Opus doesn't appear to have more safety limitations than ChatGPT. The older Claude 2.1 on the other hand was so ridiculously incapable of functioning due to a safety first design I'm guessing it inspired the goodie 2 parody AI. https://www.goody2.ai/ https://www.goody2.ai/

tptacek 2y ago

So... this is relevant if you think AGI superintelligence is a thing, and not so clearly relevant otherwise?

llamaimperative 2y ago
What does "if you think AGI superintelligence is a thing" mean? If you believe it exists currently, could exist, will exist, or something else?
- tptacek 2y ago
  Any of the above. Maybe, if you really want to put some unneeded extra rigor on the analysis, add "within the lifetime of anyone born in 2024". I don't want to, like, have the argument here about it. Nobody will persuade anybody of anything. But it is not a truth universally acknowledged that AGI superintelligence is actually a thing. There are other reasons to have qualms about OpenAI! They could be misleading the market about their capabilities. They could be abusing IP. They could be ignoring clear and obvious misuse cases, the same way Facebook/Meta did in South Asia. But this exit statement seems to be much more about AGI superintelligence than any of that stuff. OK, so if I don't think that's a thing, I don't have to pay attention, right?
  - llamaimperative 2y ago
    Not going to argue with it, was just trying to understand your statement :)
    - tptacek 2y ago
      Oh, sorry, I didn't think you in particular were going to, but I realized immediately that _someone_ was. :)
  - deleted 2y ago
    [deleted]
  - mitthrowaway2 2y ago

aantix 2y ago

He's convinced that AGI is an eventuality. His call for preparation makes it sound like it's near.

tempsy 2y ago
Hard to imagine. But if you actually believe that there's probably a bunch of ways to make millions in public markets betting on that outcome.
- MVissers 2y ago
  How? Interesting co’s are private (openai, groq, etc). Nvidia already huge. Microsoft and Apple are more users.
  - tempsy 2y ago
    so you think if they came out and said we have created AGI there would be no changes in stock prices to any stock that has anything to do with AI even if they are already "huge"? people said NVDA and FAANG was huge 5 years ago.
  - ZiiS 2y ago
    If it economically better to hire AI then humans for most jobs, and they only need Nvidia hardare and electricity then today's Nvidia are tiny, real tiny.
MVissers 2y ago
Probably within 5 years. Compute growing exponentially, algorithm improvement as well, multi modalities allows for different type of data training, etc… Yeah, this shit is near. Also- Quite a dangerous experiment we’re running. And safety-first people are not at the helm anymore.
- Aperocky 2y ago
  The premise is that LLMs are a path to AGI. I'm not convinced, you can throw all the compute (btw, it's not growing exponentially any more, we have arrived at atom scale) at it and I'm not convinced this will lead to AGI. Our rudimentary, underpowered brain is GI, now you're telling me stacking more GPU bricks will lead to AGI? If it indeed does, it would have came by now.

llamaimperative 2y ago

This won't make a dent in the logical armor of AI optimists: [ ] If you are not intimately familiar with the development of AI, your warnings on safety can be disregarded due to your basic ignorance about the development of AI [x] If you are intimately familiar with the development of AI, your warnings on safety can be disregarded due to potential conflicts of interest and koolaid drinking Unbridled optimism lives another day!

Aperocky 2y ago
That's an over complication, how about my naive belief that LLMs (and increasing the size of) don't lead to AGI. Not saying AGI is impossible, just think the large models and the underlying statistic model beneath are not the path.
- mitthrowaway2 2y ago
  I think they aren't the full answer, no matter how much they're scaled up. But they may be one essential element of a working solution, and perhaps one or two brilliant insights away. I also think that some of the money being invested into the LLM craze will be directed into the search for those other brilliant insights.
- Footkerchief 2y ago
  LLMS don't have to be smart enough to be AGI. They just have to be smart enough to create AGI. And if creating something smarter than yourself sounds crazy, remember that we were created by simpler ancestors that we now effortlessly dominate.
  - Aperocky 2y ago
    I don't disagree with the general notion, but it seem to me that LLMs being smart enough to create AGI is even more far fetched than if they are just smart enough to be AGI.
- jononor 2y ago
  Many teams are trying to combine their ideas with LLM. Because despite their weaknesses, it seems LLMs (and related concepts such as RLFH, transformers, self-supervised learning, internet-scale datasets), have made some remarkable gains. Those team are coming from the whole spectrum of ML and AI research. And they wish to use their ideas to overcome some of the weaknesses of current day LLMs. Do you also think that none of these children can lead to AGI? Why not?

daniel_iversen 2y ago

> “Over the past few months […] we were struggling for compute OpenAI literally said they were setting aside 20% of compute to ensure alignment [1] but if you read the fine print, what they said was that they are “dedicating 20% of the compute we’ve secured ‘to date’ to this effort” (emphasis mine). So if their overall compute has increased by 10x then that 20% is suddenly 2%, right? Is OpenAI going to be responsible or is it just a mad race (modelled from the top) to “win” the AI game? [1] https://openai.com/index/introducing-superalignment/?utm_source=tldrai https://openai.com/index/introducing-superalignment/?utm_sou...

PeterStuer 2y ago
Alignment with whom?
- ben_w 2y ago
  Last I heard, the concept of AI-alignment was pre-paradigmatic. This means that we're not ready to answer "with whom", because we don't know what "aligned" even means.
- jononor 2y ago
  Anyone or anything at all is probably the first steps, which we still don't know how to do - even conceptually. But the first real target would be alignment with the creator of the AI, and the goals an values they set. That is: Alignment with humanity/society as a whole is even further away. And might even be considered out-of-scope for AI research: ensuring that the AI creator (persons/organization) is aligned with society might be considered to be in the political domain.

123yawaworht456 2y ago

"sailing against the wind" is a very apt description of hardlining Yuddite philosophy when your company's models got maybe 20% better in the past two years (original GPT4 is still the best model I've dealt with to this day), while local models got 1000% better. we should all thank G-d these people weren't around during the advent of personal computing and the internet - we'd have word filters in our fucking text processors and publishing something on the internet would require written permission from your local DEI commissar. arrogance, pure fucking hubris brought about by the incomprehensibly stupid assumption that they will get to be the stewards of this technology.

llamaimperative 2y ago
we should all thank G-d similar people have been generally in charge of bioengineering and nuclear development labs else we'd have... err nothing. We'd have pretty much nothing at all.
- 123yawaworht456 2y ago
  nukes are an excellent example, actually! for how many years did the US maintain its monopoly on nuclear weapons, pray tell?
  - llamaimperative 2y ago
    Well nukes continue to be very difficult to download off the Internet or to rent access to via a web browser. I haven't seen anyone advocate that the US government needs to be the only entity able to use AI, but if you have, you knocked their socks off with that one.
    - 123yawaworht456 2y ago
      >Well nukes continue to be very difficult to download off the Internet or to rent access to via a web browser. well, the amount of damage you can inflict with an AI model is a few magnitudes lower than that of a nuclear weapon. >I haven't seen anyone advocate that the US government needs to be the only entity able to use AI, but if you have, you knocked their socks off with that one. did you miss Sam Altman's world tour last year, talking to the politicians about the ooooh so scary AI being developed by their unregulated competitors? the establishment of oversight committees and regulatory bodies in the US, from where corporate sockpuppets will presume to dictate the the terms of development and application of this technology?

ein0p 2y ago

Seems to me the management isn’t quite as high on their own supply (at least privately) and they do not believe anything even remotely resembling “AGI” is possible in the foreseeable future. Which I, and many other people in this field, do agree with.

deleted 2y ago

[deleted]

flakiness 2y ago

> "Learn to feel the AGI." This is flabbergasted statement to me but is probably the necessary attitude to push the AI/ML frontier, I guess. I feel old.

jononor 2y ago
This is some internal reference/joke. We on the internet don't really know what it actually means.
- flakiness 2y ago
  That sounds like a more plausible explanation.

amenhotep 2y ago

"Learn to feel the AGI" shades of > Anticipating the arrival of this all-powerful technology, Sutskever began to behave like a spiritual leader, three employees who worked with him told us. His constant, enthusiastic refrain was “feel the AGI,” a reference to the idea that the company was on the cusp of its ultimate goal. At OpenAI’s 2022 holiday party, held at the California Academy of Sciences, Sutskever led employees in a chant: “Feel the AGI! Feel the AGI!” The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack. From the Atlantic article. He seems a serious person but it's hard to take this circle seriously.

Aperocky 2y ago
tbh it smells like they've brought in too much to their own hype. Nothing OpenAI have done more recently suggest to me that they are closer to the AGI. They did create a bunch of other models with different input format that are probably based off pretty much the same thing. And it looks like those other models will be much more expensive to run and burn up their runway sooner.
- ben_w 2y ago
  > AGI Trouble is, everyone has a different interpretation of those words. Do you mean they're no closer to "highly autonomous systems that outperform humans at most economically valuable work", or do you mean no closer to "a general-purpose machine-learning based system, even if it's only IQ 85", or do you mean no closer to "a conscious and self-aware artificial mind", or do you mean "a super-intelligent system that beats all humans put together"? > And it looks like those other models will be much more expensive to run and burn up their runway sooner I thought "runway" was the metaphor for "before take-off"? They've definitely taken off now. If they were burning more money than they bring in, they could've just not lowered the prices.
  - Aperocky 2y ago
    Until they are turning a profit, I would assume they are on a runway.

nen-nomad 2y ago

It is excellent that he is speaking out. I don't believe we can create something more intelligent than us. Smarter, yes, but not more intelligent. We will always have a deeper and more nuanced understanding of reality than our artificial counterparts. But it does not mean that AI can not be a danger.

reducesuffering 2y ago
> We will always have a deeper and more nuanced understanding of reality than our artificial counterparts. Wishful thinking already disproven by collectives (Google and OpenAI) that are already better at understanding and acting upon reality than you or any single human intelligence. These are systems, and they can soon hollow out their biological components.
- nen-nomad 2y ago
  I agree with you. Those models are smarter than us in specific areas. But my question is, can they understand ideology, religion, etc, and feel them like we do? Can they come up with their traditions and beliefs? Because these are the core and evolving foundations of our reality and one of the most primitive activities since the dawn of humanity, I do not see any of these models even getting close to these concepts. They parrot what we told them. They have no agency of their own. It is also a sign of hubris to think that we can create an artificial construct that is more intelligent than we are.
- jononor 2y ago
  Do you not consider such collectives part of we (the humans)? I can understand arguing that organizations have some superinteligence. But they are still made of / run on humans - not AI/computers?
  - reducesuffering 2y ago
    They’re currently made up of humans, but more and more will be made of computers. In their human forms, they’re still a subversion of human values, doing things as a whole, because of incentives, that the individual humans wouldn’t do. But in the future, entire corporations will get whittled down to machine processes. An AGI decision maker at the top and each individual task replaced with robot or software agent.

Imnimo 2y ago

I was and am very skeptical of the "superalignment" plan (which was "build an AI that does alignment research for us, and then ask it what to do"). But it's a bad look for OpenAI if they pledged 20% of their compute for this effort and then just didn't actually give them the compute.

azinman2 2y ago

Seems very arrogant at the end to say humanity is waiting on you. It’s not like Covid—19 where the population was literally looking to scientist/gov for an active solution. AGI is something a select set of computer nerds want, but the public isn’t clamoring for. In fact the entire idea of safety/alignment/cultural shift shows the potential downside on society if it actually were achieved. Meanwhile deepmind with alpha fold etc is showing AI can help with pressing problems of humanity without AGI as the necessary first step.

benatkin 2y ago
> Seems very arrogant at the end to say humanity is waiting on you. It’s not like Covid—19 where the population was literally looking to scientist/gov for an active solution. AGI is something a select set of computer nerds want, but the public isn’t clamoring for. In fact the entire idea of safety/alignment/cultural shift shows the potential downside on society if it actually were achieved. It's a similar kind of waiting, with varying levels of optimism. When you say "the public isn't clamoring for" I think a lot of the clamoring over Covid—19 was a nonspecific desire to get some normalcy. With AI it's the same, with people worrying about their future careers, or those of loved ones. In either case, people want answers, whether they are enthusiastic about what the companies are doing or not. > Meanwhile deepmind with alpha fold etc is showing AI can help with pressing problems of humanity without AGI as the necessary first step. Humanity is big enough to wait on both of them.

theGnuMe 2y ago

The only sane person in AI is LeCun.

RIMR 2y ago

It really baffles me that even extremely knowledgeable insiders like this end up buying the AGI hype. LLMs are extremely novel and have lots of groundbreaking applications, but this idea that we are headed directly into fantasy sci-fi A.I. is completely bonkers. It's one thing to push this kind of hype to get people talking about A.I. if you're trying to capitalize on this space. It's something else entirely to swallow your own marketing BS as if it were gospel. But let's face it, this guy probably isn't serious, he's just spewing more hype upon departing OpenAI looking for the next tech company to hire him.