9 ms·

who is this and why is it important? [1] super-alignment co-lead with Ilya (who resigned yesterday) what is super alignment? [2] > We need scientific and tec

who is this and why is it important? [1] super-alignment co-lead with Ilya (who resigned yesterday) what is super alignment? [2] > We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. Our goal is to solve the core technical challenges of superintelligence alignment by 2027. [1] https://jan.leike.name/ https://jan.leike.name/ [2] https://openai.com/superalignment/ https://openai.com/superalignment/

jvanderbot 2y ago
My honest-to-god guess is that it just seemed like a needless cost center in a growing business, so there was pressure against them doing the work they wanted to do. I'm guessing, but OpenAI probably wants to start monetizing, and doesn't feel like they are going to hit a superintelligence, not really. That may have been the goal originally.
- for_i_in_range 2y ago
  I agree. Not everything has to be a conspiracy. Microsoft looked at a $10m+/year cost center, and deemed it unnecessary (which it arguably was), and snipped it.
- threetonesun 2y ago
  I assume a lot of companies want in on the AI-to-purchase pipeline. "Hey [AI] what kind of car is this?" with a response that helps you buy it at the very high end, or something as simple as "hey [AI] I need more bread, it's [brand and type]" and who it gets purchased from and how it shows up is the... bread and butter of the AI company. Super intelligent AI seems contrary to the goals of consumerist Capitalism, but maybe I'm just not smart enough to see the play there.
  - imzadi 2y ago
    I think what companies want is to replace as many human employees as possible. I don't think they really care what the consequences of that are.
- icapybara 2y ago
  This is the simplest explanation.
- mjr00 2y ago
  Yeah, OpenAI is all-in on the LLM golden goose and is much more focused on how to monetize it via embedding advertisements, continuing to provide "safety" via topic restrictions, etc., than going further down the AGI route. There's zero chance LLMs lead to AGI or superintelligence, so if that's all OpenAI is going to focus on for the next ~5 years, a group related to superintelligence alignment is unnecessary.
  - stewarts 2y ago
    How can you be so certain there is 0 chance LLMs lead to AGI/Superintelligence? Asking curiously, not something I've heard prior.
    - guhidalg 2y ago
      I'm 100% certain that I need to do more than just predict the next token to be considered intelligent. Also call me when ChatGPT can manipulate matter.
      - soulofmischief 2y ago
        > Also call me when ChatGPT can manipulate matter. You mean like PALM-E? https://palm-e.github.io/ https://palm-e.github.io/ Embodiment is the easy part.
      - mypalmike 2y ago
        Are you 100% certain that the human brain performs no language processing which is analogous to token prediction?
        stubish 2y ago
        A human brain certainly does do predictions, which is very useful to the bit that makes decisions. But how does a pure prediction engine make decisions? Make a judgement call? Analyze inconsistencies? Theorize? The best it can do is blindly follow the mob, a behavior we consider unintelligent even when done by human brains.
- llamaimperative 2y ago
  So you mean the things the charter foresaw and was intended to make impossible is in fact happening? Who could've thunk it (other than the creators of the charter and nearly anyone else with a loose grasp on how capitalism and technology interact).
- hollerith 2y ago
  >it just seemed like a needless cost center in a growing business To some of us, that sounds like, "Fire all the climate scientists because they are needless cost center distracting us from the noble goal of burning as much fossil fuel as possible."
  - mjr00 2y ago
    It's more like you started a company to research new fuel sources and hired climate scientists to evaluate the environmental impact of those new fuel sources, but during the course of the research you invented a new internal combustion engine that operates 1000% more efficiently so you pivoted your entire business toward that, removing the need for climate scientists. This is a tortured analogy, but what I'm getting at is, if OpenAI is no longer pursuing AGI/superintelligence, it doesn't need an expensive superintelligence alignment team.
    - hollerith 2y ago
      You're much more confident than I am that the researchers at OpenAI (or anyone else currently alive) are masters of their craft to such an extent that they would even be able to predict whether the next big training run they do will result in a superintelligence or not. Another way of saying the same thing is to say that the only way anyone knows that GPT-4 is not dangerously capable is that it has been deployed extensively enough by now that if it was going to harm us, it would've done so by now: not even the researchers that designed and coded-up GPT-4 or watched it during training could predict with any confidence how capable it would be. For example, everyone was quite surprised by its being able to score in the 90th decile on a bar exam. Also, even if they never produce a superintelligence, they are likely to produce insights that would make it easier for other teams to produce a superintelligence. (Since employees are free to leave OpenAI and join some other team, there is no practical way to prevent the flow of insights out of OpenAI.)
      - MichaelZuo 2y ago
        Why do they need to be 'masters of their craft' to place directional bets?
- hollerith 2y ago
  If AGI is no longer Sam Altman's goal, why was he recently trying to raise 7 trillion dollars for hardware to accelerate progress in AI?
legohead 2y ago
What is the "intelligence" behind a word predictor?
- DrSiemer 2y ago
  Fake it till you make it
TiredOfLife 2y ago
> We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. Our goal is to solve the core technical challenges of superintelligence alignment by 2027 Can somebody translate this to human?
- brandall10 2y ago
  That by 2027 they will figure out how to control Skynet so it doesn't kill us all when it awakens.
  - TiredOfLife 2y ago
    So chains and shackles or domestication?
colibri727 2y ago
What are these core technical challenges ?

jvanderbot 2y ago

My honest-to-god guess is that it just seemed like a needless cost center in a growing business, so there was pressure against them doing the work they wanted to do. I'm guessing, but OpenAI probably wants to start monetizing, and doesn't feel like they are going to hit a superintelligence, not really. That may have been the goal originally.

for_i_in_range 2y ago
I agree. Not everything has to be a conspiracy. Microsoft looked at a $10m+/year cost center, and deemed it unnecessary (which it arguably was), and snipped it.
threetonesun 2y ago
I assume a lot of companies want in on the AI-to-purchase pipeline. "Hey [AI] what kind of car is this?" with a response that helps you buy it at the very high end, or something as simple as "hey [AI] I need more bread, it's [brand and type]" and who it gets purchased from and how it shows up is the... bread and butter of the AI company. Super intelligent AI seems contrary to the goals of consumerist Capitalism, but maybe I'm just not smart enough to see the play there.
- imzadi 2y ago
  I think what companies want is to replace as many human employees as possible. I don't think they really care what the consequences of that are.
icapybara 2y ago
This is the simplest explanation.
mjr00 2y ago
Yeah, OpenAI is all-in on the LLM golden goose and is much more focused on how to monetize it via embedding advertisements, continuing to provide "safety" via topic restrictions, etc., than going further down the AGI route. There's zero chance LLMs lead to AGI or superintelligence, so if that's all OpenAI is going to focus on for the next ~5 years, a group related to superintelligence alignment is unnecessary.
- stewarts 2y ago
  How can you be so certain there is 0 chance LLMs lead to AGI/Superintelligence? Asking curiously, not something I've heard prior.
  - guhidalg 2y ago
    I'm 100% certain that I need to do more than just predict the next token to be considered intelligent. Also call me when ChatGPT can manipulate matter.
    - soulofmischief 2y ago
      > Also call me when ChatGPT can manipulate matter. You mean like PALM-E? https://palm-e.github.io/ https://palm-e.github.io/ Embodiment is the easy part.
    - mypalmike 2y ago
      Are you 100% certain that the human brain performs no language processing which is analogous to token prediction?
      - stubish 2y ago
        A human brain certainly does do predictions, which is very useful to the bit that makes decisions. But how does a pure prediction engine make decisions? Make a judgement call? Analyze inconsistencies? Theorize? The best it can do is blindly follow the mob, a behavior we consider unintelligent even when done by human brains.
llamaimperative 2y ago
So you mean the things the charter foresaw and was intended to make impossible is in fact happening? Who could've thunk it (other than the creators of the charter and nearly anyone else with a loose grasp on how capitalism and technology interact).
hollerith 2y ago
>it just seemed like a needless cost center in a growing business To some of us, that sounds like, "Fire all the climate scientists because they are needless cost center distracting us from the noble goal of burning as much fossil fuel as possible."
- mjr00 2y ago
  It's more like you started a company to research new fuel sources and hired climate scientists to evaluate the environmental impact of those new fuel sources, but during the course of the research you invented a new internal combustion engine that operates 1000% more efficiently so you pivoted your entire business toward that, removing the need for climate scientists. This is a tortured analogy, but what I'm getting at is, if OpenAI is no longer pursuing AGI/superintelligence, it doesn't need an expensive superintelligence alignment team.
  - hollerith 2y ago
    You're much more confident than I am that the researchers at OpenAI (or anyone else currently alive) are masters of their craft to such an extent that they would even be able to predict whether the next big training run they do will result in a superintelligence or not. Another way of saying the same thing is to say that the only way anyone knows that GPT-4 is not dangerously capable is that it has been deployed extensively enough by now that if it was going to harm us, it would've done so by now: not even the researchers that designed and coded-up GPT-4 or watched it during training could predict with any confidence how capable it would be. For example, everyone was quite surprised by its being able to score in the 90th decile on a bar exam. Also, even if they never produce a superintelligence, they are likely to produce insights that would make it easier for other teams to produce a superintelligence. (Since employees are free to leave OpenAI and join some other team, there is no practical way to prevent the flow of insights out of OpenAI.)
    - MichaelZuo 2y ago
      Why do they need to be 'masters of their craft' to place directional bets?
hollerith 2y ago
If AGI is no longer Sam Altman's goal, why was he recently trying to raise 7 trillion dollars for hardware to accelerate progress in AI?

legohead 2y ago

What is the "intelligence" behind a word predictor?

DrSiemer 2y ago
Fake it till you make it

TiredOfLife 2y ago

> We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. Our goal is to solve the core technical challenges of superintelligence alignment by 2027 Can somebody translate this to human?

brandall10 2y ago
That by 2027 they will figure out how to control Skynet so it doesn't kill us all when it awakens.
- TiredOfLife 2y ago
  So chains and shackles or domestication?

colibri727 2y ago

What are these core technical challenges ?