7 ms·

>Ms. Toner disagreed. The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,” and if the company was

>Ms. Toner disagreed. The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,” and if the company was destroyed, she said, that mission would be fulfilled. Now we know where that came from

jacquesm 3y ago
That's a bit naive, to put it mildly. It presumes that nobody else would be able to replicate the effort and that the parties that are able to replicate it would also destroy theirs after proving that it could be done. Fat chance.
- barrkel 3y ago
  Indeed. The only way to retain control of the leading ship in the race is to keep it together as you steer it. If the ship disintegrates, then you're no longer in control of the leading ship, and someone else will win the race.
  - 0xDEAFBEAD 3y ago
    The charter states that OpenAI should allow another company to win the race under certain circumstances: >We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.” https://openai.com/charter https://openai.com/charter That's why Sam pressuring Helen to shut up about Anthropic is so concerning.
    - jacquesm 3y ago
      > we commit to stop competing with and start assisting this project That bit I never understood. It doesn't even have a qualification as to who is stewarding the project as if any AI should be assisted no matter who ends up owning it. The whole thing is just so utterly disconnected from the real world.
      - 0xDEAFBEAD 3y ago
        If you believe that aligned AGI will create a post-scarcity society, there's no reason for its creator to hoard the benefits for themselves. The main thing is to ensure that post-scarcity society actually comes into being. Edit: Note also the "value-aligned, safety-conscious" caveat.
        jacquesm 3y ago
        I don't believe that. What I believe is that the first party to create an AGI will likely end up being nationalized and that that AI will be weaponized the next day to the exclusion of the rest of the world.
        comp_throw7 3y ago
        That's fine, but the OpenAI founders (including Sam Altman![1]) explicitly had in mind "AI kills everyone" as one of the main problems they were attempting to tackle when they started the organization. (Obviously starting OpenAI as a solution to this problem was a terrible mistake, but given that they did so, having a board charged with pulling the plug is better than not having that.) [1] https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-1
        jacquesm 3y ago
        That whole charter is bullshit. You don't invite Microsoft to the party if you care about ethics and you don't disable your fail-safe by giving them full access to all of your stuff. Hence my position that the charter was a fig leaf designed to throw regulators off.
        0xDEAFBEAD 3y ago
        Elsewhere in this thread you stated: "if OpenAI was 'the good guys' then any chance of the good guys getting a head start has just been blown out of the water" So basically we have two cases here: 1. The charter is a deceptive fig leaf, OpenAI aren't the good guys. In which case blowing up OpenAI looks reasonable 2. The charter is not a fig leaf, Helen is doing the right thing by upholding it I'd say upholding the charter is the right thing in intermediate situations as well. Basically, OpenAI are the good guys, in part, to the degree to which the board (and other stakeholders) actually uphold the charter instead of being purely profit-driven.
- ytoawwhra92 3y ago
  The actions of other organisations are not in the scope of the board's mission. The actions of the company the board controls are in that scope. "The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,”" We can't control other's actions, but we can control our own. If we feel that our actions are diverging from our own sense of what we ought to be doing we can change our actions, regardless of how others are behaving.
  - jacquesm 3y ago
    That is absolutely true. But you do operate on a larger stage and sticking your head in the sand and pretending the rest of the world doesn't exist is also not going to work. Altman appears to me to be reckless, ms. Toner appears to be naive. Both should not be near this kind of power.
    - nicce 3y ago
      I would not say that it is fair comparison in this case. People do the best with the control they have. In this case, if your company fails to provide for greater good, then it is best to destroy it. And it also shows some example to others. Whatever others do, you have no power. I does not matter if you pretent or not. Comparison works only if you could do something about it.
      - jacquesm 3y ago
        Yes, but then maybe you should realize that you are out of your depth and abstain. Because clear as day this kind of power is actively being sought by many operators and if OpenAI was 'the good guys' then any chance of the good guys getting a head start has just been blown out of the water. If she agreed with the Microsoft deal but not with what followed that too would be hopelessly naive. Microsoft + ethics?
        nicce 3y ago
        > If she agreed with the Microsoft deal but not with what followed that too would be hopelessly naive. Microsoft + ethics? I guess the main problem here is that nobody knows properly what this deal contains and how it was made. I believe CEO has power to do such deal alone?
nonethewiser 3y ago
Non-profit or not, steering the company towards non-existence isnt in the interest of the company.
- dontknowmuch 3y ago
  These true believers serve a higher calling. Only they can prevent an AIpocalpse.
  - jacquesm 3y ago
    Lucky us that we have such enlightened people to save us. /s
- winenbug 3y ago
  But that's the problem, the board's mission was doomed from the get-go. Their mission isn't to be "in the interest of the company" but "in the interest of humanity" i.e. if they believe OpenAI at its pace would destroy humanity, then their mission is literally to destroy OpenAI itself.
  - bnralt 3y ago
    > if they believe OpenAI at its pace would destroy humanity, then their mission is literally to destroy OpenAI itself. I'd say most people have as much faith in LessWrong eschatology as they do with Christian nationalist eschatology. I can understand how a true believer might want to destroy the company to stop the AI gods they believe in, or shut off ChatGPT every Sunday to avoid living in sin. But it can be an issue when you start viewing your personal beliefs as fundamental truths. There's something nicely secular about profit motives. I'm not sure I want technological advancement to be under the control of any religious belief.
  - QuantumGood 3y ago
    "The board's mission became to destroy OpenAI itself" is ... less sane? ... than everything else that has happened.
renewiltord 3y ago
Interesting. This is the person who "holds comparable influence to a USAF Colonel." according to "a prominent researcher in AI safety" who "discovered prompt injection". https://news.ycombinator.com/item?id=38330566 https://news.ycombinator.com/item?id=38330566 Well, I suppose this tells us something about the AI safety community and whether it makes sense to integrate safety into one's workflow. It seems that the best AI safetyists will scuttle your company from the inside at some moment not yet known. This does sort of imply that it is risky for an AI company to have a safetyist on board. That does seem to be accurate. For instance, Google had the most formidable safety team, and they've got the worst AI. Meta ignored theirs and they've given us very good open models.
- comp_throw7 3y ago
  "AI safety" should be disentangled into "AI notkilleveroneism" and "AI ethics", which are substantially non-overlapping categories. I've looked at who works at Preamble, and there aren't any names there that I recognize from the side of things that's concerned with x-risk. Take their takes with a grain of salt.
- Seanambers 3y ago
  AI safety is just woke 2.0.
  - tomrod 3y ago
    This is incorrect.
foobarian 3y ago
That doesn’t even follow when taken literally. If the company is destroyed presumably they can’t create artificial intelligence, so there is nothing there to benefit all humanity in the first place.
- comp_throw7 3y ago
  It follows just fine, I think, given that the possibility space is not limited to "create beneficial AGI" and "don't create AGI". It also includes "create unaligned AGI", which is obviously much worse "don't create AGI"; the board would be remiss in its duties if it didn't try to prevent that from happening.
  - concinds 3y ago
    Could we all stop entertaining these LessWrong hypotheticals? The notion that the board wants to shut down OpenAI to stop non-existent AGI is mad.
- Davidzheng 3y ago
  The mission of benefiting humanity can also mean not harming
- liuliu 3y ago
  The company is not destroyed. Board is not shutting down the company, they fired the CEO. The other ~700 people chose to quit. Not sure why it is "life-ruined" other than probably some tender offers withdrawn (and even this bit is unclear whether Thrive Capital will do that).
brigadier132 3y ago
Amazing how these higher minded people always forget about the little people on the ground. All these employees losing their livelihoods for the greater good. Not surprised an ethicist thinks like this.
- tokai 3y ago
  Highly educated AI specialists are the little people now? They can all find employment in an instant.
  - brigadier132 3y ago
    Do you think every single person in OpenAI is an AI Phd? We are talking about 770 people, most of which are not rich and likely do not even own their own home. So yes, they are the little people. If you are going to potentially ruin the lives of 770 people you should have a better reason than being afraid of a chatbot and laundry buddy.
    - deleted 3y ago
      [deleted]
    - BryantD 3y ago
      Given that two companies have open offers to OpenAI employees, it seems possible that the board accurately gauged the impact on human lives here. Also possible that they got lucky, of course.
    - Intralexical 3y ago
      Aside from those individuals being in-demand professionals with specialized experience from a big famous company, 770 people will be the least of our concerns if the social disruption from this stuff gets bad enough one day. Not surprised Hacker News identifies and sympathizes more with the software folks than with everyone else who has to live with the systems they build/disrupt, though.
      - 3y ago
0xDEAFBEAD 3y ago
It's pretty heavily implied by OpenAI's charter: https://openai.com/charter https://openai.com/charter It's weird to me that Helen is getting so much crap for upholding the charter. No one objected to the charter at the time it was published. The charter was always available for employees, investors, and customers to read. Did everyone expect it to be ignored when push came to shove? There's a lot of pressure on Helen right now from people who have a financial stake in this situation, but it's right there in the charter that OpenAI's primary fiduciary duty is to humanity. If employees/investors/customers weren't OK with that, they should not have worked with OpenAI.
- tapoxi 3y ago
  Investors wanted to have a commercial enterprise while pretending it was a nonprofit acting for the good of humanity. This helps market something as scary and potentially destructive as AI. "It can't be that bad, they're a nonprofit with altruistic goals!" Then the investors get mad when the board they intended to be figureheads actually try to uphold some principles. Best to rip the band-aid off and stop pretending.
  - nicce 3y ago
    Let’s assume this happened. Then everyone would make everything in their power to get their old CEO back which made this possible. Oh wait.
    - winenbug 3y ago
      Ya which is why Sam should just start a new company, maybe spend a few months to catch back up but then it won't be tied down to any of this shenanigans. It's the best solution imo. I have a feeling that's exactly what he's doing at Microsoft; at some point their "AI lab" will be spun off into a new company.
  - wvenable 3y ago
    I think you have it backwards. OpenAI specifically sought out investors because they couldn't fulfill their mission without the infrastructure to do it. Investors don't just give money away -- it's in the name itself -- they're investing. The point is for the commercial enterprise to provide a return on their investment. OpenAI is a non-profit running this commercial enterprise but they are seemingly at odds with that enterprise. Investors should rightly be very concerned about the future of their investment.

jacquesm 3y ago

That's a bit naive, to put it mildly. It presumes that nobody else would be able to replicate the effort and that the parties that are able to replicate it would also destroy theirs after proving that it could be done. Fat chance.

barrkel 3y ago
Indeed. The only way to retain control of the leading ship in the race is to keep it together as you steer it. If the ship disintegrates, then you're no longer in control of the leading ship, and someone else will win the race.
- 0xDEAFBEAD 3y ago
  The charter states that OpenAI should allow another company to win the race under certain circumstances: >We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.” https://openai.com/charter https://openai.com/charter That's why Sam pressuring Helen to shut up about Anthropic is so concerning.
  - jacquesm 3y ago
    > we commit to stop competing with and start assisting this project That bit I never understood. It doesn't even have a qualification as to who is stewarding the project as if any AI should be assisted no matter who ends up owning it. The whole thing is just so utterly disconnected from the real world.
    - 0xDEAFBEAD 3y ago
      If you believe that aligned AGI will create a post-scarcity society, there's no reason for its creator to hoard the benefits for themselves. The main thing is to ensure that post-scarcity society actually comes into being. Edit: Note also the "value-aligned, safety-conscious" caveat.
      - jacquesm 3y ago
        I don't believe that. What I believe is that the first party to create an AGI will likely end up being nationalized and that that AI will be weaponized the next day to the exclusion of the rest of the world.
        comp_throw7 3y ago
        That's fine, but the OpenAI founders (including Sam Altman![1]) explicitly had in mind "AI kills everyone" as one of the main problems they were attempting to tackle when they started the organization. (Obviously starting OpenAI as a solution to this problem was a terrible mistake, but given that they did so, having a board charged with pulling the plug is better than not having that.) [1] https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-1
        jacquesm 3y ago
        That whole charter is bullshit. You don't invite Microsoft to the party if you care about ethics and you don't disable your fail-safe by giving them full access to all of your stuff. Hence my position that the charter was a fig leaf designed to throw regulators off.
        0xDEAFBEAD 3y ago
        Elsewhere in this thread you stated: "if OpenAI was 'the good guys' then any chance of the good guys getting a head start has just been blown out of the water" So basically we have two cases here: 1. The charter is a deceptive fig leaf, OpenAI aren't the good guys. In which case blowing up OpenAI looks reasonable 2. The charter is not a fig leaf, Helen is doing the right thing by upholding it I'd say upholding the charter is the right thing in intermediate situations as well. Basically, OpenAI are the good guys, in part, to the degree to which the board (and other stakeholders) actually uphold the charter instead of being purely profit-driven.
ytoawwhra92 3y ago
The actions of other organisations are not in the scope of the board's mission. The actions of the company the board controls are in that scope. "The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,”" We can't control other's actions, but we can control our own. If we feel that our actions are diverging from our own sense of what we ought to be doing we can change our actions, regardless of how others are behaving.
- jacquesm 3y ago
  That is absolutely true. But you do operate on a larger stage and sticking your head in the sand and pretending the rest of the world doesn't exist is also not going to work. Altman appears to me to be reckless, ms. Toner appears to be naive. Both should not be near this kind of power.
  - nicce 3y ago
    I would not say that it is fair comparison in this case. People do the best with the control they have. In this case, if your company fails to provide for greater good, then it is best to destroy it. And it also shows some example to others. Whatever others do, you have no power. I does not matter if you pretent or not. Comparison works only if you could do something about it.
    - jacquesm 3y ago
      Yes, but then maybe you should realize that you are out of your depth and abstain. Because clear as day this kind of power is actively being sought by many operators and if OpenAI was 'the good guys' then any chance of the good guys getting a head start has just been blown out of the water. If she agreed with the Microsoft deal but not with what followed that too would be hopelessly naive. Microsoft + ethics?
      - nicce 3y ago
        > If she agreed with the Microsoft deal but not with what followed that too would be hopelessly naive. Microsoft + ethics? I guess the main problem here is that nobody knows properly what this deal contains and how it was made. I believe CEO has power to do such deal alone?

nonethewiser 3y ago

Non-profit or not, steering the company towards non-existence isnt in the interest of the company.

dontknowmuch 3y ago
These true believers serve a higher calling. Only they can prevent an AIpocalpse.
- jacquesm 3y ago
  Lucky us that we have such enlightened people to save us. /s
winenbug 3y ago
But that's the problem, the board's mission was doomed from the get-go. Their mission isn't to be "in the interest of the company" but "in the interest of humanity" i.e. if they believe OpenAI at its pace would destroy humanity, then their mission is literally to destroy OpenAI itself.
- bnralt 3y ago
  > if they believe OpenAI at its pace would destroy humanity, then their mission is literally to destroy OpenAI itself. I'd say most people have as much faith in LessWrong eschatology as they do with Christian nationalist eschatology. I can understand how a true believer might want to destroy the company to stop the AI gods they believe in, or shut off ChatGPT every Sunday to avoid living in sin. But it can be an issue when you start viewing your personal beliefs as fundamental truths. There's something nicely secular about profit motives. I'm not sure I want technological advancement to be under the control of any religious belief.
- QuantumGood 3y ago
  "The board's mission became to destroy OpenAI itself" is ... less sane? ... than everything else that has happened.

renewiltord 3y ago

Interesting. This is the person who "holds comparable influence to a USAF Colonel." according to "a prominent researcher in AI safety" who "discovered prompt injection". https://news.ycombinator.com/item?id=38330566 https://news.ycombinator.com/item?id=38330566 Well, I suppose this tells us something about the AI safety community and whether it makes sense to integrate safety into one's workflow. It seems that the best AI safetyists will scuttle your company from the inside at some moment not yet known. This does sort of imply that it is risky for an AI company to have a safetyist on board. That does seem to be accurate. For instance, Google had the most formidable safety team, and they've got the worst AI. Meta ignored theirs and they've given us very good open models.

comp_throw7 3y ago
"AI safety" should be disentangled into "AI notkilleveroneism" and "AI ethics", which are substantially non-overlapping categories. I've looked at who works at Preamble, and there aren't any names there that I recognize from the side of things that's concerned with x-risk. Take their takes with a grain of salt.
Seanambers 3y ago
AI safety is just woke 2.0.
- tomrod 3y ago
  This is incorrect.

foobarian 3y ago

That doesn’t even follow when taken literally. If the company is destroyed presumably they can’t create artificial intelligence, so there is nothing there to benefit all humanity in the first place.

comp_throw7 3y ago
It follows just fine, I think, given that the possibility space is not limited to "create beneficial AGI" and "don't create AGI". It also includes "create unaligned AGI", which is obviously much worse "don't create AGI"; the board would be remiss in its duties if it didn't try to prevent that from happening.
- concinds 3y ago
  Could we all stop entertaining these LessWrong hypotheticals? The notion that the board wants to shut down OpenAI to stop non-existent AGI is mad.
Davidzheng 3y ago
The mission of benefiting humanity can also mean not harming
liuliu 3y ago
The company is not destroyed. Board is not shutting down the company, they fired the CEO. The other ~700 people chose to quit. Not sure why it is "life-ruined" other than probably some tender offers withdrawn (and even this bit is unclear whether Thrive Capital will do that).

brigadier132 3y ago

Amazing how these higher minded people always forget about the little people on the ground. All these employees losing their livelihoods for the greater good. Not surprised an ethicist thinks like this.

tokai 3y ago
Highly educated AI specialists are the little people now? They can all find employment in an instant.
- brigadier132 3y ago
  Do you think every single person in OpenAI is an AI Phd? We are talking about 770 people, most of which are not rich and likely do not even own their own home. So yes, they are the little people. If you are going to potentially ruin the lives of 770 people you should have a better reason than being afraid of a chatbot and laundry buddy.
  - deleted 3y ago
    [deleted]
  - BryantD 3y ago
    Given that two companies have open offers to OpenAI employees, it seems possible that the board accurately gauged the impact on human lives here. Also possible that they got lucky, of course.
  - Intralexical 3y ago
    Aside from those individuals being in-demand professionals with specialized experience from a big famous company, 770 people will be the least of our concerns if the social disruption from this stuff gets bad enough one day. Not surprised Hacker News identifies and sympathizes more with the software folks than with everyone else who has to live with the systems they build/disrupt, though.
    - 3y ago

0xDEAFBEAD 3y ago

It's pretty heavily implied by OpenAI's charter: https://openai.com/charter https://openai.com/charter It's weird to me that Helen is getting so much crap for upholding the charter. No one objected to the charter at the time it was published. The charter was always available for employees, investors, and customers to read. Did everyone expect it to be ignored when push came to shove? There's a lot of pressure on Helen right now from people who have a financial stake in this situation, but it's right there in the charter that OpenAI's primary fiduciary duty is to humanity. If employees/investors/customers weren't OK with that, they should not have worked with OpenAI.

tapoxi 3y ago
Investors wanted to have a commercial enterprise while pretending it was a nonprofit acting for the good of humanity. This helps market something as scary and potentially destructive as AI. "It can't be that bad, they're a nonprofit with altruistic goals!" Then the investors get mad when the board they intended to be figureheads actually try to uphold some principles. Best to rip the band-aid off and stop pretending.
- nicce 3y ago
  Let’s assume this happened. Then everyone would make everything in their power to get their old CEO back which made this possible. Oh wait.
  - winenbug 3y ago
    Ya which is why Sam should just start a new company, maybe spend a few months to catch back up but then it won't be tied down to any of this shenanigans. It's the best solution imo. I have a feeling that's exactly what he's doing at Microsoft; at some point their "AI lab" will be spun off into a new company.
- wvenable 3y ago
  I think you have it backwards. OpenAI specifically sought out investors because they couldn't fulfill their mission without the infrastructure to do it. Investors don't just give money away -- it's in the name itself -- they're investing. The point is for the commercial enterprise to provide a return on their investment. OpenAI is a non-profit running this commercial enterprise but they are seemingly at odds with that enterprise. Investors should rightly be very concerned about the future of their investment.