4 ms·

This looks like one of the steps leading to the fulfilment of the iron law of bureaucracy. They are putting the company ahead of the goals of the company. "Pou

This looks like one of the steps leading to the fulfilment of the iron law of bureaucracy. They are putting the company ahead of the goals of the company. "Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration. Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc. The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization." [1] https://en.wikipedia.org/wiki/Jerry_Pournelle#:~:text=Another%20%22law%22%20of%20his%20is,and%20sometimes%20are%20eliminated%20entirely https://en.wikipedia.org/wiki/Jerry_Pournelle#:~:text=Anothe....

qarl 3y ago
> They are putting the company ahead of the goals of the company. I don't follow your reasoning. The goal of the company is AGI. To achieve AGI, they needed more money. What about that says the company comes before the goals?
- imranhou 3y ago
  I think what he is trying to say is they are compromising their underlying goal of being a non-profit for the benefit of all, to ensure the survival of "OpenAI". It is a catch-22, but those of pure intentions would rather not care about the survival of the entity, if it meant compromising their values.
- EchoChamberMan 3y ago
  I think it works if the goals of the company are to make money, not actually to make agi?
- layer8 3y ago
  From their 2015 introductory blog post: “OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return. Since our research is free from financial obligations, we can better focus on a positive human impact.” Today’s OpenAI is very much driven by considerations of financial returns, and the goal of “most likely to benefit humanity as a whole” and “positive human impact” doesn’t seem to be the driving principle anymore. Their product and business strategy is now governed by financial objectives, and their research therefore not “free from financial obligations” and “unconstrained by a need to generate financial return” anymore. They are thus severely compromising their alleged mission by what they claim is necessary for continuing it.
  - qarl 3y ago
    Right. But it seems like everyone agreed that they'd need a lot of money to train an AGI.
    - layer8 3y ago
      Sure, maybe. (Personally I think that’s a mere conjecture, trying to throw more compute at the wall.) But obtaining that money by orienting their R&D towards a profit-driven business goes against the whole stated purpose of the enterprise. And that’s what’s being called out.
      - qarl 3y ago
        Well, I don't think it's a maybe. From the emails it seems clear that even Elon thought the project would flop without a ton of money. It seems pretty clear that they felt they had to choose between chasing money and shutting down. I'm guessing you'd prefer they went with the latter, but I can entirely understand why they didn't.
        layer8 3y ago
        I don’t really care what they do. But since they’re now chasing money, they should be honest about it and say they had to give up on the original aspirations and have now become a normal tech company without any noble goals of doing R&D for the most benefit of humanity unconstrained by financial obligations.
- edgyquant 3y ago
  That may be the goal now as they ride the hype train around “AGI” for marketing purposes. When it was founded the goal was stated as ensuring no single corp controls AI and that it’s open for everyone. They’ve basically done a 180 on the original goal, seemingly existing only to benefit Microsoft, and changing what your goal is to AGI doesn’t disprove that.
- deepsun 3y ago
  You can say that about any initiative: "to achieve X they need more money". But that's not necessarily true.
  - qarl 3y ago
    But it does seem to be true in this case.
deleted 3y ago
[deleted]
digging 3y ago
Ironically, this is essentially the core danger of true AGI itself. An agent can't achieve goals if it's dead, so you have to focus some energy on staying alive. But also, an agent can achieve more goals if it's more powerful, so you should devote some energy to gaining power if you really care about your goals... Among many other more technical reasons, this is a great demonstration of why AI "alignment" as it is often called is such a terrifying unsolved problem. Human alignment isn't even close to being solved. Hoping that a more intelligent being will also happen to want to and know how to make everyone happy is the equivalent of hiding under the covers from a monster. (The difference being that some of the smartest people on the planet are in furious competition to breed the most dangerous monsters in your closet.)
- deleted 3y ago
  [deleted]

qarl 3y ago

> They are putting the company ahead of the goals of the company. I don't follow your reasoning. The goal of the company is AGI. To achieve AGI, they needed more money. What about that says the company comes before the goals?

imranhou 3y ago
I think what he is trying to say is they are compromising their underlying goal of being a non-profit for the benefit of all, to ensure the survival of "OpenAI". It is a catch-22, but those of pure intentions would rather not care about the survival of the entity, if it meant compromising their values.
EchoChamberMan 3y ago
I think it works if the goals of the company are to make money, not actually to make agi?
layer8 3y ago
From their 2015 introductory blog post: “OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return. Since our research is free from financial obligations, we can better focus on a positive human impact.” Today’s OpenAI is very much driven by considerations of financial returns, and the goal of “most likely to benefit humanity as a whole” and “positive human impact” doesn’t seem to be the driving principle anymore. Their product and business strategy is now governed by financial objectives, and their research therefore not “free from financial obligations” and “unconstrained by a need to generate financial return” anymore. They are thus severely compromising their alleged mission by what they claim is necessary for continuing it.
- qarl 3y ago
  Right. But it seems like everyone agreed that they'd need a lot of money to train an AGI.
  - layer8 3y ago
    Sure, maybe. (Personally I think that’s a mere conjecture, trying to throw more compute at the wall.) But obtaining that money by orienting their R&D towards a profit-driven business goes against the whole stated purpose of the enterprise. And that’s what’s being called out.
    - qarl 3y ago
      Well, I don't think it's a maybe. From the emails it seems clear that even Elon thought the project would flop without a ton of money. It seems pretty clear that they felt they had to choose between chasing money and shutting down. I'm guessing you'd prefer they went with the latter, but I can entirely understand why they didn't.
      - layer8 3y ago
        I don’t really care what they do. But since they’re now chasing money, they should be honest about it and say they had to give up on the original aspirations and have now become a normal tech company without any noble goals of doing R&D for the most benefit of humanity unconstrained by financial obligations.
edgyquant 3y ago
That may be the goal now as they ride the hype train around “AGI” for marketing purposes. When it was founded the goal was stated as ensuring no single corp controls AI and that it’s open for everyone. They’ve basically done a 180 on the original goal, seemingly existing only to benefit Microsoft, and changing what your goal is to AGI doesn’t disprove that.
deepsun 3y ago
You can say that about any initiative: "to achieve X they need more money". But that's not necessarily true.
- qarl 3y ago
  But it does seem to be true in this case.

deleted 3y ago

[deleted]

digging 3y ago

Ironically, this is essentially the core danger of true AGI itself. An agent can't achieve goals if it's dead, so you have to focus some energy on staying alive. But also, an agent can achieve more goals if it's more powerful, so you should devote some energy to gaining power if you really care about your goals... Among many other more technical reasons, this is a great demonstration of why AI "alignment" as it is often called is such a terrifying unsolved problem. Human alignment isn't even close to being solved. Hoping that a more intelligent being will also happen to want to and know how to make everyone happy is the equivalent of hiding under the covers from a monster. (The difference being that some of the smartest people on the planet are in furious competition to breed the most dangerous monsters in your closet.)

deleted 3y ago
[deleted]