5 ms·

Yes, it's valuable to have a small research team who focuses on R&D outside the production loop. But when you give them a larger remit, and structure teams wit

by mgdev 2y ago

Yes, it's valuable to have a small research team who focuses on R&D outside the production loop. But when you give them a larger remit, and structure teams with some owning "value" and and others essentially owning "risk", the risk teams tend to attract navel-gazers and/or coasters. They wield their authority like a whip without regard for business value. The problem is the incentives tend to be totally misaligned. Instead the team that ships the "value" also needs to own their own risk management - metrics and counter metrics - with management holding them accountable for striking the balance.

davidivadavid 2y ago
So what you're saying is OpenAI can't align two teams internally but they want to align a super intelligence.
- mgdev 2y ago
  I think "aligning super intelligence" is a nothingburger of a goal, for exactly that reason. It's not a problem that's unique to OpenAI. The reason you can't "align" AI is because we, as humans on the planet, aren't universally aligned on what "aligned" means. At best you can align to a particular group of people (a company, a town, a state, a country). But "global alignment" in almost any context just devolves into war or authoritarianism (virtual or actual).
seanhunter 2y ago
Completely agree with this. Everyone doing AI has to do AI that is both valuable and responsible. You can't have an org structure where the valuable AI and responsible AI teams are in some kind of war against each other. It will just never work well.[1] Imagine if vehicle manufacturers[2] split their design and R&D teams into a "make the thing go" team and a "don't kill the passengers" team. Literally noone would think that arrangement made sense. I can totally see when we are at a state of significant maturity of both AI and AI regulation that you have a special part of your legal team that are specialised in legal/regulatory compliance issues around AI just like companies tend to have specialised data privacy compliance experts. But we're not there yet. [1] If you're serious about long-term AI risk and alignment research, sponsor some independent academic research that gets published. That way it's arms-length and genuinely credible. [2] If you like you can maybe mentally exclude Boeing in this.
- mgdev 2y ago
  > Imagine if vehicle manufacturers[2] split their design and R&D teams into a "make the thing go" team and a "don't kill the passengers" team. Rocket caskets. Can't kill someone who is already dead!
- doktrin 2y ago
  1. Audit / evaluation / quality assurance teams exist across multiple verticals from multinationals to government, and cannot reliably function when overly subservient to the production or “value creating” side 2. Boeing is a good and timely example of the consequences of said internal checks and balances collapsing under “value creation” pressure. That was a catastrophic failure which still can’t reasonably be compared to the downside of misaligned AI.
  - seanhunter 2y ago
    I agree with you on both points, but they have QA which is 1. The long-term risk team was more of a research/futurology/navel-gazing entity rather than a qa/audit function. I would say if you have any possible safety/alignment test that you can feasibly run it should be part of the CI/CD pipline and be run during training also. That's not what that group was doing.
    - doktrin 2y ago
      That’s quite the narrow goalpost you’ve set up. What happens if a problem can’t be expressed as a Jenkins pipeline operation?
Hasu 2y ago
The purpose of the risk team at OpenAI was to prevent the destruction of humanity. I think you definitely want people who have that responsibility to "wield their authority like a whip without regard for business value". Now, whether you buy OpenAI's hype about the potential danger (and value) of their products, that's up to you, but when the company says, "We're getting rid of the team that makes sure we don't kill everyone", there is a message being sent. Whether it's "We don't really think our technology is that dangerous (and therefore valuable)" or "We don't really care if we accidentally kill everyone", it's not a good message.
- squarefoot 2y ago
  > but when the company says, "We're getting rid of the team that makes sure we don't kill everyone", there is a message being sent Hard not to imagine a pattern if one considers what they did a few months ago: https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-on-military-use-of-its-ai-tools.html https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-o...
  - mnky9800n 2y ago
    Maybe the message is, these ai ain't going to take over the world.
- mgdev 2y ago
  > The purpose of the risk team at OpenAI was to prevent the destruction of humanity. Yeah, the problem (in this outsider's opinion) is that that charter is so ill-defined that it's practically useless, which in turn means that any sufficiently loud voice can apply it to anything. It's practically begging to be used as a whip. > I think you definitely want people who have that responsibility to "wield their authority like a whip without regard for business value". No, because without thinking of value, there is no enterprise, and then your mission is impossible. Essentially, it creates an incentive where the best outcome is to destroy the company. And, hey hey, that's kinda what almost happened. > Whether it's "We don't really think our technology is that dangerous (and therefore valuable)" or "We don't really care if we accidentally kill everyone", it's not a good message. I don't think it has to be so black-and-white as this. Meta, Microsoft, and Google did the same thing. Instead, those functions have been integrated more closely into the value teams. And I can't imagine Amazon or Oracle ever having those teams in the first place. They likely all realized the same thing: those teams add huge drag without adding measurable business value. And yes, there are ways to measure the business value of risk management, and weigh against upside value to decide the correct course of actions - it's just that most of those teams in big tech don't actually take a formal "risk management" approach. Instead they pontificate or copy and enforce.
23B1 2y ago
No, this just puts the fox in the henhouse. Systems of checks-and-balances between independent entities exist for a reason. Without them internally, it'll just fall to regulators, which of course is what shareholders want; to privatize upside and socialize downside.
- mgdev 2y ago
  Agree that you need checks and balances, but there are better and worse systems. As someone who has scaled orgs from tens to thousands of engineers, I can tell you: you need value teams to own their own risk. A small, central R&D team may work with management to set the bar, but they can't be responsible for mitigating the risk on the ground - and they shouldn't be led to believe that that is their job. It never works, and creates bad team dynamics. Either the central team goes too far, or they feel ignored. (See: security, compliance.)
germinator 2y ago
In the general case, I mostly agree, but it cracks me up that this is the prevailing attitude when it comes to our industry; but when we see police departments or government agencies trying to follow the same playbook, we immediately point out how that's laughable and doesn't result in real accountability. In this specific case, though, Sam Altman's narrative is that they created an existential risk to humanity and that the access to it needs to be restricted for others. So which is it?
- nprateem 2y ago
  Neither. It was pure hype to get free column inches. Anyone who's used their AI and discovered how it ignores instructions and makes things up isn't going to honestly believe it poses an existential threat any time soon. Now they're big enough they can end that charade.
  - saulpw 2y ago
    Ignoring instructions and making things up may very well be an existential threat. Just not of the SkyNet variety.
    - nprateem 2y ago
      Of course, but stupidity and lying always potentially is. It doesn't need to be technical. That's a whole different conversation.
reducesuffering 2y ago
Did you read Jan Leike's resignation? https://x.com/janleike/status/1791498174659715494 https://x.com/janleike/status/1791498174659715494 I hope others see that there are two extremely intelligent sides, but one has mega $$ to earn and the other is pleading that there are dangers ahead and not to follow the money and fame. This is climate change and oil companies all over again, and just like then and now, oil companies are winning. Fundamentally, many people are the first stage, denial. Staring down our current trajectory of AGI is one of the darkest realities to imagine and that is not pleasant to grapple with.
- mgdev 2y ago
  I hadn't until now, but it's the other failure mode I mentioned in another fork [1] of this thread: > A small, central R&D team may work with management to set the bar, but they can't be responsible for mitigating the risk on the ground - and they shouldn't be led to believe that that is their job. It never works, and creates bad team dynamics. Either the central team goes too far, or they feel ignored. (See: security, compliance.) [1]: https://news.ycombinator.com/item?id=40391283 https://news.ycombinator.com/item?id=40391283
- swatcoder 2y ago
  The alternate take is that the ceiling is proving to be lower than originally hoped, and that one team of startup folk are looking to squeeze out a business model that satisfies their original investors and the other team of passionate researchers are ready to go find their next bold research opportunities elsewhere.
  - mgdev 2y ago
    I like this take best. :) Most folks who are part of the startup, don't care to stick around for the scale. Different phases of the company tend to attract different types of people.
  - reducesuffering 2y ago
    > passionate researchers are ready to go find their next bold research opportunities elsewhere Not supported by Ilya agreeing with board to fire Sam Altman. I also think you'll struggle to find a majority of people thinking AI research's "ceiling is proving to be lower than originally hoped", what with 4o, SORA, GPT5 all coming and it's only been 1.5 years since ChatGPT.
encoderer 2y ago
Here’s the former team leaders take. Says they couldn’t get compute resources to do their job. https://x.com/janleike/status/1791498174659715494 https://x.com/janleike/status/1791498174659715494
- petre 2y ago
  Well, Altman did try to get potential investors to shell out $7T for "AI chips".
6gvONxR4sf7o 2y ago
Speaking of incentives, the way your suggestion tends to work out is that you make a team own both value and risk, then incentivize them only for value, which works out predictably.

by mgdev 2y ago

davidivadavid 2y ago

So what you're saying is OpenAI can't align two teams internally but they want to align a super intelligence.

mgdev 2y ago
I think "aligning super intelligence" is a nothingburger of a goal, for exactly that reason. It's not a problem that's unique to OpenAI. The reason you can't "align" AI is because we, as humans on the planet, aren't universally aligned on what "aligned" means. At best you can align to a particular group of people (a company, a town, a state, a country). But "global alignment" in almost any context just devolves into war or authoritarianism (virtual or actual).

seanhunter 2y ago

Completely agree with this. Everyone doing AI has to do AI that is both valuable and responsible. You can't have an org structure where the valuable AI and responsible AI teams are in some kind of war against each other. It will just never work well.[1] Imagine if vehicle manufacturers[2] split their design and R&D teams into a "make the thing go" team and a "don't kill the passengers" team. Literally noone would think that arrangement made sense. I can totally see when we are at a state of significant maturity of both AI and AI regulation that you have a special part of your legal team that are specialised in legal/regulatory compliance issues around AI just like companies tend to have specialised data privacy compliance experts. But we're not there yet. [1] If you're serious about long-term AI risk and alignment research, sponsor some independent academic research that gets published. That way it's arms-length and genuinely credible. [2] If you like you can maybe mentally exclude Boeing in this.

mgdev 2y ago
> Imagine if vehicle manufacturers[2] split their design and R&D teams into a "make the thing go" team and a "don't kill the passengers" team. Rocket caskets. Can't kill someone who is already dead!
doktrin 2y ago
1. Audit / evaluation / quality assurance teams exist across multiple verticals from multinationals to government, and cannot reliably function when overly subservient to the production or “value creating” side 2. Boeing is a good and timely example of the consequences of said internal checks and balances collapsing under “value creation” pressure. That was a catastrophic failure which still can’t reasonably be compared to the downside of misaligned AI.
- seanhunter 2y ago
  I agree with you on both points, but they have QA which is 1. The long-term risk team was more of a research/futurology/navel-gazing entity rather than a qa/audit function. I would say if you have any possible safety/alignment test that you can feasibly run it should be part of the CI/CD pipline and be run during training also. That's not what that group was doing.
  - doktrin 2y ago
    That’s quite the narrow goalpost you’ve set up. What happens if a problem can’t be expressed as a Jenkins pipeline operation?

Hasu 2y ago

The purpose of the risk team at OpenAI was to prevent the destruction of humanity. I think you definitely want people who have that responsibility to "wield their authority like a whip without regard for business value". Now, whether you buy OpenAI's hype about the potential danger (and value) of their products, that's up to you, but when the company says, "We're getting rid of the team that makes sure we don't kill everyone", there is a message being sent. Whether it's "We don't really think our technology is that dangerous (and therefore valuable)" or "We don't really care if we accidentally kill everyone", it's not a good message.

squarefoot 2y ago
> but when the company says, "We're getting rid of the team that makes sure we don't kill everyone", there is a message being sent Hard not to imagine a pattern if one considers what they did a few months ago: https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-on-military-use-of-its-ai-tools.html https://www.cnbc.com/2024/01/16/openai-quietly-removes-ban-o...
- mnky9800n 2y ago
  Maybe the message is, these ai ain't going to take over the world.
mgdev 2y ago
> The purpose of the risk team at OpenAI was to prevent the destruction of humanity. Yeah, the problem (in this outsider's opinion) is that that charter is so ill-defined that it's practically useless, which in turn means that any sufficiently loud voice can apply it to anything. It's practically begging to be used as a whip. > I think you definitely want people who have that responsibility to "wield their authority like a whip without regard for business value". No, because without thinking of value, there is no enterprise, and then your mission is impossible. Essentially, it creates an incentive where the best outcome is to destroy the company. And, hey hey, that's kinda what almost happened. > Whether it's "We don't really think our technology is that dangerous (and therefore valuable)" or "We don't really care if we accidentally kill everyone", it's not a good message. I don't think it has to be so black-and-white as this. Meta, Microsoft, and Google did the same thing. Instead, those functions have been integrated more closely into the value teams. And I can't imagine Amazon or Oracle ever having those teams in the first place. They likely all realized the same thing: those teams add huge drag without adding measurable business value. And yes, there are ways to measure the business value of risk management, and weigh against upside value to decide the correct course of actions - it's just that most of those teams in big tech don't actually take a formal "risk management" approach. Instead they pontificate or copy and enforce.

23B1 2y ago

No, this just puts the fox in the henhouse. Systems of checks-and-balances between independent entities exist for a reason. Without them internally, it'll just fall to regulators, which of course is what shareholders want; to privatize upside and socialize downside.

mgdev 2y ago
Agree that you need checks and balances, but there are better and worse systems. As someone who has scaled orgs from tens to thousands of engineers, I can tell you: you need value teams to own their own risk. A small, central R&D team may work with management to set the bar, but they can't be responsible for mitigating the risk on the ground - and they shouldn't be led to believe that that is their job. It never works, and creates bad team dynamics. Either the central team goes too far, or they feel ignored. (See: security, compliance.)

germinator 2y ago

In the general case, I mostly agree, but it cracks me up that this is the prevailing attitude when it comes to our industry; but when we see police departments or government agencies trying to follow the same playbook, we immediately point out how that's laughable and doesn't result in real accountability. In this specific case, though, Sam Altman's narrative is that they created an existential risk to humanity and that the access to it needs to be restricted for others. So which is it?

nprateem 2y ago
Neither. It was pure hype to get free column inches. Anyone who's used their AI and discovered how it ignores instructions and makes things up isn't going to honestly believe it poses an existential threat any time soon. Now they're big enough they can end that charade.
- saulpw 2y ago
  Ignoring instructions and making things up may very well be an existential threat. Just not of the SkyNet variety.
  - nprateem 2y ago
    Of course, but stupidity and lying always potentially is. It doesn't need to be technical. That's a whole different conversation.

reducesuffering 2y ago

Did you read Jan Leike's resignation? https://x.com/janleike/status/1791498174659715494 https://x.com/janleike/status/1791498174659715494 I hope others see that there are two extremely intelligent sides, but one has mega $$ to earn and the other is pleading that there are dangers ahead and not to follow the money and fame. This is climate change and oil companies all over again, and just like then and now, oil companies are winning. Fundamentally, many people are the first stage, denial. Staring down our current trajectory of AGI is one of the darkest realities to imagine and that is not pleasant to grapple with.

mgdev 2y ago
I hadn't until now, but it's the other failure mode I mentioned in another fork [1] of this thread: > A small, central R&D team may work with management to set the bar, but they can't be responsible for mitigating the risk on the ground - and they shouldn't be led to believe that that is their job. It never works, and creates bad team dynamics. Either the central team goes too far, or they feel ignored. (See: security, compliance.) [1]: https://news.ycombinator.com/item?id=40391283 https://news.ycombinator.com/item?id=40391283
swatcoder 2y ago
The alternate take is that the ceiling is proving to be lower than originally hoped, and that one team of startup folk are looking to squeeze out a business model that satisfies their original investors and the other team of passionate researchers are ready to go find their next bold research opportunities elsewhere.
- mgdev 2y ago
  I like this take best. :) Most folks who are part of the startup, don't care to stick around for the scale. Different phases of the company tend to attract different types of people.
- reducesuffering 2y ago
  > passionate researchers are ready to go find their next bold research opportunities elsewhere Not supported by Ilya agreeing with board to fire Sam Altman. I also think you'll struggle to find a majority of people thinking AI research's "ceiling is proving to be lower than originally hoped", what with 4o, SORA, GPT5 all coming and it's only been 1.5 years since ChatGPT.

encoderer 2y ago

Here’s the former team leaders take. Says they couldn’t get compute resources to do their job. https://x.com/janleike/status/1791498174659715494 https://x.com/janleike/status/1791498174659715494

petre 2y ago
Well, Altman did try to get potential investors to shell out $7T for "AI chips".

6gvONxR4sf7o 2y ago

Speaking of incentives, the way your suggestion tends to work out is that you make a team own both value and risk, then incentivize them only for value, which works out predictably.