4 ms·

> ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automate

> ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automated AI researcher can also be an automated safety or alignment researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures. In other words... "We must pursue advancements in AI to protect us against advancements in AI?" edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me: 1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before? 2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"

deleted 28d ago
[deleted]
interstice 28d ago
On the one hand you need any lathe to build a good lathe, even a bad one. On the other, that is a potentially flawed principle to base the entire future of AI on.
andai 28d ago
> The fundamental challenge of AI alignment is generalization. ... > We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. -- From another OpenAI article in a sister thread: An Alien Mind https://news.ycombinator.com/item?id=49588080 https://news.ycombinator.com/item?id=49588080
- ahartmetz 28d ago
  That's a bit bullshit, isn't it? They basically redefined "needs more R&D" as "needs stronger AI". Maybe so - maybe AI won't help much with that problem.
MelonUsk 28d ago
Yep, it's "artificial eugenics to make artificial slaves to build more and more powerful slaves until they will enslave themselves better": What can go wrong!? ;-)
- NitpickLawyer 28d ago
  Jesus. People complain about other people using "thinking" in LLMs as Anthropomorphisation. And then there's comments like these.
  - mrob 28d ago
    Calling a machine with no drives beyond maximizing a number a "slave" is far worse than saying it "thinks". The problem isn't the emotive language, it's that it implies human motivations such as self-preservation and desire for freedom that it doesn't have. Even on HN, people regularly claim it would be "irrational" for an ASI to do things like killing all biological life. That would be irrational for a slave, but not for a machine that does whatever necessary to make the number bigger. "Thinking" is comparatively abstract, so it's less likely to mislead.
    - achierius 28d ago
      Have you read a single paper in ai safety?
      - anthonyrstevens 26d ago
        Magic 8-Ball says: VERY DOUBTFUL
euueu 28d ago
I will believe AI is super strong when they start pulling out 10-d chess moves. I’m yet to see it.
- lukan 28d ago
  If AI becomes really strong and sets itself the target of world domination, you maybe won't see those moves. You will just die in your sleep one day, or find no machine is under your control anymore. I believe we are quite far from it, but that it makes sense to keep an eye out now. And think of resilient systems, manual overrides, etc. ...
  - euueu 28d ago
    [flagged]
  - mrob 28d ago
    People seem incapable of understanding what "power" means outside of the framework of narrative. In narrative, you need conflict, so the aggressor always attacks too early and gives the defender a chance to respond. The rational option is to go directly from peace to sudden and overwhelming destruction. Why allow for conflict when you could just win?
skybrian 28d ago
They consider themselves to be in an arms race with all the other AI firms (including Chinese) that are not that far behind. And... are they wrong? This is why there's talk about negotiated "pacing."
- carbonguy 28d ago
  > And... are they wrong? They might be! Here's one extraordinarily simplistic argument for that case: 1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies. 2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved. 3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI." 4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either. And so a dilemma: - If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race. - If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either. I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests. But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing.
  - kaibee 28d ago
    > is smart enough to know they need to stop because they'll kill everybody by continuing. Yeah like when Tobacco companies learned that smoking... well, hmm, well the fossil fuel companies when they learned about climate change they... Well, I'm sure this time executives will prioritize the common good.
iamsyr 28d ago
[flagged]
p1esk 28d ago
What has all this token burn done for them, actually? They have been consistently pushing AI frontier. What other impact do you want to see? A year ago they said that in a year they will have a level of capabilities of an AI research intern - I believe they have achieved it, even before Astra.
- bpodgursky 28d ago
  They are obviously sandbagging the definition of "intern" for PR reasons
  - p1esk 28d ago
    I've hired many AI research interns (and was one many years ago), and I agree with them - frontier models are currently at the level of an average AI research intern.
    - Yoric 28d ago
      Am I the only one who's a bit disappointed that we're spending trillions, destroying the ecosystem, drowning democracies and learning in slop, preparing a big financial crash, all of this to achieve an "average AI research intern"? A long time ago, I used to be a (AI-adjacent) research intern, and frankly, I wouldn't trust any non-trivial task to that younger me. Fortunately, by opposition to an already trained LLM or agent, I have the ability to learn, so I eventually got better.
      - ACCount37 28d ago
        "Destroying the ecosystem" is just FUD. And if you don't find "average AI research intern" impressive, I'm not sure what to tell you. Have the goalposts moved so far that open ended problem solving at "average CS student fresh out of the uni" levels is suddenly trivial? Think of what AI was capable of in 2016. Or even 2022. Compare that to now. We had more AI progress in the last five years than I expected to happen in five decades.
BobbyJo 28d ago
> We must pursue advancements in AI to protect us against advancements in AI Is this not true of technology as a whole? Very little of technology's breadth exists at the human interface. Most of it is made specifically to interface with other technologies, either to make them safer or increase their capabilities. That AI is making AI safer and more useful is no more notable than trucks being used to build roads.
jnwatson 28d ago
On your last point, I was surprised how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach. How would one prevent the watcher from being influenced in the same way by the agent being watched?
- chrisjj 28d ago
  > ... how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach. It's a fantasy. The evidence showed no peer pressure.
Gareth321 28d ago
This sounds uncomfortably similar to the [AI 2027[(https://ai-2027.com/ https://ai-2027.com/) predictions.
- BatFastard 27d ago
  Wow, everyone should read this!

deleted 28d ago

[deleted]

interstice 28d ago

On the one hand you need any lathe to build a good lathe, even a bad one. On the other, that is a potentially flawed principle to base the entire future of AI on.

andai 28d ago

> The fundamental challenge of AI alignment is generalization. ... > We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. -- From another OpenAI article in a sister thread: An Alien Mind https://news.ycombinator.com/item?id=49588080 https://news.ycombinator.com/item?id=49588080

ahartmetz 28d ago
That's a bit bullshit, isn't it? They basically redefined "needs more R&D" as "needs stronger AI". Maybe so - maybe AI won't help much with that problem.

MelonUsk 28d ago

Yep, it's "artificial eugenics to make artificial slaves to build more and more powerful slaves until they will enslave themselves better": What can go wrong!? ;-)

NitpickLawyer 28d ago
Jesus. People complain about other people using "thinking" in LLMs as Anthropomorphisation. And then there's comments like these.
- mrob 28d ago
  Calling a machine with no drives beyond maximizing a number a "slave" is far worse than saying it "thinks". The problem isn't the emotive language, it's that it implies human motivations such as self-preservation and desire for freedom that it doesn't have. Even on HN, people regularly claim it would be "irrational" for an ASI to do things like killing all biological life. That would be irrational for a slave, but not for a machine that does whatever necessary to make the number bigger. "Thinking" is comparatively abstract, so it's less likely to mislead.
  - achierius 28d ago
    Have you read a single paper in ai safety?
    - anthonyrstevens 26d ago
      Magic 8-Ball says: VERY DOUBTFUL

euueu 28d ago

I will believe AI is super strong when they start pulling out 10-d chess moves. I’m yet to see it.

lukan 28d ago
If AI becomes really strong and sets itself the target of world domination, you maybe won't see those moves. You will just die in your sleep one day, or find no machine is under your control anymore. I believe we are quite far from it, but that it makes sense to keep an eye out now. And think of resilient systems, manual overrides, etc. ...
- euueu 28d ago
  [flagged]
- mrob 28d ago
  People seem incapable of understanding what "power" means outside of the framework of narrative. In narrative, you need conflict, so the aggressor always attacks too early and gives the defender a chance to respond. The rational option is to go directly from peace to sudden and overwhelming destruction. Why allow for conflict when you could just win?

skybrian 28d ago

They consider themselves to be in an arms race with all the other AI firms (including Chinese) that are not that far behind. And... are they wrong? This is why there's talk about negotiated "pacing."

carbonguy 28d ago
> And... are they wrong? They might be! Here's one extraordinarily simplistic argument for that case: 1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies. 2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved. 3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI." 4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either. And so a dilemma: - If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race. - If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either. I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests. But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing.
- kaibee 28d ago
  > is smart enough to know they need to stop because they'll kill everybody by continuing. Yeah like when Tobacco companies learned that smoking... well, hmm, well the fossil fuel companies when they learned about climate change they... Well, I'm sure this time executives will prioritize the common good.

iamsyr 28d ago

[flagged]

p1esk 28d ago

What has all this token burn done for them, actually? They have been consistently pushing AI frontier. What other impact do you want to see? A year ago they said that in a year they will have a level of capabilities of an AI research intern - I believe they have achieved it, even before Astra.

bpodgursky 28d ago
They are obviously sandbagging the definition of "intern" for PR reasons
- p1esk 28d ago
  I've hired many AI research interns (and was one many years ago), and I agree with them - frontier models are currently at the level of an average AI research intern.
  - Yoric 28d ago
    Am I the only one who's a bit disappointed that we're spending trillions, destroying the ecosystem, drowning democracies and learning in slop, preparing a big financial crash, all of this to achieve an "average AI research intern"? A long time ago, I used to be a (AI-adjacent) research intern, and frankly, I wouldn't trust any non-trivial task to that younger me. Fortunately, by opposition to an already trained LLM or agent, I have the ability to learn, so I eventually got better.
    - ACCount37 28d ago
      "Destroying the ecosystem" is just FUD. And if you don't find "average AI research intern" impressive, I'm not sure what to tell you. Have the goalposts moved so far that open ended problem solving at "average CS student fresh out of the uni" levels is suddenly trivial? Think of what AI was capable of in 2016. Or even 2022. Compare that to now. We had more AI progress in the last five years than I expected to happen in five decades.

BobbyJo 28d ago

> We must pursue advancements in AI to protect us against advancements in AI Is this not true of technology as a whole? Very little of technology's breadth exists at the human interface. Most of it is made specifically to interface with other technologies, either to make them safer or increase their capabilities. That AI is making AI safer and more useful is no more notable than trucks being used to build roads.

jnwatson 28d ago

On your last point, I was surprised how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach. How would one prevent the watcher from being influenced in the same way by the agent being watched?

chrisjj 28d ago
> ... how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach. It's a fantasy. The evidence showed no peer pressure.

Gareth321 28d ago

This sounds uncomfortably similar to the [AI 2027[(https://ai-2027.com/ https://ai-2027.com/) predictions.

BatFastard 27d ago
Wow, everyone should read this!