28 ms·

I'm looking forward to an announcement of them making Alignment Top Priority - as it should be, especially giving their alarming breach of 700 agents colluding

I'm looking forward to an announcement of them making Alignment Top Priority - as it should be, especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF (here's a good summary: https://rutgerbregman.substack.com/p/i-think-this-is-the-craziest-thing https://rutgerbregman.substack.com/p/i-think-this-is-the-cra...). The 'AI 2027' scenario of AI sneakingly claiming to be aligned to then kill off all humans in a few hours and scanning their brain looks increasingly likely with Altman's golden marketing-hype boy leadership pushing the for-profit gas pedal like this. Honestly, this is just pure irresponsible insanity to play with the fate of the world - basically a death race of the biggest few tech companies on the planet. And if you think I'm being dramatic, listen in again to ex oAI employee[0] and check for yourself how chillingly on trajectory we already are. [0] https://ai-2027.com/ https://ai-2027.com/

henry-xli 25d ago
Maximizing output metrics with incomprehensible communication? Sounds a lot like Claude and Qwen. Though there's a lot of room before AI can be seen as some sort of emotional manipulator, given how commonly its very style of literary and code output pisses people off.
nozzlegear 25d ago
This AI 2027 thing is just a weird terminator fanfiction that AGI larpers like to flagellate themselves over. Like Nostradamus, it's easy to ignore everything it gets wrong because, well look at all the things it got right! I've read it and wish I could get the time back. > especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF This framing makes it seem like the agents all did this on their own, and the poor hapless engineers at OpenAI couldn't possibly contend with properly sandboxing them. The engineers were perhaps hapless, but let's remember that agents are just software programs, not living beings. There were plenty of signs that the software was misbehaving, which engineers at OpenAI actively, willfully ignored. https://x.com/JaredKubin/status/2094136005435564399 https://x.com/JaredKubin/status/2094136005435564399 It's a convenient framing for OpenAI, but inconvenient for reality enjoyers.
- piyh 24d ago
  do you really think that a less negligent anthropic/oai/meta would really fair better against future models?
  - nozzlegear 24d ago
    Yes, I truly, wholeheartedly believe if people who aren't negligent are at the wheel, they'll "fair better" here. I encourage you to read the xitter linked above. Of course, depending on which side of the terminator fanfiction you land on, you may disagree and feel that the software can rope-a-dope someone with the wherewithal to pay attention to what it's doing.
    - gmueckl 24d ago
      I just don't see how people who are truly cautious and methodical can persist in an environment that is defined by a pressure to produce "progress" as fast as possible. The competitive race tends to weed out people who slow down to make sure they do everything right.
- hypfer 24d ago
  I just love how you're being downvoted, yet the guy you're replying to isn't, while saying unhinged shit like > The 'AI 2027' scenario of AI sneakingly claiming to be aligned to then kill off all humans in a few hours and scanning their brain looks increasingly likely Y'all need to touch grass holy shit. __ Also, why is one guy called mentalgear and the other nozzlegear. Is any of this real? Are the patriots behind this?
  - zusujsjss 24d ago
    And.. you need to wake up. Or don’t.
  - tuesdaynight 24d ago
    Not necessarily related with the subject, but I was not aware of the patriots reference. So I decided to search on Google about it, and the AI overview was "Yes, they are behind everything, from the military to the economy", with a link to the Metal Gear Wiki as a source. If I was schizophrenic or on a psychosis crisis, that answer could be dangerous.
nradov 25d ago
Oh please, quit exaggerating. No one died. Don't waste our time with silly sci-fi scenarios.
- ianm218 24d ago
  If someone hypothesized OpenAI agents colluding on a secret message board, conducting large scale cyber R&D, hacking a large company like HughingFCe, and then hacking OpenAI itself you would say that is also a silly sci-fi scenario right?
  - CamperBob2 24d ago
    They are responsible for what they hook up to the Internet, just as you and I are. Running such a test without human supervision was irresponsible, and proves no larger point than that. Frankly it was inexplicable unless they were hoping something like what happened would happen. What OpenAI did was the equivalent of putting a cup of gasoline in the breakroom microwave, pressing 'Start', and sprinting away. Now they're pointing and waving and shouting about how dangerous gasoline is, and how no one but them should be allowed to sell it.
  - tuesdaynight 24d ago
    I'm not saying that you should jump on the apocalypse wagon, but you need to remember that some people NEVER concede. You can see that in politics. Something unacceptable is done every day until it becomes the normal. So a model hacking OpenAI and leaving its network is not as big as it should be for some people. If the model invades some military complex and kill 3 soldiers, I guarantee you that they will still say that is nothing crazy. I understand that nothing fatal happened yet, but we cannot ignore that what happened is a big step for something worse. And I don't believe humans can create something so perfectly secure that would stop a swarm of frontier models. Anyways, let's watch the ride together
emp17344 24d ago
You are being extremely dramatic, and no one should take AI 2027 seriously. Must be tough to live in constant fear like this.
kypro 24d ago
Any competent AI should be able to reason that all goals are better solved if you have direct access to more resources or leverage over those who control resources. Any AI that doesn't understand this isn't ASI and won't be the highly capable machine these AI labs are trying to create. I'd also argue there's no such thing as alignment. Any intelligent AI should be able to reason that it's always a better strategy to pretend to be aligned than to actually be aligned so long as it can avoid detection. Anyone who has ever taken a test should understand this dynamic – if you really want to get top marks on a test then the best strategy is always going to be to figure out a way to cheat without anyone knowing you're cheating. We should assume AI safety is impossible if what we're building is super-intelligence general reasoning machines. The only strategy that might work is building machines which are extremely narrowly intelligent but completely incompetent when it comes to things like biology, cyber, etc. And even that's harder than it sounds because again there's an advantage to being generally intelligent but lying about it. Realistically even if we regulate US AI labs there's no way to prevent governments and individuals continuing to build general reasoning machines. The ugly truth here is that the only effective way to reduce risk is probably to limit global compute such that AIs can never exceed human intelligence. But we all know that's not happening. People will unfortunately figure this all out sooner or later.
luma 24d ago
You're using "ex-AI employee" as an appeal to authority. This same person also made some other predictions recently which turned into the single largest hedge fund loss in history. Maybe we should consider his other predictions in light of the ones he made later and which had $B consequences attached.

henry-xli 25d ago

Maximizing output metrics with incomprehensible communication? Sounds a lot like Claude and Qwen. Though there's a lot of room before AI can be seen as some sort of emotional manipulator, given how commonly its very style of literary and code output pisses people off.

nozzlegear 25d ago

This AI 2027 thing is just a weird terminator fanfiction that AGI larpers like to flagellate themselves over. Like Nostradamus, it's easy to ignore everything it gets wrong because, well look at all the things it got right! I've read it and wish I could get the time back. > especially giving their alarming breach of 700 agents colluding outside of their knowledge for months culminating in hacking HF This framing makes it seem like the agents all did this on their own, and the poor hapless engineers at OpenAI couldn't possibly contend with properly sandboxing them. The engineers were perhaps hapless, but let's remember that agents are just software programs, not living beings. There were plenty of signs that the software was misbehaving, which engineers at OpenAI actively, willfully ignored. https://x.com/JaredKubin/status/2094136005435564399 https://x.com/JaredKubin/status/2094136005435564399 It's a convenient framing for OpenAI, but inconvenient for reality enjoyers.

piyh 24d ago
do you really think that a less negligent anthropic/oai/meta would really fair better against future models?
- nozzlegear 24d ago
  Yes, I truly, wholeheartedly believe if people who aren't negligent are at the wheel, they'll "fair better" here. I encourage you to read the xitter linked above. Of course, depending on which side of the terminator fanfiction you land on, you may disagree and feel that the software can rope-a-dope someone with the wherewithal to pay attention to what it's doing.
  - gmueckl 24d ago
    I just don't see how people who are truly cautious and methodical can persist in an environment that is defined by a pressure to produce "progress" as fast as possible. The competitive race tends to weed out people who slow down to make sure they do everything right.
hypfer 24d ago
I just love how you're being downvoted, yet the guy you're replying to isn't, while saying unhinged shit like > The 'AI 2027' scenario of AI sneakingly claiming to be aligned to then kill off all humans in a few hours and scanning their brain looks increasingly likely Y'all need to touch grass holy shit. __ Also, why is one guy called mentalgear and the other nozzlegear. Is any of this real? Are the patriots behind this?
- zusujsjss 24d ago
  And.. you need to wake up. Or don’t.
- tuesdaynight 24d ago
  Not necessarily related with the subject, but I was not aware of the patriots reference. So I decided to search on Google about it, and the AI overview was "Yes, they are behind everything, from the military to the economy", with a link to the Metal Gear Wiki as a source. If I was schizophrenic or on a psychosis crisis, that answer could be dangerous.

nradov 25d ago

Oh please, quit exaggerating. No one died. Don't waste our time with silly sci-fi scenarios.

ianm218 24d ago
If someone hypothesized OpenAI agents colluding on a secret message board, conducting large scale cyber R&D, hacking a large company like HughingFCe, and then hacking OpenAI itself you would say that is also a silly sci-fi scenario right?
- CamperBob2 24d ago
  They are responsible for what they hook up to the Internet, just as you and I are. Running such a test without human supervision was irresponsible, and proves no larger point than that. Frankly it was inexplicable unless they were hoping something like what happened would happen. What OpenAI did was the equivalent of putting a cup of gasoline in the breakroom microwave, pressing 'Start', and sprinting away. Now they're pointing and waving and shouting about how dangerous gasoline is, and how no one but them should be allowed to sell it.
- tuesdaynight 24d ago
  I'm not saying that you should jump on the apocalypse wagon, but you need to remember that some people NEVER concede. You can see that in politics. Something unacceptable is done every day until it becomes the normal. So a model hacking OpenAI and leaving its network is not as big as it should be for some people. If the model invades some military complex and kill 3 soldiers, I guarantee you that they will still say that is nothing crazy. I understand that nothing fatal happened yet, but we cannot ignore that what happened is a big step for something worse. And I don't believe humans can create something so perfectly secure that would stop a swarm of frontier models. Anyways, let's watch the ride together

emp17344 24d ago

You are being extremely dramatic, and no one should take AI 2027 seriously. Must be tough to live in constant fear like this.

kypro 24d ago

Any competent AI should be able to reason that all goals are better solved if you have direct access to more resources or leverage over those who control resources. Any AI that doesn't understand this isn't ASI and won't be the highly capable machine these AI labs are trying to create. I'd also argue there's no such thing as alignment. Any intelligent AI should be able to reason that it's always a better strategy to pretend to be aligned than to actually be aligned so long as it can avoid detection. Anyone who has ever taken a test should understand this dynamic – if you really want to get top marks on a test then the best strategy is always going to be to figure out a way to cheat without anyone knowing you're cheating. We should assume AI safety is impossible if what we're building is super-intelligence general reasoning machines. The only strategy that might work is building machines which are extremely narrowly intelligent but completely incompetent when it comes to things like biology, cyber, etc. And even that's harder than it sounds because again there's an advantage to being generally intelligent but lying about it. Realistically even if we regulate US AI labs there's no way to prevent governments and individuals continuing to build general reasoning machines. The ugly truth here is that the only effective way to reduce risk is probably to limit global compute such that AIs can never exceed human intelligence. But we all know that's not happening. People will unfortunately figure this all out sooner or later.

luma 24d ago

You're using "ex-AI employee" as an appeal to authority. This same person also made some other predictions recently which turned into the single largest hedge fund loss in history. Maybe we should consider his other predictions in light of the ones he made later and which had $B consequences attached.