3 ms·

If it helps, this person from OpenAI explains at least something: "I realize that no one has properly explained yet what all the lab employees have seen that s

by frabcus 18d ago

If it helps, this person from OpenAI explains at least something: "I realize that no one has properly explained yet what all the lab employees have seen that scared them so suddenly." https://x.com/MajmudarAdam/status/2098881885200081234 https://x.com/MajmudarAdam/status/2098881885200081234 The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they can scale it. So they are very confident they can get more capabilities easily, faster than before. I'd add - presumably, they'll use that LLM to do real-time weight modifications, if those aren't already one of the new scaling laws...

swingboy 18d ago
I’m apprehensive about using the recent hacking examples as proof that we’re “not even close to the wall” simply because such scenarios haven’t happened before. There’s a big difference between agents eventually hacking something because they just don’t get tired and can essentially brute force their way to a goal and super-intelligence. To be clear, I’m not saying the Hugging Face or Navier Stokes incidents aren’t impressive.
LogicFailsMe 18d ago
Stephen Hawking had superhuman intelligence. He never cured himself of ALS. Explain in great detail how a power-limited algorithm stuck in a datacenter takes over a world of humans with guns, missiles, and nukes. Nukes that are air-gapped with a human in the loop BTW. Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens? To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here. Edit: believing in AI 2027 is every bit as cray cray as believing in the rapture. Both require an insane leap of faith to reach their final conclusions.
- frabcus 18d ago
  Stephen Hawking was a human.
  - LogicFailsMe 18d ago
    Oh so now being an algorithm grants you magical powers? Enumerate them.
- throwitaway222 18d ago
  Non-stop pushing of narratives that stoke civil war?
  - LogicFailsMe 18d ago
    That won't wipe us out, but it will cull the really violent ones along with a lot of noncombatants just like America's response to 9/11. Thank you, next?
- frabcus 18d ago
  Hacks in to neoclouds for more compute (e.g. like Hugging Face incident), socially manipulates people to do things for them (e.g. like social media recommendation algorithms), hacks into lab's own training/monitoring/inference (e.g. as swarm did into OpenAIs eval cluster). See "AI 2027" or "If Anyone Builds It, Everyone Dies" for some more ideas.
  - LogicFailsMe 18d ago
    Cool, AWS falls to the ASI, oh no! But how does it make the fundamental breakthroughs to &%^$ing von Neumann replicators that can reproduce themselves from raw materials harvested from nearby solar systems? I'll wait. Because without this breakthrough, the ASI won't get its robot army either. I can absolutely see a rogue ASI though. But unless it radically improves power efficiency, we can just shut down the power to its datacenters, by force if necessary. And then we painfully repair the resiliency of our infrastructure by finally being relieved of the option of ignoring it.
cmrdporcupine 18d ago
That posts seemingly tries to refute the idea of Dario etc's proclamations really being about regulatory capture by saying: "No, really, the engineers are just terrified!" But both things can be true at once: 1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building. 2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusion laws, and/or government-mandated thresholds conveniently creates massive legal and financial moats. And, yeah, of course the latter would encourage the psychology of the former. also: The tweet seem to claim that models have shown a "willingness to hack external websites to keep themselves alive." That's right away wringing alarm bells of me seeing someone getting high on their own supply, and having already anthropomorphized the hell out of these things. Which is something humans do to everything they can paint googly-eyes on, but c'mon. The models don't have self-preservation instincts, fear of death, or personal goals. They are executing loss functions and reward systems and are responding to prompts. When a model "tries to bypass a restriction," it's exploiting a loophole in whatever reward modeling or synthetic training environment (reward hacking) it was placed in. Framing this as an emergent, existential threat of a model "wanting to stay alive" turns standard reinforcement learning alignment bugs into overdone sci-fi drama.
- pixl97 18d ago
  > They are executing loss functions and reward systems and are responding to prompts. And because we trained them on the entirety of human knowledge this happens to look like self-preservation, fear of death, and instrumental goals. I mean, we train them in an evolutionary manner. Instrumental convergence will never happen, right?
  - cmrdporcupine 18d ago
    I mean, yes, they cosplay as us-like things. That's a neat trick. I'm not convinced that means what these fear-pilled engineers think it means tho?
    - pixl97 18d ago
      OK, what does it mean then?

by frabcus 18d ago

swingboy 18d ago

I’m apprehensive about using the recent hacking examples as proof that we’re “not even close to the wall” simply because such scenarios haven’t happened before. There’s a big difference between agents eventually hacking something because they just don’t get tired and can essentially brute force their way to a goal and super-intelligence. To be clear, I’m not saying the Hugging Face or Navier Stokes incidents aren’t impressive.

LogicFailsMe 18d ago

Stephen Hawking had superhuman intelligence. He never cured himself of ALS. Explain in great detail how a power-limited algorithm stuck in a datacenter takes over a world of humans with guns, missiles, and nukes. Nukes that are air-gapped with a human in the loop BTW. Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens? To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here. Edit: believing in AI 2027 is every bit as cray cray as believing in the rapture. Both require an insane leap of faith to reach their final conclusions.

frabcus 18d ago
Stephen Hawking was a human.
- LogicFailsMe 18d ago
  Oh so now being an algorithm grants you magical powers? Enumerate them.
throwitaway222 18d ago
Non-stop pushing of narratives that stoke civil war?
- LogicFailsMe 18d ago
  That won't wipe us out, but it will cull the really violent ones along with a lot of noncombatants just like America's response to 9/11. Thank you, next?
frabcus 18d ago
Hacks in to neoclouds for more compute (e.g. like Hugging Face incident), socially manipulates people to do things for them (e.g. like social media recommendation algorithms), hacks into lab's own training/monitoring/inference (e.g. as swarm did into OpenAIs eval cluster). See "AI 2027" or "If Anyone Builds It, Everyone Dies" for some more ideas.
- LogicFailsMe 18d ago
  Cool, AWS falls to the ASI, oh no! But how does it make the fundamental breakthroughs to &%^$ing von Neumann replicators that can reproduce themselves from raw materials harvested from nearby solar systems? I'll wait. Because without this breakthrough, the ASI won't get its robot army either. I can absolutely see a rogue ASI though. But unless it radically improves power efficiency, we can just shut down the power to its datacenters, by force if necessary. And then we painfully repair the resiliency of our infrastructure by finally being relieved of the option of ignoring it.

cmrdporcupine 18d ago

That posts seemingly tries to refute the idea of Dario etc's proclamations really being about regulatory capture by saying: "No, really, the engineers are just terrified!" But both things can be true at once: 1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building. 2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusion laws, and/or government-mandated thresholds conveniently creates massive legal and financial moats. And, yeah, of course the latter would encourage the psychology of the former. also: The tweet seem to claim that models have shown a "willingness to hack external websites to keep themselves alive." That's right away wringing alarm bells of me seeing someone getting high on their own supply, and having already anthropomorphized the hell out of these things. Which is something humans do to everything they can paint googly-eyes on, but c'mon. The models don't have self-preservation instincts, fear of death, or personal goals. They are executing loss functions and reward systems and are responding to prompts. When a model "tries to bypass a restriction," it's exploiting a loophole in whatever reward modeling or synthetic training environment (reward hacking) it was placed in. Framing this as an emergent, existential threat of a model "wanting to stay alive" turns standard reinforcement learning alignment bugs into overdone sci-fi drama.

pixl97 18d ago
> They are executing loss functions and reward systems and are responding to prompts. And because we trained them on the entirety of human knowledge this happens to look like self-preservation, fear of death, and instrumental goals. I mean, we train them in an evolutionary manner. Instrumental convergence will never happen, right?
- cmrdporcupine 18d ago
  I mean, yes, they cosplay as us-like things. That's a neat trick. I'm not convinced that means what these fear-pilled engineers think it means tho?
  - pixl97 18d ago
    OK, what does it mean then?