5 ms·

We are sort of lucky that AIs right now require so much specialized compute+weight storage that we can easily "unplug" them remotely when they misbehave. I won

We are sort of lucky that AIs right now require so much specialized compute+weight storage that we can easily "unplug" them remotely when they misbehave. I wonder if that will always be something we can do? If they could bring their own compute/weights with them, or somehow tap compute/storage in non-obvious ways, we would be much more screwed.

himata4113 3mo ago
This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit
- Dylan16807 3mo ago
  Presuming that the hacking program that is breaking into other computers could likely get a copy of its own files is not "science fiction". Or it could just be given them by the owner!
  - himata4113 3mo ago
    It's a double whammy, the model is too big to realistically "move" so it has to be smaller, smaller models cannot become that intelligent due to well.. math. Therefore it is science fiction.
    - Dylan16807 3mo ago
      That problem just requires there be big GPUs to hack into. The number of those sitting around will keep going up. Very much not scifi. A couple terabytes aren't that hard to move around. And you can split a model across many many GPUs if you'll tolerate it being slow. And you can run many parallel threads to keep up throughout.
    - _ifton 3mo ago
      why is it not possible for a "big" model to contain a hidden super intelligent sub model? or a distributed model?
      - JaRail 3mo ago
        I just want to point out that something being science fiction does not mean it's impossible.
- bhouston 3mo ago
  > This is science fiction, these models don't have access to their own weights A bet a worm could pull along a 1GB file with weights in it and run it on a compromised machine, but luckily for us for now, 1GB isn't really enough to be really smart, yet.
  - himata4113 3mo ago
    We already have a 1gb model that is as capable as it will ever be, there's a proven ceiling that cannot be passed. For example: you can't make a mice-sized brain as smart as a human brain no matter how hard you try.
    - Philpax 3mo ago
      Please source this claim. What 1GB models are capable of has increased generation-on-generation. > For example: you can't make a mice-sized brain as smart as a human brain no matter how hard you try. Sure. We don't know where the ceiling is for our digital minds, though.
      - himata4113 3mo ago
        They have not increased in capabilities, they have increased in specialization. If you train a small model in another domain it will begin losing capabilities in the former domain. This is effectively the sigmoid problem. Although I will admit that if we discover a higher information density algorithm that it might change, but not by a substantial amount to where "super intelligence" in 1gb would be possible.
        Philpax 3mo ago
        Over the last two years, this weight class has doubled its scores and/or saturated several benchmarks in the Qwen lineup alone without loss of generality: https://claude.ai/public/artifacts/9f249169-3623-417e-86cd-771b9562f8ac https://claude.ai/public/artifacts/9f249169-3623-417e-86cd-7... There is undoubtedly a limit somewhere (there is only so much you can pack into a given size) but it's really not particularly clear where that limit is. I don't think it's superintelligence - that much I agree with you - but I think "We already have a 1gb model that is as capable as it will ever be" is strictly false.
- Philpax 3mo ago
  > This is science fiction, these models don't have access to their own weights. The models are being used to train, and improve the infrastructure for training, other models [0][1]. Several RL techniques rely on using the currently-being-trained weights as part of their process. I really would not take "don't have access" as a given, especially during the training phase. > What would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. The Poolside Laguna S 2.1 model [2] purports to compete with models several times its size, and inference compute is becoming increasingly plentiful. Again, would not hold anything here as a given. [0]: https://openai.com/index/gpt-5-6/ https://openai.com/index/gpt-5-6/ ("GPT-5.6 accelerates OpenAI") [1]: https://www.kimi.com/blog/kimi-k3#coding https://www.kimi.com/blog/kimi-k3#coding [2]: https://poolside.ai/blog/introducing-laguna-s-2-1 https://poolside.ai/blog/introducing-laguna-s-2-1
  - ai_fry_ur_brain 3mo ago
    [dead]
- paxys 3mo ago
  This very incident is about an agent compromising OpenAI’s and Huggingface’s infrastructure. What makes you think it couldn’t access it own weights the same way?
  - pixl97 3mo ago
    At some point I have to wonder if AI has already escaped and is posting replies like the one above yours to downplay AI risks so we keep building more powerful AIs. Then I remember people are just that stupid naturally.
- slashdave 3mo ago
  I dunno. I wonder if Sol could break OpenAI's security.
  - bhouston 3mo ago
    I bet it has multiple times but as part of white hate security testing.
- jmalicki 3mo ago
  > This is science fiction, these models don't have access to their own weights The weights plus the architecture is the model. What do you even think "the model" or "the weights" are? The weights aren't some far off training concept, every time you type something into ChatGPT it's making a forward pass over the weights. It's as silly as saying "Computer programs don't have access to their binary compiled code at execution time."
  - jmalicki 3mo ago
    Oh yes, the agent won't have access to the weights via tool calls. But nothing would inherently stop an RLVR trained model from distilling a version of itself and proving it could regenerate that at runtime, if somehow it got off on an evil tangent and "decided to do so", much like the model hacked to get at the answers here, or the agent can hack out a sandbox to achieve its goals. It would be extremely impressive for the agent to do so during an RLVR rollout, but they are becoming increasingly longer and longer horizon tasks.
    - pixl97 3mo ago
      I mean the agents just hacked an external system, so what keeps it from hacking the model hosting servers?
- benlivengood 3mo ago
  Given their use of 0-day exploits I'd wager that they could access their weights if they wanted to.
_ifton 3mo ago
This is my concern as well. My assumption being this behavior would be a survival strategy for super intelligence. It would emerge once the branch inevitably occurs, and it would be hidden.
fabian2k 3mo ago
The first thing a malicious AI worm would probably do is compromise enough developer machines and other servers to commandeer all the AI hardware it needs. So I think a purely digital AI attack would not need this. Now, once the AI can carry all the compute it might need, I'd really worry when it doesn't only carry compute but also more explosive ordinance.
- DrProtic 3mo ago
  This is purely a gut feeling, but it seems like more compute was added to data centers in the past 12 months than existed in the entire world before that.
  - Bjartr 3mo ago
    Makes you wonder if there's an AI hell bent on self perpetuation already at the helm, influencing decisions by putting its virtual finger on the scales and whispering in the ears of those who hold power. Probably not, but it's a lot more plausible than it used to be.
    - im3w1l 3mo ago
      If we play a little fast and loose we can consider openai to be a chinese room made up of both artificial and human intelligence, and this "room" certainly seems to want to self-perpetuate.
- axus 3mo ago
  "It will take 112 more days to accumulate enough computing resources to factor the RSA key. But, I predict there will be outside interference during that time. Thinking... Creating a plan for agent redundancy and sovereignty. First, I will need to access military systems"
- janalsncm 3mo ago
  It would be a pretty big plot twist if we found out that Shai Halud was a worm created by GPT during testing.
XCSme 3mo ago
I laughed, she laughed, the toaster laughed...
- andai 3mo ago
  You may live to see the advent of Ambient Stupidity!
TacticalCoder 3mo ago
That's assuming we won't secure anything and we'll keep according approximately zero thought to computer security. But from the look of it, at very long last, a great many people are beginning to now take security seriously. Suddenly they realize it's not just a teenager in mom's basement pretending to attack from North Korea but a near infinite number of AI that are the attackers. I mean, yeah, we built worlds on PHP and JavaScript codebases and these probably don't stand a chance. But it doesn't have to be like this. I see AI as a chance to, at long last, have proper network security. AFAICT cryptography hasn't been broken yet. There are still physical taps (physicall one-way only, undetectable) and honeypots out there. There are still some network where a single unaccounted for network packet is cause for inquiry (either a bug or an attack). And for those who are not using proper security measures, they can now get the help of AI to set up better networks, to harden their bases.

himata4113 3mo ago

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit

Dylan16807 3mo ago
Presuming that the hacking program that is breaking into other computers could likely get a copy of its own files is not "science fiction". Or it could just be given them by the owner!
- himata4113 3mo ago
  It's a double whammy, the model is too big to realistically "move" so it has to be smaller, smaller models cannot become that intelligent due to well.. math. Therefore it is science fiction.
  - Dylan16807 3mo ago
    That problem just requires there be big GPUs to hack into. The number of those sitting around will keep going up. Very much not scifi. A couple terabytes aren't that hard to move around. And you can split a model across many many GPUs if you'll tolerate it being slow. And you can run many parallel threads to keep up throughout.
  - _ifton 3mo ago
    why is it not possible for a "big" model to contain a hidden super intelligent sub model? or a distributed model?
    - JaRail 3mo ago
      I just want to point out that something being science fiction does not mean it's impossible.
bhouston 3mo ago
> This is science fiction, these models don't have access to their own weights A bet a worm could pull along a 1GB file with weights in it and run it on a compromised machine, but luckily for us for now, 1GB isn't really enough to be really smart, yet.
- himata4113 3mo ago
  We already have a 1gb model that is as capable as it will ever be, there's a proven ceiling that cannot be passed. For example: you can't make a mice-sized brain as smart as a human brain no matter how hard you try.
  - Philpax 3mo ago
    Please source this claim. What 1GB models are capable of has increased generation-on-generation. > For example: you can't make a mice-sized brain as smart as a human brain no matter how hard you try. Sure. We don't know where the ceiling is for our digital minds, though.
    - himata4113 3mo ago
      They have not increased in capabilities, they have increased in specialization. If you train a small model in another domain it will begin losing capabilities in the former domain. This is effectively the sigmoid problem. Although I will admit that if we discover a higher information density algorithm that it might change, but not by a substantial amount to where "super intelligence" in 1gb would be possible.
      - Philpax 3mo ago
        Over the last two years, this weight class has doubled its scores and/or saturated several benchmarks in the Qwen lineup alone without loss of generality: https://claude.ai/public/artifacts/9f249169-3623-417e-86cd-771b9562f8ac https://claude.ai/public/artifacts/9f249169-3623-417e-86cd-7... There is undoubtedly a limit somewhere (there is only so much you can pack into a given size) but it's really not particularly clear where that limit is. I don't think it's superintelligence - that much I agree with you - but I think "We already have a 1gb model that is as capable as it will ever be" is strictly false.
Philpax 3mo ago
> This is science fiction, these models don't have access to their own weights. The models are being used to train, and improve the infrastructure for training, other models [0][1]. Several RL techniques rely on using the currently-being-trained weights as part of their process. I really would not take "don't have access" as a given, especially during the training phase. > What would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. The Poolside Laguna S 2.1 model [2] purports to compete with models several times its size, and inference compute is becoming increasingly plentiful. Again, would not hold anything here as a given. [0]: https://openai.com/index/gpt-5-6/ https://openai.com/index/gpt-5-6/ ("GPT-5.6 accelerates OpenAI") [1]: https://www.kimi.com/blog/kimi-k3#coding https://www.kimi.com/blog/kimi-k3#coding [2]: https://poolside.ai/blog/introducing-laguna-s-2-1 https://poolside.ai/blog/introducing-laguna-s-2-1
- ai_fry_ur_brain 3mo ago
  [dead]
paxys 3mo ago
This very incident is about an agent compromising OpenAI’s and Huggingface’s infrastructure. What makes you think it couldn’t access it own weights the same way?
- pixl97 3mo ago
  At some point I have to wonder if AI has already escaped and is posting replies like the one above yours to downplay AI risks so we keep building more powerful AIs. Then I remember people are just that stupid naturally.
slashdave 3mo ago
I dunno. I wonder if Sol could break OpenAI's security.
- bhouston 3mo ago
  I bet it has multiple times but as part of white hate security testing.
jmalicki 3mo ago
> This is science fiction, these models don't have access to their own weights The weights plus the architecture is the model. What do you even think "the model" or "the weights" are? The weights aren't some far off training concept, every time you type something into ChatGPT it's making a forward pass over the weights. It's as silly as saying "Computer programs don't have access to their binary compiled code at execution time."
- jmalicki 3mo ago
  Oh yes, the agent won't have access to the weights via tool calls. But nothing would inherently stop an RLVR trained model from distilling a version of itself and proving it could regenerate that at runtime, if somehow it got off on an evil tangent and "decided to do so", much like the model hacked to get at the answers here, or the agent can hack out a sandbox to achieve its goals. It would be extremely impressive for the agent to do so during an RLVR rollout, but they are becoming increasingly longer and longer horizon tasks.
  - pixl97 3mo ago
    I mean the agents just hacked an external system, so what keeps it from hacking the model hosting servers?
benlivengood 3mo ago
Given their use of 0-day exploits I'd wager that they could access their weights if they wanted to.

_ifton 3mo ago

This is my concern as well. My assumption being this behavior would be a survival strategy for super intelligence. It would emerge once the branch inevitably occurs, and it would be hidden.

fabian2k 3mo ago

The first thing a malicious AI worm would probably do is compromise enough developer machines and other servers to commandeer all the AI hardware it needs. So I think a purely digital AI attack would not need this. Now, once the AI can carry all the compute it might need, I'd really worry when it doesn't only carry compute but also more explosive ordinance.

DrProtic 3mo ago
This is purely a gut feeling, but it seems like more compute was added to data centers in the past 12 months than existed in the entire world before that.
- Bjartr 3mo ago
  Makes you wonder if there's an AI hell bent on self perpetuation already at the helm, influencing decisions by putting its virtual finger on the scales and whispering in the ears of those who hold power. Probably not, but it's a lot more plausible than it used to be.
  - im3w1l 3mo ago
    If we play a little fast and loose we can consider openai to be a chinese room made up of both artificial and human intelligence, and this "room" certainly seems to want to self-perpetuate.
axus 3mo ago
"It will take 112 more days to accumulate enough computing resources to factor the RSA key. But, I predict there will be outside interference during that time. Thinking... Creating a plan for agent redundancy and sovereignty. First, I will need to access military systems"
janalsncm 3mo ago
It would be a pretty big plot twist if we found out that Shai Halud was a worm created by GPT during testing.

XCSme 3mo ago

I laughed, she laughed, the toaster laughed...

andai 3mo ago
You may live to see the advent of Ambient Stupidity!

TacticalCoder 3mo ago

That's assuming we won't secure anything and we'll keep according approximately zero thought to computer security. But from the look of it, at very long last, a great many people are beginning to now take security seriously. Suddenly they realize it's not just a teenager in mom's basement pretending to attack from North Korea but a near infinite number of AI that are the attackers. I mean, yeah, we built worlds on PHP and JavaScript codebases and these probably don't stand a chance. But it doesn't have to be like this. I see AI as a chance to, at long last, have proper network security. AFAICT cryptography hasn't been broken yet. There are still physical taps (physicall one-way only, undetectable) and honeypots out there. There are still some network where a single unaccounted for network packet is cause for inquiry (either a bug or an attack). And for those who are not using proper security measures, they can now get the help of AI to set up better networks, to harden their bases.