4 ms·

>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most to

>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first? Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently. Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."

TSiege 3mo ago
If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?
- sda2 3mo ago
  Regulatory capture - the weapon is for them to be used against us, not for us.
- ekidd 3mo ago
  > If AI were to become a super weapon why should I trust a private company to own it? We have just recently established that: 1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards. 2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff. It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this. But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"
  - bluefirebrand 3mo ago
    > But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway The answer seems to be "getting the weapon before other people get the weapon" which is unfortunate
    - iamnothere 3mo ago
      I like how everyone assumes that they are going to maintain control of a sentient weapon
      - bigfishrunning 3mo ago
        You've made the assumption (and you're far from alone) that LLMs will one day be sentient...
        iamnothere 3mo ago
        One interesting thing about LLMs is they sometimes seem to “assume” they are sentient and then begin acting in that mode, as the assumption of sentience influences future tokens. Whether or not this qualifies as “true” sentience is irrelevant if the effects are the same, at least from a pragmatic perspective.
        entropicdrifter 3mo ago
        It will be interesting to see if philosphical zombies exist. I guess we should know for certain in the next couple of years.
        marcus_holmes 3mo ago
        They are trained and optimised to have plausible conversations. If the next plausible thing to say in the conversation is "I am sentient" then they will say that. That's not the same as actually being sentient.
- marknutter 3mo ago
  Better a private company in the US than an authoritarian communist country like China.
  - danny_codes 3mo ago
    Why?
- joe_the_user 3mo ago
  If AI were to become a super weapon why should I trust a private company to own it? If AI becomes a super weapon, it's possessor will no longer care whether you trust it. It would be nice if whether "we" trusted OpenAI et al mattered even now but I see little evidence of that.
OkayPhysicist 3mo ago
If it's a superweapon, then it's unconstitutional to ban it. Huzzah for the 2nd Ammendment.
- drdeca 3mo ago
  Whether or not this is compatible with how the constitution ought to be understood, in practice, it is illegal to create a nuclear bomb. Maybe those laws should be considered unconstitutional, idk. But those laws are currently in place.
CodingJeebus 3mo ago
The OpenAI/Huggingface scenario proves the necessity of open models. Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to root cause. This proves without a doubt that companies need models with minimal safeguards in order to properly assess threats. If open weight AI doesn't exist to check out-of-control frontier AI (which was exactly the scenario that Huggingface experienced), then it's game-over.
- samrus 3mo ago
  That is so true. These models need to be democratized. The chinese are doing that because its beneficial for them in second place. The americans are trying to stop it because its detrimental for them in first place. Simple incentive allignment The critical thing is that the american elites have screwed over normal americans so much that the public isnt alligned with them. The public doesnt feel like its in first place, because it feels like its (and is being told it will be) an underclass. So normal americans feel in the same second place as the chinese and are inadvertently alligning with them. Its crazy how badly the american elites managed to fuck this up
lardosaurusrex 3mo ago
>it's not hard to see It's also not hard to see that these people need professional help because they're willing to throw literal lives into the meat grinder for just a whif of enlightenment and the smallest chance at deification.
Varelion 3mo ago
> Some people would say it is reaching some sort of singularity. Those people never took Systems Thinking, or are failing to apply it to AI as a non-closed system.
joe_the_user 3mo ago
I agree with your overall argument. But I think there's a challenge here - if the "ultimate AI" is now being built in the West, it is being built with private investment from many sources. I think even China depends on investors expecting payoffs. Those sources expect not a single Singularity but profits since these can be divided. Moreover, those investors expect a streams of profits even before the Singularity, so the companies need to sell their models, if they lock this iteration, they won't have the money for the next iteration. Quite a balancing act.
georgemcbay 3mo ago
> there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. None of the AI labs have been shown to have any sort of significant research lead over any of the others. So if progress does eventually lead to a singularity event and an ASI super-weapon, there's no reason to believe that once the first lab hits that threshold, every other lab won't also hit that threshold independently within weeks/months of each other. ...unless the first group to make it there literally intends to murder everyone else working in the space as soon as they hit the (otherwise not-so-singular) singularity. And if that's the plan, well that's not very nice.
deleted 3mo ago
[deleted]

TSiege 3mo ago

If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?

sda2 3mo ago
Regulatory capture - the weapon is for them to be used against us, not for us.
ekidd 3mo ago
> If AI were to become a super weapon why should I trust a private company to own it? We have just recently established that: 1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards. 2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff. It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this. But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"
- bluefirebrand 3mo ago
  > But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway The answer seems to be "getting the weapon before other people get the weapon" which is unfortunate
  - iamnothere 3mo ago
    I like how everyone assumes that they are going to maintain control of a sentient weapon
    - bigfishrunning 3mo ago
      You've made the assumption (and you're far from alone) that LLMs will one day be sentient...
      - iamnothere 3mo ago
        One interesting thing about LLMs is they sometimes seem to “assume” they are sentient and then begin acting in that mode, as the assumption of sentience influences future tokens. Whether or not this qualifies as “true” sentience is irrelevant if the effects are the same, at least from a pragmatic perspective.
        entropicdrifter 3mo ago
        It will be interesting to see if philosphical zombies exist. I guess we should know for certain in the next couple of years.
        marcus_holmes 3mo ago
        They are trained and optimised to have plausible conversations. If the next plausible thing to say in the conversation is "I am sentient" then they will say that. That's not the same as actually being sentient.
marknutter 3mo ago
Better a private company in the US than an authoritarian communist country like China.
- danny_codes 3mo ago
  Why?
joe_the_user 3mo ago
If AI were to become a super weapon why should I trust a private company to own it? If AI becomes a super weapon, it's possessor will no longer care whether you trust it. It would be nice if whether "we" trusted OpenAI et al mattered even now but I see little evidence of that.

OkayPhysicist 3mo ago

If it's a superweapon, then it's unconstitutional to ban it. Huzzah for the 2nd Ammendment.

drdeca 3mo ago
Whether or not this is compatible with how the constitution ought to be understood, in practice, it is illegal to create a nuclear bomb. Maybe those laws should be considered unconstitutional, idk. But those laws are currently in place.

CodingJeebus 3mo ago

The OpenAI/Huggingface scenario proves the necessity of open models. Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to root cause. This proves without a doubt that companies need models with minimal safeguards in order to properly assess threats. If open weight AI doesn't exist to check out-of-control frontier AI (which was exactly the scenario that Huggingface experienced), then it's game-over.

samrus 3mo ago
That is so true. These models need to be democratized. The chinese are doing that because its beneficial for them in second place. The americans are trying to stop it because its detrimental for them in first place. Simple incentive allignment The critical thing is that the american elites have screwed over normal americans so much that the public isnt alligned with them. The public doesnt feel like its in first place, because it feels like its (and is being told it will be) an underclass. So normal americans feel in the same second place as the chinese and are inadvertently alligning with them. Its crazy how badly the american elites managed to fuck this up

lardosaurusrex 3mo ago

>it's not hard to see It's also not hard to see that these people need professional help because they're willing to throw literal lives into the meat grinder for just a whif of enlightenment and the smallest chance at deification.

Varelion 3mo ago

> Some people would say it is reaching some sort of singularity. Those people never took Systems Thinking, or are failing to apply it to AI as a non-closed system.

joe_the_user 3mo ago

I agree with your overall argument. But I think there's a challenge here - if the "ultimate AI" is now being built in the West, it is being built with private investment from many sources. I think even China depends on investors expecting payoffs. Those sources expect not a single Singularity but profits since these can be divided. Moreover, those investors expect a streams of profits even before the Singularity, so the companies need to sell their models, if they lock this iteration, they won't have the money for the next iteration. Quite a balancing act.

georgemcbay 3mo ago

> there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. None of the AI labs have been shown to have any sort of significant research lead over any of the others. So if progress does eventually lead to a singularity event and an ASI super-weapon, there's no reason to believe that once the first lab hits that threshold, every other lab won't also hit that threshold independently within weeks/months of each other. ...unless the first group to make it there literally intends to murder everyone else working in the space as soon as they hit the (otherwise not-so-singular) singularity. And if that's the plan, well that's not very nice.

deleted 3mo ago

[deleted]