5 ms·

> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.

by pu_pe 27d ago

> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI. So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve. Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.

dgellow 27d ago
Yep. Such a disgusting industry. They created the arm race, push for the arm race, put themselves in position to benefit from the arm race
- estearum 27d ago
  The entire problem with arms races is that any individual entity cannot avoid participating. sama and his cadre are uniquely evil captains in this race, but they're completely replaceable and the dynamic would remain the same.
  - 0xDEAFBEAD 27d ago
    They could work to coordinate an end to the arms race.
    - estearum 27d ago
      Uhhh... the mechanism by which they'd do that is via government regulation. Most of the frontier labs have been openly requesting regulation: "the structure of this competition is not suitable for the development of this particular technology" The reactions vary from: 1. China will do it anyway (need some supranational governance scheme) 2. This is just an attempt at regulatory capture 3. This is just marketing 4. Regulation is bad mmmmkay
- wartywhoa23 27d ago
  Just like the rest of military stuff.
  - goatlover 27d ago
    "A strange game. The only winning move is not to play."
  - dgellow 27d ago
    But done by corporations, and selling that service to the general public, including their competitors and adversary countries
- mrshadowgoose 27d ago
  The arms race is an intrinsic game theoretical property of a multi-adversarial-actor scenario involving exponential growth of a universally potent technology. It's almost certainly winner-take-all, on a global scale, which behooves everyone to participate. And no, I don't think it will end well.
  - reasonableklout 27d ago
    The actors here are states or corporations embedded in societies that risk growing popular backlash against the technology. The other factor is that if it is truly an "alien mind", racing incurs risks to all players. In game-theoretic terms it may be more like a stag hunt than a prisoner's dilemma. In which case cooperation is an equilibrium.
  - nradov 27d ago
    No previous technological arms race has ever concluded with a single winner.
skybrian 27d ago
"Defensive systems" can be interpreted broadly to include cybersecurity. But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong. Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D? Individuals can quit, but for society, getting out of an arms race is harder than just quitting. You don't get to be Switzerland without having a strong defensive position and the right foreign relations. But there's at least talk about "pacing" and that's a start.
- Fraterkes 27d ago
  The point of the comment you're responding to is that it is at this point hardly an arms race: The frontier of ai development is happening completely inside 2 American companies, the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models. So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party. An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
  - throwthrowuknow 27d ago
    Development and improvement of nuclear weapons was an entirely American project until the technology was exfiltrated and then it became an instant arms race. That cat is already out of the bag with LLMs. Distillation is just the fastest way to keep pace but that in no way prevents other countries and actors from doing it the hard way. An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s a race to keep pace or retake the lead position which can oscillate between the parties involved indefinitely. The other option is to agree to make no further progress or to disarm.
    - goatlover 27d ago
      There were prominent scientists like Oppenheimer who did not think it needed to become an arms race, and campaigned against that. But there were others like Teller and the military who made it into an arms race, and kept upping the ante with more powerful nukes. Point being humans make theses decisions. It's not an inevitability.
pet_the_bird 27d ago
I am personally concerned by what defensive can mean. Alignment of these models is inherently a non-neutral proces, and currently what values are reinforced is decided by a few OpenAI engineers. I feel that any 'defensive model' will further ingrain current values and actively resist the natural progression of our society. This is especially the case for any use of these models for policing or military. Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?
- IncreasePosts 27d ago
  Defense can just be patching unintended vulnerabilities in code.
paxys 27d ago
I don't understand what you are trying to say. Do you believe AI is not an arms race?
w10-1 27d ago
> the best argument for AI is that it's an arms race It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?). The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages. In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).
asdff 27d ago
People expect a sort of arms race, at least the AI providers. But you don't need a more capable ai to stop ai from mucking with your systems today. Airgaps are the solution. Protected networks with independent infrastructure from the public internet. Most of the truly important stuff operates this way already. Eventually you might sever yourself off as well, you might say you will stop going to HN or other sites one day as signal to noise is too poor with AI fodder slop, you might use local models you control, and you might keep most of your hardware from connecting to any untrusted hosts. Essentially, you go dark. It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.
- esafak 27d ago
  Airgaps are a temporary solution. AI can manipulate humans, and robots will soon traverse the gap physically.
sleight42 27d ago
It's capitalism taken to its extreme, yeah? You have to be more cost efficient or more capable or seem more or you lose to the competition who outperforms you there. Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,
deleted 27d ago
[deleted]
_bobm 27d ago
I think it is simpler, but bannin open weight is a collateral worth scoring nonetheless. AI fails to deliver on the multi trilion usd promises: Gov/venture says: oh, wow, what will we do with all these datacenters, we need money back! OpenAI/ Anthropic: let's monitor the citizenry. They may be plotting nefarious schemes using AI models. Gov: great idea. Whew. Investors: whew! Tax payers: paying to be in prison. So not so much as defense against foreign actors, but failure of the self-tooted AGI goal + gov being gov.

by pu_pe 27d ago

dgellow 27d ago

Yep. Such a disgusting industry. They created the arm race, push for the arm race, put themselves in position to benefit from the arm race

estearum 27d ago
The entire problem with arms races is that any individual entity cannot avoid participating. sama and his cadre are uniquely evil captains in this race, but they're completely replaceable and the dynamic would remain the same.
- 0xDEAFBEAD 27d ago
  They could work to coordinate an end to the arms race.
  - estearum 27d ago
    Uhhh... the mechanism by which they'd do that is via government regulation. Most of the frontier labs have been openly requesting regulation: "the structure of this competition is not suitable for the development of this particular technology" The reactions vary from: 1. China will do it anyway (need some supranational governance scheme) 2. This is just an attempt at regulatory capture 3. This is just marketing 4. Regulation is bad mmmmkay
wartywhoa23 27d ago
Just like the rest of military stuff.
- goatlover 27d ago
  "A strange game. The only winning move is not to play."
- dgellow 27d ago
  But done by corporations, and selling that service to the general public, including their competitors and adversary countries
mrshadowgoose 27d ago
The arms race is an intrinsic game theoretical property of a multi-adversarial-actor scenario involving exponential growth of a universally potent technology. It's almost certainly winner-take-all, on a global scale, which behooves everyone to participate. And no, I don't think it will end well.
- reasonableklout 27d ago
  The actors here are states or corporations embedded in societies that risk growing popular backlash against the technology. The other factor is that if it is truly an "alien mind", racing incurs risks to all players. In game-theoretic terms it may be more like a stag hunt than a prisoner's dilemma. In which case cooperation is an equilibrium.
- nradov 27d ago
  No previous technological arms race has ever concluded with a single winner.

skybrian 27d ago

"Defensive systems" can be interpreted broadly to include cybersecurity. But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong. Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D? Individuals can quit, but for society, getting out of an arms race is harder than just quitting. You don't get to be Switzerland without having a strong defensive position and the right foreign relations. But there's at least talk about "pacing" and that's a start.

Fraterkes 27d ago
The point of the comment you're responding to is that it is at this point hardly an arms race: The frontier of ai development is happening completely inside 2 American companies, the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models. So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party. An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
- throwthrowuknow 27d ago
  Development and improvement of nuclear weapons was an entirely American project until the technology was exfiltrated and then it became an instant arms race. That cat is already out of the bag with LLMs. Distillation is just the fastest way to keep pace but that in no way prevents other countries and actors from doing it the hard way. An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s a race to keep pace or retake the lead position which can oscillate between the parties involved indefinitely. The other option is to agree to make no further progress or to disarm.
  - goatlover 27d ago
    There were prominent scientists like Oppenheimer who did not think it needed to become an arms race, and campaigned against that. But there were others like Teller and the military who made it into an arms race, and kept upping the ante with more powerful nukes. Point being humans make theses decisions. It's not an inevitability.

pet_the_bird 27d ago

I am personally concerned by what defensive can mean. Alignment of these models is inherently a non-neutral proces, and currently what values are reinforced is decided by a few OpenAI engineers. I feel that any 'defensive model' will further ingrain current values and actively resist the natural progression of our society. This is especially the case for any use of these models for policing or military. Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?

IncreasePosts 27d ago
Defense can just be patching unintended vulnerabilities in code.

paxys 27d ago

I don't understand what you are trying to say. Do you believe AI is not an arms race?

w10-1 27d ago

> the best argument for AI is that it's an arms race It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?). The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages. In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).

asdff 27d ago

People expect a sort of arms race, at least the AI providers. But you don't need a more capable ai to stop ai from mucking with your systems today. Airgaps are the solution. Protected networks with independent infrastructure from the public internet. Most of the truly important stuff operates this way already. Eventually you might sever yourself off as well, you might say you will stop going to HN or other sites one day as signal to noise is too poor with AI fodder slop, you might use local models you control, and you might keep most of your hardware from connecting to any untrusted hosts. Essentially, you go dark. It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.

esafak 27d ago
Airgaps are a temporary solution. AI can manipulate humans, and robots will soon traverse the gap physically.

sleight42 27d ago

It's capitalism taken to its extreme, yeah? You have to be more cost efficient or more capable or seem more or you lose to the competition who outperforms you there. Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,

deleted 27d ago

[deleted]

_bobm 27d ago

I think it is simpler, but bannin open weight is a collateral worth scoring nonetheless. AI fails to deliver on the multi trilion usd promises: Gov/venture says: oh, wow, what will we do with all these datacenters, we need money back! OpenAI/ Anthropic: let's monitor the citizenry. They may be plotting nefarious schemes using AI models. Gov: great idea. Whew. Investors: whew! Tax payers: paying to be in prison. So not so much as defense against foreign actors, but failure of the self-tooted AGI goal + gov being gov.