12 ms·

This. Will. Never. Work. This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will

This. Will. Never. Work. This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow. The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect. The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection. Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology. Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals. Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.

red_green_yell 3mo ago
The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.
- vessenes 3mo ago
  I don’t think it’s clear we’ll get commoditized AI out of this research loop. The only real datapoints we have are that Chinese open weights models seem able to stay a certain amount of distance ‘behind’ released frontier models. It’s clear that the models are trained on exfiltrated data from the frontier models => we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum. I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. You might believe differently. But, if I’m right, we are nowhere near some massive collapse of these companies, quite the opposite. For the opposite take to be correct, you’d need to believe that the ‘value’ of tokens will not increase significantly and that the current demand for tokens is at or near a peak. I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I believe instead that religion and money combined create strange behaviors and beliefs - in this case “what’s good for us could be implemented” — combined with the already endemic “what’s good for us is good for humanity.” Together we get really bad outcomes.
  - red_green_yell 3mo ago
    > we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum. Whether open-weight labs can get to the frontier on their own is irrelevant IMO. Even if all R&D were to stop today it would take a decade at least for the productivity gains from current open-weight models to trickle through the economy. For 99% of use cases of AI there simply isn't a need for further development of the frontier, where we are very clearly at diminishing returns. How many white collar workers need to write an OS or develop sophisticated software? Think about a normal office worker - they an easily double their productivity by creating a well-thought-out partnership between the human and today's open weight AI. Humans and today's AI are very complementary. AI brings vast knowledge, pattern recognition, and infinite patience for grunt work. Human brings context and reasoning and verification. Frontier labs are trying to replace the human altogether by trying to make AI do things that AI is weak at and humans are strong at. This is silly but they have no choice because of the massive profits they've promised their investors. It's not enough to double every knowledge worker's productivity. They want to replace the knowledge worker altogether. This is creating a massive spike in training & inference cost that is almost certainly more expensive than the human labor its intended to replace. Fable and 5.6 sol are supposedly around 8-10T parameters. Is there enough compute on earth to make a 100T or 1Q model? We would need models that size to replace the human altogether. > I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. I agree with this, but the massive demand for inference will force companies to be price conscious and drive them away from ultra high margin frontier models to low margin open weight models. This is why it will become a low-margin, commodity market. Immense innovation right now is going into tiny, specialized models that run on device. See PrismML getting a mid sized QWEN model to run on an iPhone. When on-device inference becomes a thing we'll see another explosion in inference that will further leach demand away from frontier labs. > I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I agree if you could read their minds they might think they're doing it for the "right reasons." But massive $$$ can create equally massive self justification/deception.
levocardia 3mo ago
Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.
- OutOfHere 3mo ago
  Bioweapons don't make money. AI does. The two are therefore not comparable.
- vessenes 3mo ago
  Bioweapons are good for .. war. Agentic AI is good for changing your economy, strategic advantage, labor explosion and making more effective Agentic AI. They are not the same category of thing.
lifeformed 3mo ago
Some reason this kind of argument always by default assumes the US will be the good guys.
- vessenes 3mo ago
  I didn’t say that. I said the US is ahead, and should consider its strategic adversaries and their behavior.
frotaur 3mo ago
"and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology." These research groups are almost unanimously advocating for a pause/slowdown as our best chance.
- vessenes 3mo ago
  … while they address absolutely none of the above points, except for tacitly or overtly agreeing a panopticon surveillance state is needed / desirable to get their desired ‘slow down’, a situation where the cure is infinitely worse than the disease, not least because it will not be effective at the original goal of eradicating the disease, and will instead merely persist in all the global harms such a system has as a necessary side effect. When you are ahead, ‘slow down’ = better strategic positioning for yourself. When you are behind, ‘slow down’ = time to catch up.

red_green_yell 3mo ago

The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.

vessenes 3mo ago
I don’t think it’s clear we’ll get commoditized AI out of this research loop. The only real datapoints we have are that Chinese open weights models seem able to stay a certain amount of distance ‘behind’ released frontier models. It’s clear that the models are trained on exfiltrated data from the frontier models => we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum. I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. You might believe differently. But, if I’m right, we are nowhere near some massive collapse of these companies, quite the opposite. For the opposite take to be correct, you’d need to believe that the ‘value’ of tokens will not increase significantly and that the current demand for tokens is at or near a peak. I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I believe instead that religion and money combined create strange behaviors and beliefs - in this case “what’s good for us could be implemented” — combined with the already endemic “what’s good for us is good for humanity.” Together we get really bad outcomes.
- red_green_yell 3mo ago
  > we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum. Whether open-weight labs can get to the frontier on their own is irrelevant IMO. Even if all R&D were to stop today it would take a decade at least for the productivity gains from current open-weight models to trickle through the economy. For 99% of use cases of AI there simply isn't a need for further development of the frontier, where we are very clearly at diminishing returns. How many white collar workers need to write an OS or develop sophisticated software? Think about a normal office worker - they an easily double their productivity by creating a well-thought-out partnership between the human and today's open weight AI. Humans and today's AI are very complementary. AI brings vast knowledge, pattern recognition, and infinite patience for grunt work. Human brings context and reasoning and verification. Frontier labs are trying to replace the human altogether by trying to make AI do things that AI is weak at and humans are strong at. This is silly but they have no choice because of the massive profits they've promised their investors. It's not enough to double every knowledge worker's productivity. They want to replace the knowledge worker altogether. This is creating a massive spike in training & inference cost that is almost certainly more expensive than the human labor its intended to replace. Fable and 5.6 sol are supposedly around 8-10T parameters. Is there enough compute on earth to make a 100T or 1Q model? We would need models that size to replace the human altogether. > I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. I agree with this, but the massive demand for inference will force companies to be price conscious and drive them away from ultra high margin frontier models to low margin open weight models. This is why it will become a low-margin, commodity market. Immense innovation right now is going into tiny, specialized models that run on device. See PrismML getting a mid sized QWEN model to run on an iPhone. When on-device inference becomes a thing we'll see another explosion in inference that will further leach demand away from frontier labs. > I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I agree if you could read their minds they might think they're doing it for the "right reasons." But massive $$$ can create equally massive self justification/deception.

levocardia 3mo ago

Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.

OutOfHere 3mo ago
Bioweapons don't make money. AI does. The two are therefore not comparable.
vessenes 3mo ago
Bioweapons are good for .. war. Agentic AI is good for changing your economy, strategic advantage, labor explosion and making more effective Agentic AI. They are not the same category of thing.

lifeformed 3mo ago

Some reason this kind of argument always by default assumes the US will be the good guys.

vessenes 3mo ago
I didn’t say that. I said the US is ahead, and should consider its strategic adversaries and their behavior.

frotaur 3mo ago

"and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology." These research groups are almost unanimously advocating for a pause/slowdown as our best chance.

vessenes 3mo ago
… while they address absolutely none of the above points, except for tacitly or overtly agreeing a panopticon surveillance state is needed / desirable to get their desired ‘slow down’, a situation where the cure is infinitely worse than the disease, not least because it will not be effective at the original goal of eradicating the disease, and will instead merely persist in all the global harms such a system has as a necessary side effect. When you are ahead, ‘slow down’ = better strategic positioning for yourself. When you are behind, ‘slow down’ = time to catch up.