7 ms·

> Anthropic occupies a peculiar position in the AI landscape: a company that genuinely believes it might be building one of the most transformative and potentia

> Anthropic occupies a peculiar position in the AI landscape: a company that genuinely believes it might be building one of the most transformative and potentially dangerous technologies in human history, yet presses forward anyway. This isn't cognitive dissonance but rather a calculated bet—if powerful AI is coming regardless, Anthropic believes it's better to have safety-focused labs at the frontier than to cede that ground to developers less focused on safety (see our core views). Ah, yes, safety, because what is more safe than to help DoD/Palantir kill people[1]? No, the real risk here is that this technology is going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to a lobotomized and heavily censored version of it, if at all. [1] - https://www.anthropic.com/news/anthropic-and-the-department-of-defense-to-advance-responsible-ai-in-defense-operations https://www.anthropic.com/news/anthropic-and-the-department-...

reissbaker 11mo ago
This is the major reason China has been investing in open-source LLMs: because the U.S. publicly announced its plans to restrict AI access into tiers, and certain countries — of course including China — were at the lowest tier of access. [1] If the U.S. doesn't control the weights, though, it can't restrict China from accessing the models... 1: https://thefuturemedia.eu/new-u-s-rules-aim-to-govern-ais-global-expansion/ https://thefuturemedia.eu/new-u-s-rules-aim-to-govern-ais-gl...
- dist-epoch 11mo ago
  It isn't "China" which open-source LLMs, but individual Chinese labs. China didn't yet made a sovereign move on AI, besides investing in research/hardware.
  - iambateman 11mo ago
    This is a distinction without a difference.
  - baq 11mo ago
    Axiom of China: nothing of importance happens in China without CCP involvement.
    - yorwba 11mo ago
      The CCP controlling the government doesn't mean they micromanage everything. Some Chinese AI companies release the weights of even their best models (DeepSeek, Moonshot AI), others release weights for small models, but not the largest ones (Alibaba, Baidu), some keep almost everything closed (Bytedance and iFlytek, I think). There is no CCP master plan for open models, any more than there is a Western master plan for ignoring Chinese models only available as an API.
      - baq 11mo ago
        Never suggested anything of the sort, involvement doesn’t mean direct control, it might be a passive ‘let us know if there’s progress’ issued privately, it might also be a passive ‘we want to be #1 in AI in 2030’ announced publicly, neither requires any micromanagement whatsoever: CCP’s expectation is companies figuring out how to align to party directives themselves… or face consequences.
        shroobani 11mo ago
        Unlike the US, where there are no consequences for not aligning with the ruling party's directives.
        mlinhares 11mo ago
        The US couldn't make China follow it, so it is now following China's lead LOL.
        baq 11mo ago
        https://en.wikipedia.org/wiki/Whataboutism https://en.wikipedia.org/wiki/Whataboutism (not that I disagree)
        11mo ago
  - throwup238 11mo ago
    As far as I can tell AI is already playing a big part in the Chinese Fifteenth five year plan (2026-2030) which is their central top-down planning mechanism. That’s about as big a move as they can make.
    - esafak 11mo ago
      I think the plan is due next March? I believe it includes at AI Plus initiative: https://triviumchina.com/research/the-ai-plus-initiative-chinas-blueprint-for-ai-diffusion/ https://triviumchina.com/research/the-ai-plus-initiative-chi...
  - reissbaker 11mo ago
    I think "investing in research and hardware" is fairly relevant to my claim of "China has been investing in open-source LLMs." China also has partial ownership of several major labs via "golden shares" [1] like Alibaba (Qwen) and Zai (GLM) [2], albeit not DeepSeek as far as I know. 1: https://www.theguardian.com/world/2023/jan/13/china-to-take-golden-shares-in-tech-firms-alibaba-and-tencent https://www.theguardian.com/world/2023/jan/13/china-to-take-... 2: https://www.globalneighbours.org/chinas-zhipu-ai-secures-140-million-investment-from-shanghai-state-funds-amid-ipo-push/ https://www.globalneighbours.org/chinas-zhipu-ai-secures-140...
- slanterns 11mo ago
  and Anthropic bans access from China along with throwing some politic propagenda bs
  - UltraSane 11mo ago
    Ask deepseek about how many people the CCP killed during the 1989 Tiananmen Square massacre.
    - r14c 11mo ago
      [flagged]
      - justinclift 11mo ago
        Pretty sure they're asking for the narrative that's widely known about everywhere _except_ by the er... non-leadership people of China.
      - UltraSane 11mo ago
        I'm genuinely curious how one develops a world view like this.
        dmayle 11mo ago
        I recently learned about the (ancient?) greek concept of amathia. It's a willful ignorance, often cultivated as a preference for identity and ego over learning. It's not about a lack of intelligence, but rather a willful pattern of subverting learning in favor of cult and ideology.
        r14c 11mo ago
        I read a lot. I'm not saying nobody died at Tiananmen, but framing it as a massacre is specifically a US/NATO narrative.
        UltraSane 11mo ago
- IncreasePosts 11mo ago
  Why wouldn't China just keep their own weights secret as well? If this really is a geopolitical play(I'm not sure if it is or isn't), it could be along the lines of: 1) most AI development in the US is happening at private companies with balance sheets, share holders, and profit motives. 2) China may be lagging in compute to beat everyone to the punch in a naked race Therefore, releasing open weights may create a situation where AI companies can't as effectively sell their services, meaning they may curtail r&d at a certain point. China can then pour nearly infinite money into it and eventually get up to speed on compute and win the race
  - bamboozled 11mo ago
    I think it's just because China makes it's money from other sources, not from AI, and from what I've read, the advantage of China killing the US's AI advantage is killing it's stock market / disrupting. Seems like it may have a chance of working if you look at the companies highest valued on the S&P 500: NVIDIA, Microsoft, Apple, Amazon, Meta Platforms, Broadcom, Alphabet (Class C),
    - adventured 11mo ago
      The share of revenue that Microsoft, Google, Meta, Apple, Alphabet and Amazon are currently deriving from the AI market as a share of their total revenue, is less than 10%.
      - tigershark 11mo ago
        What about NVDA (~$4.5T) and AVGO (~$1.8T)?
      - LunaSea 11mo ago
        What about all the investments made by these companies and other VCs, hedge funds, angel investors, pension funds, etc?
  - zamalek 11mo ago
    They are taking the gun out of USA's hand and unloading it, figuratively speaking. With this strategy they don't have the compete at full competency with the US, because everyone else will with cheaper models. If a cheaper model can do it, then why fork out for Opus?
regularization 11mo ago
> to ensure AI development strengthens democratic values globally I wonder if that's helping the US Navy shoot up fishing boats in the Caribbean or facilitating the bombing of hospitals, schools and refugee camps in Gaza.
- odiroot 11mo ago
  > Please don't use Hacker News for political or ideological battle. It tramples curiosity.
- ch2026 11mo ago
  It helps provide the therapy bot used by struggling sailors who are questioning orders and reducing "hey this isn’t what i signed up for" mental breakdowns.
  - conception 11mo ago
    "Wait, this seems like a war crime." "You're absolutely right!"
Aarostotle 11mo ago
A narrow and cynical take, my friend. With all technologies, "safety" doesn't equate to plushie harmlessness. There is, for example, a valid notion of "gun safety." Long-term safety for free people entails military use of new technologies. Imagine if people advocating airplane safety groused about the use of bomber and fighter planes being built and mobilized in the Second World War. Now, I share your concern about governments who unjustly wield force (either in war or covert operations). That is an issue to be solved by articulating a good political philosophy and implementing it via policy, though. Sadly, too many of the people who oppose the American government's use of such technology have deeply authoritarian views themselves — they would just prefer to see a different set of values forced upon people. Last: Is there any evidence that we're getting some crappy lobotomized models while the companies keep the best for themselves? It seems fairly obvious that they're tripping over each other in a race to give the market the highest intelligence at the lowest price. To anyone reading this who's involved in that, thank you!
- ceejayoz 11mo ago
  > Long-term safety for free people entails military use of new technologies. Long-term safety also entails restraining the military-industrial complex from the excesses it's always prone to. Remember, Teller wanted to make a 10 gigaton nuke. https://en.wikipedia.org/wiki/Sundial_(weapon) https://en.wikipedia.org/wiki/Sundial_(weapon)
  - Aarostotle 11mo ago
    I agree, your point is compatible with my view. My sense is that this essentially an optimization question within how a government ought to structures its contracts with builders of weapons. The current system is definitely suboptimal (put mildly) and corrupt. The integrity of a free society's government is the central issue here, not the creation of tools which could be militarily useful to a free society.
- gausswho 11mo ago
  Exhibit A of 'grousing': Guernica. There was indeed a moment where civilization asked this question before.
skybrian 11mo ago
I don't think that's a real risk. There are strong competitors from multiple countries releasing new models all the time, and some of them are open weights. That's basically the opposite of a monopoly.
- thoughtpeddler 11mo ago
  Unless back-channel conversations keep 'competitors' colluding to ensure that 'public SOTA' is ~uniformly distributed...
ardata 11mo ago
risk? certainty. it's pretty much guaranteed. the most capable models are already behind closed doors for gov/military use and that's not ever changing. the public versions are always going to be several steps behind whatever they're actually running internally. the question is what the difference will be between the corporation and pleb versions is
- habinero 11mo ago
  That's movies. Ask anyone in the military what "military grade" means.
flatline 11mo ago
Ironically, this is one the part of the document that jumped out at me as having been written by AI. The em-dash and "this isn't...but" pattern are louder than the text at this point. It seriously calls into question who is authoring what, and what their actual motives are.
- gnatman 11mo ago
  Every time I see the em-dash call out on here I get defensive because I’ve been writing like that forever! Where do people think that came from anyway? It’s obviously massively represented in the training data!
  - observationist 11mo ago
    Where's the emdash key on your keyboard? There isn't one? Oh, maybe that's why people who didn't already know or care about emdashes are very alert to their presence. If you have to do something very exotic with keypresses or copypaste from a tool or build your own macro to get something like an emdash, or , it's going to stand out, even if it's an integral part of standard operating systems.
    - nonfamous 11mo ago
      Typing hyphen-hyphen-space is hardly exotic — I've been doing that since well beyond the advent of generative AI.
      - belter 11mo ago
        Nice try bot....
      - observationist 11mo ago
        Right, just saying things like that -- aren't immediately apparent unless they're pointed out to you. The extended palette of alt+123 keycodes, unicode characters, stuff like that requires "exotic" macros or keypresses to type out. Despite decades of extensive experience with writing, writing software, programming, etc, I never crossed paths with em-dashes. They were a niche thing prior to AI making them a thing. I basically thought they were a font or style choice prior to ChatGPT. Most people wouldn't have a clue unless they went through classes that specifically trained on the use of emdashes. I like them as an AI shibboleth, though -- the antennae go up, and I pay more attention to what I'm reading when I see it, so it raises the bar for the humans that ostensibly ought to be better at writing than the rest of us. Edit: Interesting. I tried using -- and it doesn't work for me. I'd have to go change settings somewhere, or switch the browser I'm using to elicit an em-dash. I don't think I've ever actually written one, at least intentionally, and it wasn't until today that I was even aware of hyphen-hyphen. Edit again: I had to go into system settings and assign a compose key — after that, I can now do em-dashes. Having degrees° will be nice, too, I guess.
UltraSane 11mo ago
I predict that billionaires will pay to build their own completely unrestricted LLMs that will happily help them get away with crimes and steal as much money as possible.
- astrange 11mo ago
  Crimes generally don't pay and are not worth anyone's time. The reason poor people imagine billionaires commit lots of crimes is that the poor people don't know how to become rich; if they did, they would've done it already. Since they do know how to commit crimes, they imagine that's how you do it but bigger. The reason criminals commit crimes is that criminals are dumb and have poor impulse control. (This is the same concept as "Trump is the poor person's idea of a rich person." He actually did get there through crime, which is why poor criminals like him, but he's inhumanly lucky.)
  - eadler 11mo ago
    > The reason criminals commit crimes is that criminals are dumb and have poor impulse control. What makes you believe this? Any data to support this claim? It's inconsistent with the majority of research I've read on the topic but I'm no expert.
    - astrange 11mo ago
      You're reading research that says they're geniuses? As far as I know lack of self-control is the main factor. https://pmc.ncbi.nlm.nih.gov/articles/PMC8095718/ https://pmc.ncbi.nlm.nih.gov/articles/PMC8095718/ (see "Self-Control as Criminality" although it has a lot of caveats) The other two are "being a young man" and lead poisoning, which are both versions of being dumb. https://www.sciencedirect.com/science/article/pii/S0166046222000667 https://www.sciencedirect.com/science/article/pii/S016604622...
      - eadler 11mo ago
        > You're reading research that says they're geniuses? I didn't say this > ... Re the rest. Thanks. I had implicitly assumed we were talking about financial or white collar crimes rather than all crimes. In other words the types of crimes people generally assume that richer people commit (insider training, tax evasion, wage theft, etc.) I think you are correct in the most general sense of "all crime"
patcon 11mo ago
what if more power (from state) goes to the group that does engage in those activities, and therefore Anthropic gets marginalized as shadow sectors of state power pick a different winner? These things are not clear. I do not envy those who must neurotically think through the first-order, second-order, third-order judgements of all of justice, "evil" and "good" that one must do. It's a statescraft level of hierarchy of concerns that would leave me immensely challenged
ben_w 11mo ago
> No, the real risk here is that this technology is going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to a lobotomized and heavily censored version of it, if at all. Given the number of leaks, deliberate publications of weights, and worldwide competition, why do you believe this? (Even if by "lobotomised" you mean "refuses to assist with CNB weapon development"). Also, you can have more than one failure mode both be true. A protest against direct local air polution from a coal plant is still valid even though the greenhouse effect exists, and vice versa.
- kouteiheika 11mo ago
  > Given the number of leaks, deliberate publications of weights, and worldwide competition, why do you believe this? So where can I find the leaked weights of GPT-3/GPT-4/GPT-5? Or Claude? Or Gemini? The only weights we are getting are those which the people on the top decided we can get, and precisely because they're not SOTA. If any of those companies stumbles upon true AGI (as unlikely as it is), you can bet it will be tightly controlled and normal people will either have an extremely limited access to it, or none at all. > Even if by "lobotomised" you mean "refuses to assist with CNB weapon development" Right, because people who design/manufacture weapons of mass destruction will surely use ChatGPT to do it. The same ChatGPT who routinely hallucinates widely incorrect details even for the most trifling queries. If anything, that'd only sabotage their efforts if they're stupid enough to use an LLM for that. Nevertheless, it's always fun when you ask an LLM to translate something from another language, and the line you're trying to translate coincidentally contains some "unsafe" language, and your query gets deleted and you get a nice, red warning that "your request violates our terms and conditions". Ah, yes, I'm feeling "safe" already.
  - astrange 11mo ago
    Kimi-K2-Thinking and DeepSeek-V3.2 are open and pretty near SOTA.
  - ben_w 11mo ago
    Imagine saying Operating systems are going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to what computers can really do! Getting the reply We have open-source OSes And then replying So where can I find the leaked source of Windows? Or MacOS? We have a bajillion Linuxes. There's a lot of open-weights GenAI models. Including from OpenAI, whose open models beat everything in their own GPT-3 and 4 families. But also not "those which the people on the top decided we can get", which is why Meta sued over the initial leak of the original LLaMa's weights. > true AGI Is ill-defined. Like, I don't think I've seen any two people agree on what it means… unless they're the handful that share the definition I'd been using before I realised how rare it was ("a general-purpose AI model", which they all meet). If your requirement includes anything like "learns quickly from few examples", which is a valid use of the word "intelligence" and one where all ML training methods known fail because they are literally too stupid to live (no single organism would survive long enough to make that many mistakes), and AI generally only make up for this by doing what passes for thinking faster than anything alive to the degree to which we walk faster than continental drift, then whoever first tasks such a model with taking over the world, succeeds. To emphasise two points: 1. Not "trains", "tasks". 2. It succeeds because anything which can learn from as few examples as us, while operating so quickly that it can ingest the entire internet in a few months, is going to be better at everything than anyone. At which point, you'd better hope that either whoever trained it, trained it in a way that respects concepts like "liberty" and "democracy" and "freedom" and "humans are not to be disassembled for parts", or that whoever tasked it with taking over the world both cares about those values and rules-lawyers the AI like a fictional character dealing with a literal-minded genie. > Right, because people who design/manufacture weapons of mass destruction will surely use ChatGPT to do it. The same ChatGPT who routinely hallucinates widely incorrect details even for the most trifling queries. If anything, that'd only sabotage their efforts if they're stupid enough to use an LLM for that. First, yes of course they will, even existing professionals, even when they shouldn't. Have you not seen the huge number of stories about everyone using it for everything, including generals? Second, the risk is new people making them. My experience of using LLMs is as a software engineer, not as a biologist, chemist, or physicist: LLMs can do fresh-graduate software engineering tasks at fresh-graduate competence levels. Can LLMs display fresh-graduate level competence in NBC? If LLMs can do that, they necessarily expand the number of groups who can run NBC programs to include any random island nation with not enough grads to run a NBC program, or mid-sized organised crime group, or Hamas. They don't even need to do all of it, just be good enough to help. "Automate cognitive tasks" is basically the entire point of these things, after all. And if the AI isn't competent to help with those things, if they're e.g. at the level of competence of "sure mix those two bleaches without checking what they are" (explosion hazard) or "put that raw garlic in that olive oil and just leave it at room temperature for a few weeks it will taste good" (biohazard, and one model did this), then surely it's a matter of general public safety to make them not talk about those things because of all the lazy students who are already demonstrating they're just as lazy as whoever wrote the US tariff policy that put a different tariff on an island occupied by only penguins vs. the country which owned it and which a lot of people suspect came out of an LLM. > Nevertheless, it's always fun when you ask an LLM to translate something from another language, and the line you're trying to translate coincidentally contains some "unsafe" language, and your query gets deleted and you get a nice, red warning that "your request violates our terms and conditions". Ah, yes, I'm feeling "safe" already. Use Google Translate. It's the same architecture, trained to give a translation instead of a reply. Or, equivalently, the chat models (and code generators like Claude) are the same architecture as Google Translate, trained to "translate" your prompt into an answer.
jimbo808 11mo ago
I don't believe that they believe it, I believe that they're all in on doing all the things you'd do if your goal was to demonstrate to investors that you truly believe it. The safety-focused labs are the marketing department. An AI that can actually think and reason, and not just pretend to by regurgitating/paraphrasing text that humans wrote, is not something we're on any path to building right now. They keep telling us these things are going to discover novel drugs and do all sorts of important science, but internally, they are well aware that these LLM architectures fundamentally can't do that. A transformer-based LLM can't do any of the things you'd need to be able to do as an intelligent system. It has no truth model, and lacks any mechanism of understanding its own output. It can't learn and apply new information, especially not if it can't fit within one context window. It has no way to evaluate if a particular sequence of tokens is likely to be accurate, because it only selects them based on the probability of appearing in a similar sequence, based on the training data. It can't internally distinguish "false but plausible" from "true but rare." Many things that would be obviously wrong to a human, would appear to be "obviously" correct when viewed from the perspective of an LLM's math. These flaws are massive, and IMO, insurmountable. It doesn't matter if it can do 50% of a person's work effectively, because you can't reliably predict which 50% it will do. Given this unpredictability, its output has to be very carefuly reviewed by an expert in order to be used for any work that matters. Even worse, the mistakes it makes are meant to be difficult to spot, because it will always generate the text that looks the most right. Spotting the fuckup in something that was optimized not to look like a fuckup is much more difficult than reviewing work done by a well-intentioned human.
- HDThoreaun 11mo ago
  If you dont believe they believe it you havent paid any attention to the company. Maybe Dario is lying, although that would be an extremely long con, but the rank and file 100% believe it.
- vancroft 11mo ago
  Sounds like the old saying about the advertising industry: "I know half of my spending on advertising is wasted - I just don't know which half."
- 11mo ago
beefnugs 11mo ago
Its just with piss and fentanyl were the CEOs exact words, i think the AI would humanely use enough piss to wash away the fentanyl so that minimal deaths will occur. Morality Achieved!
antonvs 11mo ago
The trick here is to focus on imaginary safety from intentional AIs while ignoring the risks posed by real people using AI against other people.
bionhoward 11mo ago
Anthropic also has legal terms that say no one is allowed to use the service for anything work related, but nobody seems to care

reissbaker 11mo ago

This is the major reason China has been investing in open-source LLMs: because the U.S. publicly announced its plans to restrict AI access into tiers, and certain countries — of course including China — were at the lowest tier of access. [1] If the U.S. doesn't control the weights, though, it can't restrict China from accessing the models... 1: https://thefuturemedia.eu/new-u-s-rules-aim-to-govern-ais-global-expansion/ https://thefuturemedia.eu/new-u-s-rules-aim-to-govern-ais-gl...

dist-epoch 11mo ago
It isn't "China" which open-source LLMs, but individual Chinese labs. China didn't yet made a sovereign move on AI, besides investing in research/hardware.
- iambateman 11mo ago
  This is a distinction without a difference.
- baq 11mo ago
  Axiom of China: nothing of importance happens in China without CCP involvement.
  - yorwba 11mo ago
    The CCP controlling the government doesn't mean they micromanage everything. Some Chinese AI companies release the weights of even their best models (DeepSeek, Moonshot AI), others release weights for small models, but not the largest ones (Alibaba, Baidu), some keep almost everything closed (Bytedance and iFlytek, I think). There is no CCP master plan for open models, any more than there is a Western master plan for ignoring Chinese models only available as an API.
    - baq 11mo ago
      Never suggested anything of the sort, involvement doesn’t mean direct control, it might be a passive ‘let us know if there’s progress’ issued privately, it might also be a passive ‘we want to be #1 in AI in 2030’ announced publicly, neither requires any micromanagement whatsoever: CCP’s expectation is companies figuring out how to align to party directives themselves… or face consequences.
      - shroobani 11mo ago
        Unlike the US, where there are no consequences for not aligning with the ruling party's directives.
        mlinhares 11mo ago
        The US couldn't make China follow it, so it is now following China's lead LOL.
        baq 11mo ago
        https://en.wikipedia.org/wiki/Whataboutism https://en.wikipedia.org/wiki/Whataboutism (not that I disagree)
        11mo ago
- throwup238 11mo ago
  As far as I can tell AI is already playing a big part in the Chinese Fifteenth five year plan (2026-2030) which is their central top-down planning mechanism. That’s about as big a move as they can make.
  - esafak 11mo ago
    I think the plan is due next March? I believe it includes at AI Plus initiative: https://triviumchina.com/research/the-ai-plus-initiative-chinas-blueprint-for-ai-diffusion/ https://triviumchina.com/research/the-ai-plus-initiative-chi...
- reissbaker 11mo ago
  I think "investing in research and hardware" is fairly relevant to my claim of "China has been investing in open-source LLMs." China also has partial ownership of several major labs via "golden shares" [1] like Alibaba (Qwen) and Zai (GLM) [2], albeit not DeepSeek as far as I know. 1: https://www.theguardian.com/world/2023/jan/13/china-to-take-golden-shares-in-tech-firms-alibaba-and-tencent https://www.theguardian.com/world/2023/jan/13/china-to-take-... 2: https://www.globalneighbours.org/chinas-zhipu-ai-secures-140-million-investment-from-shanghai-state-funds-amid-ipo-push/ https://www.globalneighbours.org/chinas-zhipu-ai-secures-140...
slanterns 11mo ago
and Anthropic bans access from China along with throwing some politic propagenda bs
- UltraSane 11mo ago
  Ask deepseek about how many people the CCP killed during the 1989 Tiananmen Square massacre.
  - r14c 11mo ago
    [flagged]
    - justinclift 11mo ago
      Pretty sure they're asking for the narrative that's widely known about everywhere _except_ by the er... non-leadership people of China.
    - UltraSane 11mo ago
      I'm genuinely curious how one develops a world view like this.
      - dmayle 11mo ago
        I recently learned about the (ancient?) greek concept of amathia. It's a willful ignorance, often cultivated as a preference for identity and ego over learning. It's not about a lack of intelligence, but rather a willful pattern of subverting learning in favor of cult and ideology.
      - r14c 11mo ago
        I read a lot. I'm not saying nobody died at Tiananmen, but framing it as a massacre is specifically a US/NATO narrative.
        UltraSane 11mo ago
IncreasePosts 11mo ago
Why wouldn't China just keep their own weights secret as well? If this really is a geopolitical play(I'm not sure if it is or isn't), it could be along the lines of: 1) most AI development in the US is happening at private companies with balance sheets, share holders, and profit motives. 2) China may be lagging in compute to beat everyone to the punch in a naked race Therefore, releasing open weights may create a situation where AI companies can't as effectively sell their services, meaning they may curtail r&d at a certain point. China can then pour nearly infinite money into it and eventually get up to speed on compute and win the race
- bamboozled 11mo ago
  I think it's just because China makes it's money from other sources, not from AI, and from what I've read, the advantage of China killing the US's AI advantage is killing it's stock market / disrupting. Seems like it may have a chance of working if you look at the companies highest valued on the S&P 500: NVIDIA, Microsoft, Apple, Amazon, Meta Platforms, Broadcom, Alphabet (Class C),
  - adventured 11mo ago
    The share of revenue that Microsoft, Google, Meta, Apple, Alphabet and Amazon are currently deriving from the AI market as a share of their total revenue, is less than 10%.
    - tigershark 11mo ago
      What about NVDA (~$4.5T) and AVGO (~$1.8T)?
    - LunaSea 11mo ago
      What about all the investments made by these companies and other VCs, hedge funds, angel investors, pension funds, etc?
- zamalek 11mo ago
  They are taking the gun out of USA's hand and unloading it, figuratively speaking. With this strategy they don't have the compete at full competency with the US, because everyone else will with cheaper models. If a cheaper model can do it, then why fork out for Opus?

regularization 11mo ago

> to ensure AI development strengthens democratic values globally I wonder if that's helping the US Navy shoot up fishing boats in the Caribbean or facilitating the bombing of hospitals, schools and refugee camps in Gaza.

odiroot 11mo ago
> Please don't use Hacker News for political or ideological battle. It tramples curiosity.
ch2026 11mo ago
It helps provide the therapy bot used by struggling sailors who are questioning orders and reducing "hey this isn’t what i signed up for" mental breakdowns.
- conception 11mo ago
  "Wait, this seems like a war crime." "You're absolutely right!"

Aarostotle 11mo ago

A narrow and cynical take, my friend. With all technologies, "safety" doesn't equate to plushie harmlessness. There is, for example, a valid notion of "gun safety." Long-term safety for free people entails military use of new technologies. Imagine if people advocating airplane safety groused about the use of bomber and fighter planes being built and mobilized in the Second World War. Now, I share your concern about governments who unjustly wield force (either in war or covert operations). That is an issue to be solved by articulating a good political philosophy and implementing it via policy, though. Sadly, too many of the people who oppose the American government's use of such technology have deeply authoritarian views themselves — they would just prefer to see a different set of values forced upon people. Last: Is there any evidence that we're getting some crappy lobotomized models while the companies keep the best for themselves? It seems fairly obvious that they're tripping over each other in a race to give the market the highest intelligence at the lowest price. To anyone reading this who's involved in that, thank you!

ceejayoz 11mo ago
> Long-term safety for free people entails military use of new technologies. Long-term safety also entails restraining the military-industrial complex from the excesses it's always prone to. Remember, Teller wanted to make a 10 gigaton nuke. https://en.wikipedia.org/wiki/Sundial_(weapon) https://en.wikipedia.org/wiki/Sundial_(weapon)
- Aarostotle 11mo ago
  I agree, your point is compatible with my view. My sense is that this essentially an optimization question within how a government ought to structures its contracts with builders of weapons. The current system is definitely suboptimal (put mildly) and corrupt. The integrity of a free society's government is the central issue here, not the creation of tools which could be militarily useful to a free society.
gausswho 11mo ago
Exhibit A of 'grousing': Guernica. There was indeed a moment where civilization asked this question before.

skybrian 11mo ago

I don't think that's a real risk. There are strong competitors from multiple countries releasing new models all the time, and some of them are open weights. That's basically the opposite of a monopoly.

thoughtpeddler 11mo ago
Unless back-channel conversations keep 'competitors' colluding to ensure that 'public SOTA' is ~uniformly distributed...

ardata 11mo ago

risk? certainty. it's pretty much guaranteed. the most capable models are already behind closed doors for gov/military use and that's not ever changing. the public versions are always going to be several steps behind whatever they're actually running internally. the question is what the difference will be between the corporation and pleb versions is

habinero 11mo ago
That's movies. Ask anyone in the military what "military grade" means.

flatline 11mo ago

Ironically, this is one the part of the document that jumped out at me as having been written by AI. The em-dash and "this isn't...but" pattern are louder than the text at this point. It seriously calls into question who is authoring what, and what their actual motives are.

gnatman 11mo ago
Every time I see the em-dash call out on here I get defensive because I’ve been writing like that forever! Where do people think that came from anyway? It’s obviously massively represented in the training data!
- observationist 11mo ago
  Where's the emdash key on your keyboard? There isn't one? Oh, maybe that's why people who didn't already know or care about emdashes are very alert to their presence. If you have to do something very exotic with keypresses or copypaste from a tool or build your own macro to get something like an emdash, or , it's going to stand out, even if it's an integral part of standard operating systems.
  - nonfamous 11mo ago
    Typing hyphen-hyphen-space is hardly exotic — I've been doing that since well beyond the advent of generative AI.
    - belter 11mo ago
      Nice try bot....
    - observationist 11mo ago
      Right, just saying things like that -- aren't immediately apparent unless they're pointed out to you. The extended palette of alt+123 keycodes, unicode characters, stuff like that requires "exotic" macros or keypresses to type out. Despite decades of extensive experience with writing, writing software, programming, etc, I never crossed paths with em-dashes. They were a niche thing prior to AI making them a thing. I basically thought they were a font or style choice prior to ChatGPT. Most people wouldn't have a clue unless they went through classes that specifically trained on the use of emdashes. I like them as an AI shibboleth, though -- the antennae go up, and I pay more attention to what I'm reading when I see it, so it raises the bar for the humans that ostensibly ought to be better at writing than the rest of us. Edit: Interesting. I tried using -- and it doesn't work for me. I'd have to go change settings somewhere, or switch the browser I'm using to elicit an em-dash. I don't think I've ever actually written one, at least intentionally, and it wasn't until today that I was even aware of hyphen-hyphen. Edit again: I had to go into system settings and assign a compose key — after that, I can now do em-dashes. Having degrees° will be nice, too, I guess.

UltraSane 11mo ago

I predict that billionaires will pay to build their own completely unrestricted LLMs that will happily help them get away with crimes and steal as much money as possible.

astrange 11mo ago
Crimes generally don't pay and are not worth anyone's time. The reason poor people imagine billionaires commit lots of crimes is that the poor people don't know how to become rich; if they did, they would've done it already. Since they do know how to commit crimes, they imagine that's how you do it but bigger. The reason criminals commit crimes is that criminals are dumb and have poor impulse control. (This is the same concept as "Trump is the poor person's idea of a rich person." He actually did get there through crime, which is why poor criminals like him, but he's inhumanly lucky.)
- eadler 11mo ago
  > The reason criminals commit crimes is that criminals are dumb and have poor impulse control. What makes you believe this? Any data to support this claim? It's inconsistent with the majority of research I've read on the topic but I'm no expert.
  - astrange 11mo ago
    You're reading research that says they're geniuses? As far as I know lack of self-control is the main factor. https://pmc.ncbi.nlm.nih.gov/articles/PMC8095718/ https://pmc.ncbi.nlm.nih.gov/articles/PMC8095718/ (see "Self-Control as Criminality" although it has a lot of caveats) The other two are "being a young man" and lead poisoning, which are both versions of being dumb. https://www.sciencedirect.com/science/article/pii/S0166046222000667 https://www.sciencedirect.com/science/article/pii/S016604622...
    - eadler 11mo ago
      > You're reading research that says they're geniuses? I didn't say this > ... Re the rest. Thanks. I had implicitly assumed we were talking about financial or white collar crimes rather than all crimes. In other words the types of crimes people generally assume that richer people commit (insider training, tax evasion, wage theft, etc.) I think you are correct in the most general sense of "all crime"

patcon 11mo ago

what if more power (from state) goes to the group that does engage in those activities, and therefore Anthropic gets marginalized as shadow sectors of state power pick a different winner? These things are not clear. I do not envy those who must neurotically think through the first-order, second-order, third-order judgements of all of justice, "evil" and "good" that one must do. It's a statescraft level of hierarchy of concerns that would leave me immensely challenged

ben_w 11mo ago

> No, the real risk here is that this technology is going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to a lobotomized and heavily censored version of it, if at all. Given the number of leaks, deliberate publications of weights, and worldwide competition, why do you believe this? (Even if by "lobotomised" you mean "refuses to assist with CNB weapon development"). Also, you can have more than one failure mode both be true. A protest against direct local air polution from a coal plant is still valid even though the greenhouse effect exists, and vice versa.

kouteiheika 11mo ago
> Given the number of leaks, deliberate publications of weights, and worldwide competition, why do you believe this? So where can I find the leaked weights of GPT-3/GPT-4/GPT-5? Or Claude? Or Gemini? The only weights we are getting are those which the people on the top decided we can get, and precisely because they're not SOTA. If any of those companies stumbles upon true AGI (as unlikely as it is), you can bet it will be tightly controlled and normal people will either have an extremely limited access to it, or none at all. > Even if by "lobotomised" you mean "refuses to assist with CNB weapon development" Right, because people who design/manufacture weapons of mass destruction will surely use ChatGPT to do it. The same ChatGPT who routinely hallucinates widely incorrect details even for the most trifling queries. If anything, that'd only sabotage their efforts if they're stupid enough to use an LLM for that. Nevertheless, it's always fun when you ask an LLM to translate something from another language, and the line you're trying to translate coincidentally contains some "unsafe" language, and your query gets deleted and you get a nice, red warning that "your request violates our terms and conditions". Ah, yes, I'm feeling "safe" already.
- astrange 11mo ago
  Kimi-K2-Thinking and DeepSeek-V3.2 are open and pretty near SOTA.
- ben_w 11mo ago
  Imagine saying Operating systems are going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to what computers can really do! Getting the reply We have open-source OSes And then replying So where can I find the leaked source of Windows? Or MacOS? We have a bajillion Linuxes. There's a lot of open-weights GenAI models. Including from OpenAI, whose open models beat everything in their own GPT-3 and 4 families. But also not "those which the people on the top decided we can get", which is why Meta sued over the initial leak of the original LLaMa's weights. > true AGI Is ill-defined. Like, I don't think I've seen any two people agree on what it means… unless they're the handful that share the definition I'd been using before I realised how rare it was ("a general-purpose AI model", which they all meet). If your requirement includes anything like "learns quickly from few examples", which is a valid use of the word "intelligence" and one where all ML training methods known fail because they are literally too stupid to live (no single organism would survive long enough to make that many mistakes), and AI generally only make up for this by doing what passes for thinking faster than anything alive to the degree to which we walk faster than continental drift, then whoever first tasks such a model with taking over the world, succeeds. To emphasise two points: 1. Not "trains", "tasks". 2. It succeeds because anything which can learn from as few examples as us, while operating so quickly that it can ingest the entire internet in a few months, is going to be better at everything than anyone. At which point, you'd better hope that either whoever trained it, trained it in a way that respects concepts like "liberty" and "democracy" and "freedom" and "humans are not to be disassembled for parts", or that whoever tasked it with taking over the world both cares about those values and rules-lawyers the AI like a fictional character dealing with a literal-minded genie. > Right, because people who design/manufacture weapons of mass destruction will surely use ChatGPT to do it. The same ChatGPT who routinely hallucinates widely incorrect details even for the most trifling queries. If anything, that'd only sabotage their efforts if they're stupid enough to use an LLM for that. First, yes of course they will, even existing professionals, even when they shouldn't. Have you not seen the huge number of stories about everyone using it for everything, including generals? Second, the risk is new people making them. My experience of using LLMs is as a software engineer, not as a biologist, chemist, or physicist: LLMs can do fresh-graduate software engineering tasks at fresh-graduate competence levels. Can LLMs display fresh-graduate level competence in NBC? If LLMs can do that, they necessarily expand the number of groups who can run NBC programs to include any random island nation with not enough grads to run a NBC program, or mid-sized organised crime group, or Hamas. They don't even need to do all of it, just be good enough to help. "Automate cognitive tasks" is basically the entire point of these things, after all. And if the AI isn't competent to help with those things, if they're e.g. at the level of competence of "sure mix those two bleaches without checking what they are" (explosion hazard) or "put that raw garlic in that olive oil and just leave it at room temperature for a few weeks it will taste good" (biohazard, and one model did this), then surely it's a matter of general public safety to make them not talk about those things because of all the lazy students who are already demonstrating they're just as lazy as whoever wrote the US tariff policy that put a different tariff on an island occupied by only penguins vs. the country which owned it and which a lot of people suspect came out of an LLM. > Nevertheless, it's always fun when you ask an LLM to translate something from another language, and the line you're trying to translate coincidentally contains some "unsafe" language, and your query gets deleted and you get a nice, red warning that "your request violates our terms and conditions". Ah, yes, I'm feeling "safe" already. Use Google Translate. It's the same architecture, trained to give a translation instead of a reply. Or, equivalently, the chat models (and code generators like Claude) are the same architecture as Google Translate, trained to "translate" your prompt into an answer.

jimbo808 11mo ago

I don't believe that they believe it, I believe that they're all in on doing all the things you'd do if your goal was to demonstrate to investors that you truly believe it. The safety-focused labs are the marketing department. An AI that can actually think and reason, and not just pretend to by regurgitating/paraphrasing text that humans wrote, is not something we're on any path to building right now. They keep telling us these things are going to discover novel drugs and do all sorts of important science, but internally, they are well aware that these LLM architectures fundamentally can't do that. A transformer-based LLM can't do any of the things you'd need to be able to do as an intelligent system. It has no truth model, and lacks any mechanism of understanding its own output. It can't learn and apply new information, especially not if it can't fit within one context window. It has no way to evaluate if a particular sequence of tokens is likely to be accurate, because it only selects them based on the probability of appearing in a similar sequence, based on the training data. It can't internally distinguish "false but plausible" from "true but rare." Many things that would be obviously wrong to a human, would appear to be "obviously" correct when viewed from the perspective of an LLM's math. These flaws are massive, and IMO, insurmountable. It doesn't matter if it can do 50% of a person's work effectively, because you can't reliably predict which 50% it will do. Given this unpredictability, its output has to be very carefuly reviewed by an expert in order to be used for any work that matters. Even worse, the mistakes it makes are meant to be difficult to spot, because it will always generate the text that looks the most right. Spotting the fuckup in something that was optimized not to look like a fuckup is much more difficult than reviewing work done by a well-intentioned human.

HDThoreaun 11mo ago
If you dont believe they believe it you havent paid any attention to the company. Maybe Dario is lying, although that would be an extremely long con, but the rank and file 100% believe it.
vancroft 11mo ago
Sounds like the old saying about the advertising industry: "I know half of my spending on advertising is wasted - I just don't know which half."
11mo ago

beefnugs 11mo ago

Its just with piss and fentanyl were the CEOs exact words, i think the AI would humanely use enough piss to wash away the fentanyl so that minimal deaths will occur. Morality Achieved!

antonvs 11mo ago

The trick here is to focus on imaginary safety from intentional AIs while ignoring the risks posed by real people using AI against other people.

bionhoward 11mo ago

Anthropic also has legal terms that say no one is allowed to use the service for anything work related, but nobody seems to care