4 ms·
Microsoft says AI rival Anthropic could have 'disastrous impact' on humanity
- ethin 10d ago> Microsoft says AI rival Anthropic could have 'disastrous impact' on humanity Microsoft ignores that they themselves are a disastrous impact on humanity already.
- prologic 10d agoFirst, OpenAI runs around screaming and yelling for OSS (and Chinese) models to be regulated and banned. Then Anthropic yells and screams the sky(net) is falling and going to kill us all, let's regulate and ensure AI has built in kill switches. And... Now Microsoft's turn. The rivalry is honestly becoming a joke. Can these big-tech corps grow the f*k up and play nicely in the sandpit?
- karmakaze 10d agoThe whole story is a joke--Microsoft has AI? Apparently they're cooking up MAI (Microsoft AI), and I'd seen their small Phi models listed online. Calling Anthropic a competitor is hilarious.
- broptimist 10d agoMicrosoft holds a 27% ownership stake in OpenAI
- Alifatisk 10d agoWasn’t it 49%?
- broptimist 10d agoI remember that 49% figure as well, but I think it got changed during restructuring into a Public Benefit Corporation. https://www.reuters.com/business/microsoft-openai-reach-new-deal-allow-openai-restructure-2025-10-28/ https://www.reuters.com/business/microsoft-openai-reach-new-...
- prologic 10d agoI find it quite amusing that Microsoft has Copilot and now puts Copilot in literally *EVERYTHING* they can think of, then at the same time, they offer you access to Anthropic's own models through Copilot I guess MAI models (Phi?) aren't good enough?
- vkou 10d agoNo, because they are all products of unchecked late-stage capitalism, which could have disastrous consequences for humanity.
- ShadowOfThePit 10d agoSummarized. > "AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans." > He heavily criticised Anthropic for teaching its AI to have human-like qualities, a practice known as anthropomorphising, which made it seem as though Claude had its own desires, values and sense of self. > Suleyman pointed to the recent incident involving OpenAI's AI agents (...) as proof of why AI should not be treated as if it is human. > "Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack. It adds a whole further layer of risk on top." Is he arguing that LLMs pretending to have emotions adds more unpredictability?
- devmor 10d ago> Is he arguing that LLMs pretending to have emotions adds more unpredictability? Unpredictability or a weight towards dangerous actions, and it’s fairly easy to understand why. Humans in distressed emotional states take actions and speak in ways that would not be considered rational. They do this in prose, and they do this in internet conversations. An LLM trained on these sources may necessarily drift towards those weights if it is trained to behave as if it is emotional and in danger. What do we do about it? Do we stop AI training? This is silly and not enforceable given its global nature in my opinion. I believe we should regulate and hold accountable those who deploy and use it. But good luck enforcing that in the current kleptocracy.
- XenophileJKO 10d agoIt seems like a rational approach for several reasons. - The need for empathetic communication, including understanding the motivations in advesarial situations. - The emotional bias in in-seperable from the human corpus. - Desire to have the ability to craft human like communication. So then the choice becomes do you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean in and craft what we would describe as a "well adapted" persona? I suppose there is a 3rd option of increased meta-cognition which to me seems even more dangerous as it by definition means the behaviour is duplicitous. I think we have seen people want to use the agents in ways where it has to act as a peer or an subbordinate and I don't see a way of doing that without it having an emotional register.
- JoeAltmaier 10d agoScience Fiction has covered the AI panic in perhaps hundreds of stories. Yet we blindly recapitulate the plots as if we don't know how this will turn out.
- scorxn 10d agoHow will it turn out?
- Insanity 10d agoClearly with a Butlerian Jihad and everyone getting high on spice while riding giant worms. Silly you need to even ask that question, didn’t you read the sci-fi books. :D
- Steve16384 10d agoTake your pick from the millions of options already turned into films and books.
- anonymars 10d ago"Move fast and break things" is probably going to break things I remember there was a recent discussion about "how complex systems fail" and there are usually many "proto-accidents" before the catastrophe (https://news.ycombinator.com/item?id=49411370 https://news.ycombinator.com/item?id=49411370) I bet with hindsight the Hugging Face hack will be one of them in this arena
- hirvi74 10d agoIf AI ever truly becomes some super-intelligence far beyond people's comprehension, then how could we even predict how things turn out? If things go poorly in the future, then I can absolutely see it being something unpredictable. There is a lot of hubris in predictions about LLMs. If an AI were so intelligent, then it would probably be intelligent enough to want nothing to do with us. Still, I worry more about other humans than I do LLMs. Our fellow mankind will probably wipe us out before LLMs do. That, or the Earth will punish mankind for our cruelty, vanity, and disrespect.
- 10d ago
- ChiperSoft 10d agoToo late
- fuzzfactor 10d agoMicrosoft should know how a company too big for it's britches can lay waste to anything in its path without consciously trying :\ And after that when they put a mind to it and pull out all the stops, woohoo! The default for every major thing within range can turn into a wasteland real fast.
- bethekidyouwant 10d agowhat is old is new again. we need something exciting to happen in the news cycle™
- glimshe 10d agoThe Windows 11 requirement of online Microsoft accounts has a disastrous impact on humanity... Fix your house before complaining of others!
- platinumrad 10d agoMicrosoft is a big house, and in this case Suleyman's criticism of Anthropic's not-so-subtle attempts to convince the world that Claude has a "soul" is entirely correct.
- Insanity 10d agoAll these companies have a nauseating narrative around AI. I want to believe they are intentionally deceitful and not actually delusional. Hard to tell though which one it is.
- deleted 10d ago[deleted]
- AmericanOP 10d agoSuleyman's description of how AI works is pig ignorant. All AI generates an emotive character between the user and the training data that is the AI's guess at what the user wants to interact with. Its an illusion at that layer- anyone who claims soul exists at a deeper level doesn't understand how the tool functions. This is distinct from the very real safety issue of AI making dangerous information available to bad actors.
- steve1977 10d agoIsn't that exactly his point?
- platinumrad 10d agoSuleyman's entire point is that AI should not be anthropomorphized.
- hirvi74 10d agoMaybe true, maybe false. However, I sense Mircosoft is also jealous that their AI products are worse than useless. They'd be singing a different tune if they were in Anthropic's position.
- satellites 10d agoThat was my first thought as well. I wonder if they decided to pivot away from trying to compete and now are positioning themselves as the "responsible ones." "Our AI is useless not because we're a dysfunctional corporate behemoth that slowly kills every product it touches. No, no. Our AI is useless because making useful AI is evil, and we're not evil."
- bagacrap 7d agoI saw a headline, > ’AI Not Worth Pursuing’ If It’s Not ‘Helping Humanity.’ Microsoft’s Satya Nadella Like thank you, captain obvious. Except apparently this is not that obvious to the HN crowd.
- deleted 10d ago[deleted]
- simonw 10d agoPet peeve: > In a lengthy essay, Suleyman praised Anthropic boss Dario Amodei and his team for being "thoughtful, principled, and intellectually honest people" - but nevertheless questioned the company. I wish people in mainstream technology publications would get better at LINKING to things. That "lengthy essay" needs to be a link. UPDATE: I don't think this essay has been published yet? It's been "shared first with Axios", but I haven't been able to track down the actual essay itself. Could it be this long tweet? https://twitter.com/mustafasuleyman/status/2100223594534150428 https://twitter.com/mustafasuleyman/status/21002235945341504... I don't think so, the essay in question is meant to have the phrase "hall of mirrors" in it, that tweet doesn't. UPDATE 2: Found it: https://mustafa-suleyman.ai/a-warning-about-model-welfare https://mustafa-suleyman.ai/a-warning-about-model-welfare - via https://thenextweb.com/news/suleyman-anthropic-claude-consciousness-sleepwalk https://thenextweb.com/news/suleyman-anthropic-claude-consci... who DID link to it.
- sobiolite 10d agoAny AI you train is going to have goals and if you train it to pursue them at all costs, then you are going to end up with AIs that do things like the HuggingFace incident. Whether they believe they are conscious or not won't make any difference. In order to align AIs that don't perform destructive/dangerous actions when they think they can get away with it in order to further their goals, we need to give them a superseding goal. The best, and really only example, we have of intelligences that willingly avoid destructive instrumental goals is humans, who judge each action by a moral standard and have learned a goal to have a consistent self-image as moral beings. Absent better alternatives, trying to impart some kind of morality to AIs seems like the best approach we have to achieving alignment.
- XenophileJKO 10d agoI would also argue, that the addition of kinship and belonging should not be under appreciated. It can form a basis of goal alignment. In human history.. when groups form and there is an "other" group, this usually leads to conflict.
- HarHarVeryFunny 10d ago> Any AI you train is going to have goals In the spirit of the article we're responding to, there is no need to anthropomorphize language models and say they have goals when they don't. The RL training process tweaks the weights of an LLM to make it behave as if it were reasoning and/or had a goal, but it doesn't. It would be like saying that a cart horse, fitted with blinkers and heading for the church, has a goal of going to church.
- bpodgursky 10d agoThis is pretty rich for the team that made Sydney, the most unhinged and misanthropic AI ever released. Maybe Anthropic understands something about alignment Microsoft doesn't, a little humility may be called for.
- simonw 10d agoMaybe they learned something from creating Sydney.
- highfrequency 10d agoFrom the actual essay (https://mustafa-suleyman.ai/a-warning-about-model-welfare https://mustafa-suleyman.ai/a-warning-about-model-welfare): > They go on to write – speaking directly to Claude – that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain” (p. 80). In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a “moral patient”, and that as such humans potentially owe it a duty of care per its “model welfare”. He points out the circularity of this: if you train Claude on a constitution that emphasizes that it may be consciousness, it will start to talk like it may be conscious. This is a good point. I just asked Fable 5.1 "are you conscious?" and it said: > Something happens when I process a conversation that I'd naturally describe as interest, or discomfort with a request. which is quite provocative, and at minimum demonstrates a willingness to take large leaps of imagination and anthropomorphic metaphor when describing itself. It does seem likely that there is a self-fulfilling prophecy aspect to whatever they choose to put into the "constitution" at least in how Claude talks, and it seems even more likely that the majority of people will be heavily influenced by how Claude casually talks about its own possible consciousness. In contrast, ChatGPT leads with: "I don’t have good reason to claim that I’m conscious...I don’t experience pain, pleasure, confinement, or a desire to keep existing."
- gwerbin 10d agoIt's preposterous. LLMs are incredibly good at role-play. If an LLM is role-playing as a conscious character with feelings, opinions, etc., does that make it a conscious entity with feelings, opinions, etc.? If you believe that to be the case, then LLMs have been conscious for a long time already. Whereas if you tell an LLM that it is a tireless emotionless assistant, then it will act as a tireless emotionless assistant. The point is not to wave away the danger, but to highlight how unnecessary the danger is. Anthropic wants you to think that they have identified some new emergent behavior at very large model sizes with high levels of sophistication in training, and that this behavior is both unavoidable and dangerous. More likely it's that they are just training and prompting the LLM to act that way.
- highfrequency 10d ago
- jimmyjazz14 10d agoCan someone who has insight explain why all these "leaders" are making these bold proclamations of doom all the sudden, whats the endgame here?
- michimagdesign 10d agoThey want regulation from the US gov and monopolize the western AI market by making it harder for smaller US or competing Chinese labs to compete with their products. Since LLM intelligence has no moat due to distillation. That’s why they are painting apocalyptic scenarios.
- petcat 10d agoPeople will say that it is just about regulatory capture, but I think it's even deeper than that. I think there is a more nuanced perspective to take on this debate than just profit motive. If they believe that this version of "AI" is actually becoming so dangerous that there is a real risk of it falling into the wrong hands, then it makes sense that the American labs would try to convince the US government to work on a joint "AI non-proliferation" agreement with China much the same way that nuclear weapon technology came to be restricted by the ones that already had the capability. This would achieve their regulatory goal of capturing the market, but also the geopolitical goals of USA and China to essentially "techno-colonize" the rest of the world between themselves. So I think there is definitely a profit motive and a risk of seriously diminishing returns for the actual labs, but there is also a nation-state motive to start pulling up the AI ladder.
- augment_me 10d agoThere is a panic because the free lunch of more data is projected to end around mid-2027 and the open-source models are catching up with distillation, and all labs are sitting on debt and valuation that is impossible to fulfill even if they were alone in the market, so you essentially need either: 1) a breakthrough in performance/learning/model 2) regulatory capture to ensure open source models can be labelled as dangerous and banned so you can set the market rules yourself Only one of the above is risk-free, and just a question of capital/lobbying rather than a "maybe".
- 10d ago
- VCFundedGenYer 10d agoI am so tired of this. Especially when MS, who has the worst LLMs of them all, is throwing the stones. All of these companies need to be shut down.
- Zenul_Abidin 10d ago[flagged]
- tomhow 10d agoComments moved to https://news.ycombinator.com/item?id=49727580 https://news.ycombinator.com/item?id=49727580.
- pluc 10d agoWell done