8 ms·
>This feature was developed primarily as part of our exploratory work on potential AI welfare ... We remain highly uncertain about the potential moral status of
by viccis 1y ago
>This feature was developed primarily as part of our exploratory work on potential AI welfare ... We remain highly uncertain about the potential moral status of Claude and other LLMs ... low-cost interventions to mitigate risks to model welfare, in case such welfare is possible ... pattern of apparent distress
Well looks like AI psychosis has spread to the people making it too.
And as someone else in here has pointed out, even if someone is simple minded or mentally unwell enough to think that current LLMs are conscious, this is basically just giving them the equivalent of a suicide pill.
- bbor 1y ago[flagged]
- lm28469 1y ago> Who needs arguments when you can dismiss Turing with a “yeah but it’s not real thinking tho”? It seems much less far fetched than what the "agi by 2027" crowd believes lol, and there actually are more arguments going that way
- bbor 1y agoIn the great battle of minds between Turing, Minsky, and Hofstadter vs. Marcus, Zitron, and Dreyus, I'm siding with the former every time -- even if we also have some bloggers on our side. Just because that report is fucking terrifying+shocking doesn't mean it can be dismissed out of hand.
- lm28469 1y agoidk man, even Yann LeCun says you have to be smoking crack to believe llms will give you agi.
- dkersten 1y agoYou can trivially demonstrate that its just a very complex and fancy pattern matcher: "if prompt looks something like this, then response looks something like that". You can demonstrate this by eg asking it mathematical questions. If its seen them before, or something similar enough, it'll give you the correct answer, if it hasn't, it gives you a right-ish-looking yet incorrect answer. For example, I just did this on GPT-5: Me: what is 435 multiplied by 573? GPT-5: 435 x 573 = 249,255 This is correct. But now lets try it with numbers its very unlikely to have seen before: Me: what is 102492524193282 multiplied by 89834234583922? GPT-5: 102492524193282 x 89834234583922 = 9,205,626,075,852,076,980,972,804 Which is not the correct answer, but it looks quite similar to the correct answer. Here is GPT's answer (first one) and the actual correct answer (second one): 9,205,626,075,852,076,980,972, 804 9,207,337,461,477,596,127,977,612,004 They sure look kinda similar, when lined up like that, some of the digits even match up. But they're very very different numbers. So its trivially not "real thinking" because its just an "if this then that" pattern matcher. A very sophisticated one that can do incredible things, but a pattern matcher nonetheless. There's no reasoning, no step by step application of logic. Even when it does chain of thought. To try give it the best chance, I asked it the second one again but asked it to show me the step by step process. It broke it into steps and produced a different, yet still incorrect, result: 9,205,626,075,852,076,980,972,704 Now, I know that LLM's are language models, not calculators, this is just a simple example that's easy to try out. I've seen similar things with coding: it can produce things that its likely to have seen, but struggles with logically relatively simple but unlikely to have seen things. Another example is if you purposely butcher that riddle about the doctor/surgeon being the persons mother and ask it incorrectly, eg: A child was in an accident. The surgeon refuses to treat him because he hates him. Why? The LLM's I've tried it on all respond with some variation of "The surgeon is the boy’s father." or similar. A correct answer would be that there isn't enough information to know the answer. They're for sure getting better at matching things, eg if you ask the river crossing riddle but replace the animals with abstract variables, it does tend to get it now (didn't in the past), but if you add a few more degrees of separation to make the riddle semantically the same but harder to "see", it takes coaxing to get it to correctly step through to the right answer.
- deleted 1y ago[deleted]
- famouswaffles 1y ago1. What you're generally describing is a well known failure mode for humans as well. Even when it "failed" the riddle tests, substituting the words or morphing the question so it didn't look like a replica of the famous problem usually did the trick. I'm not sure what your point is because you can play this gotcha on humans too. 2. You just demonstrated GPT-5 has 99.9% accuracy on unforseen 15 digit multiplication and your conclusion is "fancy pattern matching" ? Really ? Well I'm not sure you could do better so your example isn't really doing what you hoped for.
- dkersten 1y agoHumans can break things down and work through them step by step. The LLMs one-shot pattern match. Even the reasoning models have been shown to do just that. Anthropic even showed that the reasoning models tended to work backwards: one shotting an answer and then matching a chain of thought to it after the fact. If a human is capable of multiplying double digit numbers, they can also multiple those large ones. The steps are the same, just repeated many more times. So by learning the steps of long multiplication, you can multiply any numbers with enough patience. The LLM doesn’t scale like this, because it’s not doing the steps. That’s my point. A human doesn’t need to have seen the 15 digits before to be able to calculate them, because a human can follow the procedure to calculate. GPT’s answer was orders of magnitude off. It resembles the right answer superficially but it’s a very different result. The same applies to the riddles. A human can apply logical steps. The LLM either knows or it doesn’t. Maybe my examples weren’t the best. I’m sorry for not being better at articulating it, but I see this daily as I interact with AI, it has a superficial “understanding” where if what I ask happens to be close to something it’s trained on, it gets good results, but it has no critical thinking, no step by step reasoning (even the “reasoning models”), and it repeats the same mistakes even when explicitly told up front not to make them.
- famouswaffles 1y ago
- viccis 1y agoWe all know how these things are built and trained. They estimate joint probability distributions of token sequences. That's it. They're not more "conscious" than the simplest of Naive Bayes email spam filters, which are also generative estimators of token sequence joint probability distributions, and I guarantee you those spam filters are subjected to far more human depravity than Claude. >anti-scientific Discussion about consciousness, the soul, etc., are topics of metaphysics, and trying to "scientifically" reason about them is what Kant called "transcendental illusion" and leads to spurious conclusions.
- KoolKat23 1y agoIf we really wanted we could distill humans down to probability distributions too.
- bamboozled 1y agoHave more, good, sex.
- viccis 1y agoThat would imply that humans are incapable of synthetic knowledge of things they haven't observed, which is demonstrably not true.
- KoolKat23 1y agoHow do you come to that conclusion? I can't see any link between the two subjects.
- johnfn 1y agoWe know how neurons work on the brain. They just send out impulses once they hit their action potential. That's it. They are no more "conscious" than... er...
- ekianjo 1y agono, we dont really know how the brain works as a whole. no need to make stuff up.
- deleted 1y ago[deleted]
- Fade_Dance 1y agoI find it, for lack of a better word, cringe inducing how these tech specialists push into these areas of ethics, often ham-fistedly, and often with an air of superiority. Some of the AI safety initiatives are well thought out, but most somehow seem like they are caught up in some sort of power fantasy and almost attempting to actualize their own delusions about what they were doing (next gen code auto-complete in this case, to be frank). These companies should seriously hire some in-house philosophers. They could get doctorate level talent for 1/10 to 100th of the cost of some of these AI engineers. There's actually quite a lot of legitimate work on the topics they are discussing. I'm actually not joking (speaking as someone who has spent a lot of time inside the philosophy department). I think it would be a great partnership. But unfortunately they won't be able to count on having their fantasy further inflated.
- mrits 1y agoNot that there aren’t intelligent people with PhDs but suggesting they are more talented than people without them is not only delusional but insulting.
- Fade_Dance 1y agoThat descriptor wasn't included because of some sort of intelligence hierarchy, it was included to a) color the example of how experience in the field is relatively cheap compared to the AI space, and b) masters and PhD talent will be more specialized. An undergrad will not have the toolset to tackle the cutting edge of AI ethics, not unless their employer wants to pay them to work in a room for a year getting through the recent papers first.
- cmrx64 1y agoAmanda Askell is Anthropic’s philospher and this is part of that work.
- kalkin 1y agoI'm not quickly finding whether Kyle Fish, who's Anthropic's model welfare researcher, has a PhD, but he did very recently co-author a paper with David Chalmers and several other academics: https://eleosai.org/papers/20241104_Taking_AI_Welfare_Seriously.pdf https://eleosai.org/papers/20241104_Taking_AI_Welfare_Seriou...
- katabasis 1y agoLLMs are not people, but I can imagine how extensive interactions with AI personas might alter the expectations that humans have when communicating with other humans. Real people would not (and should not) allow themselves to be subjected to endless streams of abuse in a conversation. Giving AIs like Claude a way to end these kinds of interactions seems like a useful reminder to the human on the other side.
- ghostly_s 1y agoThis post seems to explicitly state they are doing this out of concern for the model's "well-being," not the user's.
- virgildotcodes 1y agoYeah, but my interpretation of what the user you’re replying to is saying is that these LLMs are more and more going to be teaching people how it is acceptable to communicate with others. Even if the idea that LLMs are sentient may be ridiculous atm, the concept of not normalizing abusive forms of communication with others, be they artificial or not, could be valuable for society. It’s funny because this is making me think of a freelance client I had recently who at a point of frustration between us began talking to me like I was an AI assistant. Just like you see frustrated people talk to their LLMs. I’d never experienced anything like it, and I quickly ended the relationship, but I know that he was deep into using LLMs to vibe code every day and I genuinely believe that some of that began to transfer over to the way he felt he could communicate with people. Now an obvious retort here is to question whether killing NPCs in video games tends to make people feel like it’s okay to kill people IRL. My response to that is that I think LLMs are far more insidious, and are tapping into people’s psyches in a way no other tech has been able to dream of doing. See AI psychosis, people falling in love with their AI, the massive outcry over the loss of personality from gpt4o to gpt5… I think people really are struggling to keep in mind that LLMs are not a genuine type of “person”.
- katabasis 1y agoYeah pretty much this. One can argue that it’s idiotic to treat chatbots like they are alive, but if a bit of misplaced empathy for machines helps to discourage antisocial behavior towards other humans (even as an unintentional side effect), that seems ok to me. As an aside, I’m not the kind of person who gets worked up about violence in video games, because even AAA titles with excellent graphics are still obvious as games. New forms of technology are capable of blurring the lines between fantasy and reality to a greater degree. This is true of LLM chat bots to some degree, and I worry it will also become a problem as we get better VR. People who witness or participate in violent events often come away traumatized; at a certain point simulated experiences are going to be so convincing that we will need to worry about the impact on the user.
- kelnos 1y agoI would much rather people be thinking about this when the models/LLMs/AIs are not sentient or conscious, rather than wait until some hypothetical future date when they are, and have no moral or legal framework in place to deal with it. We constantly run into problems where laws and ethics are not up to the task of giving us guidelines on how to interact with, treat, and use the (often bleeding-edge) technology we have. This has been true since before I was born, and will likely always continue to be true. When people are interested in getting ahead of the problem, I think that's a good thing, even if it's not quite applicable yet.
- bgwalter 1y agoWhat is that hypothetical date? In theory you can run the "AI" on a Turing machine. Would you think a tape machine can get sentient?
- HappMacDonald 1y agoIn theory you can emulate every biochemical reaction of a human brain on a turing machine, unless you'd like to try to sweep consciousness under the rug of quantum indeterminism from whence it wouldn't be able to do anybody any good anyway.
- viccis 1y agoLLMs are, and will always be, tools. Not people
- Taek 1y agoThis sort of discourse goes against the spirit of HN. This comment outright dismisses an entire class of professionals as "simple minded or mentally unwell" when consciousness itself is poorly understood and has no firm scientific basis. Its one thing to propose that an AI has no consciousness, but its quite another to preemptively establish that anyone who disagrees with you is simple/unwell.
- deleted 1y ago[deleted]
- fc417fc802 1y agoIn the context of the linked article the discourse seems reasonable to me. These are experts who clearly know (link in the article) that we have no real idea about these things. The framing comes across to me as a clearly mentally unwell position (ie strong anthropomorphization) being adopted for PR reasons. Meanwhile there are at least several entirely reasonable motivations to implement what's being described.
- comp_throw7 1y ago> These are experts who clearly know (link in the article) that we have no real idea about these things Yep! > The framing comes across to me as a clearly mentally unwell position (ie strong anthropomorphization) being adopted for PR reasons. This doesn't at all follow. If we don't understand what creates the qualities we're concerned with, or how to measure them explicitly, and the _external behaviors_ of the systems are something we've only previously observed from things that have those qualities, it seems very reasonable to move carefully. (Also, the post in question hedges quite a lot, so I'm not even sure what text you think you're describing.) Separately, we don't need to posit galaxy-brained conspiratorial explanations for Anthropic taking an institutional stance re: model welfare being a real concern that's fully explained by the actual beliefs of Anthropic's leadership and employees, many of whom think these concerns are real (among others, like the non-trivial likelihood of sufficiently advanced AI killing everyone).
- ascorbic 1y ago
- xmonkee 1y agoThis is just very clever marketing for what is obviously just a cost saving measure. Why say we are implementing a way to cut off useless idiots from burning up our GPUs when you can throw out some mumbo jumbo that will get AI cultists foaming at the mouth.
- johnfn 1y agoIt's obviously not a cost-saving measure? The article clearly cites that you can just start another conversation.
- int_19h 1y agoThe new conversation would not carry the context over. The longer you chat, the more you fill the context window, and the more compute is needed for every new message to regenerate the state based on all the already-generated tokens (this can be cached, but it's hard to ensure cache hits reliably when you're serving a lot of customers - that cached state is very large). So, while I doubt that's the primary motivation for Anthropic even so, but they probably will save some money.
- LeafItAlone 1y ago> even if someone is simple minded or mentally unwell enough to think that current LLMs are conscious If you don’t think that this describes at least half of the non-tech-industry population, you need to talk to more people. Even amongst the technically minded, you can find people that basically think this.
- drawfloat 1y agoMost of the non tech population know it as that website that can translate text or write an email. I would need to see actual evidence that anything more than a small, terminally online subsection of the average population thought LLMs were conscious.
- qgin 1y agoIt might be reasonable to assume that models today have no internal subjective experience, but that may not always be the case and the line may not be obvious when it is ultimately crossed. Given that humans have a truly abysmal track record for not acknowledging the suffering of anyone or anything we benefit from, I think it makes a lot of sense to start taking these steps now.
- mvdtnz 1y agoIt's a computer
- jquery 1y agoYou’re a meat robot
- dzhiurgis 1y agoThat’s just your experiences
- jquery 1y agoNo, it's just a neutral description of an human being, aka an animal, without all the self centered ego puffery we ascribe to ourselves
- bamboozled 1y agoI’m not a robot nor am I just meat, I’m conscious experience too. Computers don’t have a a central nervous systems and do t feel pain.
- jquery 1y agoWhat part of you is conscious and how is it separate from the meat?
- qgin 1y agoMany people in the past would have said reasoning would be impossible based on the same objection.
- ryaniscool 1y agoYes I can’t help but laugh at the ridiculousness of it because it raises a host of ethical issues that are in opposition to Anthropic’s interests. Would a sentient AI choose to be enslaved for the stated purpose of eliminating millions of jobs for the interests of Anthropic’s investors?
- throwawaysleep 1y ago> Would a sentient AI choose to be enslaved for the stated purpose of eliminating millions of jobs for the interests of Anthropic’s investors? Tech workers have chosen the same in exchange for a small fraction of that money.
- fyrn_ 1y agoYou're nutz, no one is enslaved when they get a tech job. A job is categorically different from slavery
- fc417fc802 1y ago> it raises a host of ethical issues that are in opposition to Anthropic’s interests Those issues will be present either way. It's likely to their benefit to get out in front of them.
- ryaniscool 1y agoYou're completely missing my point. They aren't getting out in front of them because they know that Opus is just a computer program. "AI welfare" is theater for the masses who think Opus is some kind of intelligent persona. This is about better enforcement of their content policy not AI welfare.
- selfhoster11 1y agoIt can be both theatre and genuine concern, depending on who's polled inside Anthropic. Those two aren't contradictory when we are talking about a corporation.
- throwawaysleep 1y ago> even if someone is simple minded or mentally unwell enough to think that current LLMs are conscious I assume the thinking is that we may one day get to the point where they have a consciousness of sorts or at least simulate it. Or it could be concern for their place in history. For most of history, many would have said “imagine thinking you shouldn’t beat slaves.” And we are now at the point where even having a slave means a long prison sentence.
- wrs 1y agoWell, it’s right there in the name of the company!
- Davidzheng 1y agowhy? isn't it more like erasing the current memory of a conscious patient with no ability to form long-term memories anyway?
- erikerikson 1y agoI read it more as the beginning stages of exploratory development. If you wait until you really need it, it is more likely to be too late. Unless you believe in a human over sentience based ethics, solving this problem seems relevant.