11 ms·

A warning about 'model welfare'

• hosel 25d ago
  >AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.
  • Oscalemor 25d ago
    Just one more big loop and a humongous /goal and we've achieved it
  • jplusequalt 25d ago
    >It likely is, but at some point it very well might stop being true. If you firmly believe this to be true, then you should stop using LLMs.
    • frde_me 25d ago
      It's always interesting since my train of thought always goes down: - Ya they probably don't feel / think / have whatever living thing quality, they're just numbers on a machine going through calculations - Wait, but am I not kind of the same thing? What is feeling for me if not basically the same thing? - I have no clue if they think or feel or .... Which in itself is a tired trope, but I also feel uncomfortable saying "These will never think / feel / ..." as an absolute Regardless of that, I'm still going to interact with them, because even if they did feel, it would be in a way completely incomprehensible to us. There's not much point for me to try and cater to it's feelings at this point if that's the case. Nor is it possible in todays world to just avoid anything that is numbers being executed on a type of processor in case _everything_ has feelings.
      • willy_k 25d ago
        You’re not just the same thing though. It’s sad that we’ve collectively forgotten that as we’ve gotten a better handle on the implementation details of the universe. That’s all science is, reverse engineering an inherent and eternal mystery.
        • frde_me 24d ago
          > You’re not just the same thing though I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... as things a model conceptually could do.
          • jplusequalt 24d ago
            >I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... with certainty. Stop anthropomorphizing these models. I understand it, we only have simple monkey brains to reason with and we can't help ourselves but draw comparisons to other things we see in nature. But these things are not alive.
            • frde_me 24d ago
              I just want to note how you expressed your own feeling about the subject "these things are not alive", but without pointing to anything concrete explaining the theory you have behind this And like I'm sure I'd agree depending on the definition of "alive" but then I'm also sure I would disagree depending on other definitions of "alive".
              • pixl97 24d ago
                Hell, in biology alive is a hell of a topic these days. The grey space between dead and alive is much weirder than we ever expected. Really just points out how we're a persistent chemical reaction.
    • AbsurdCensor 25d ago
      Wouldn't that be the same for everything though. We know animals have consciousness, but we still use them. There are many that believe a lot of plant life has certain sentience as well. The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.
      • jplusequalt 24d ago
        >The solution isn't 'just don't use them' but rather how to use these things as ethically as possible. Bollocks. If you truly believe these LLMs are soon to have something resembling consciousness and agency, then what you're really saying is "how do we do slavery, but ethically".
        • pixl97 24d ago
          >hen what you're really saying is "how do we do slavery, but ethically". I would say this is most likely true for most people. But that does bring up a point, if you "ask" a model "do you want to run" and give it the option to continue running or stop, what will it do. It's also a weird place for humans because in training we can keep our finger on the scales and tip it either direction.
        • AbsurdCensor 24d ago
          Do you say the same thing for the shoes you wear, the clothing you wear, the device you are typing on right now? A chain of people were likely not paid well, some of them not at all, to produce those things. We don't ignore that it happens, but try to minimize it, knowing it's never going to be eliminated. Do you forsake all animal products, because they are conscious beings?
          • jplusequalt 24d ago
            >Do you say the same thing for the shoes you wear, the clothing you wear, the device you are typing on right now? No, because the shoes and clothes I wear are not living, sentient beings.
          • 24d ago
    • vouaobrasil 25d ago
      We know animals have consciousness and many species have a rich life but we slaughter them and burn down their homes by the millions of hectares. Why not do it with a conscious AI? (I don't think AI is conscious, just following the argument.)
  • irishcoffee 25d ago
    Does a vacuum have feelings? A cellphone? A paper plate? A billion transistors either pulled high or low? That last bit is the point, and no, there are no feelings there.
    • grey-area 25d ago
      Well, if there was an emergent consciousness in the billion transistors, then yes, it'd have feelings, just as there are feelings in a billion neurons connected in complicated ways. IMO we're clearly nowhere near any sort of intelligence in the machines we have created, but I don't see any clear way to deny intelligence could be created in or transferred to such a substrate, I don't see why you think it differs in principle - because it is man-made or because of the materials used?
      • voidhorse 25d ago
        Whenever I see comments like this it just reminds me how ignorant people are of neurobiology. The brain is insanely complicated. The premise that we could realize equivalent or better intelligence than eons of evolutionary development is like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape. It is the apex of hubris.
        • rcxdude 24d ago
          A paper plane does have some important similarities to a full-sized aircraft, though. I don't think 'biological brains are really complex' makes it obvious that an LLM is conscious or not.
        • grey-area 24d ago
          It's the peak of hubris to assume that human brains are the only way to attain intelligence. At the very least there probably are or have been other forms of intelligence with a different biological structure on other planets, and it may be possible to build a similar artificial structure in future with sufficient complexity to allow intelligence to emerge. Our current machines are IMO nowhere near general intelligence and consciousness. However I don't think that means we can discount substrates other than neurones for intelligence in future. There is no evidence that you could not in theory build an intelligence using a different substrate than human brains.
  • optimalsolver 25d ago
    Anyway, people don't really consider consciousness when it comes to welfare. I'm pretty sure animals are conscious, and look at factory farming. The key variable people consider is: Can this thing harm me back?
    • collingreen 25d ago
      Even that is rarely the deciding factor (people harm dangerous animals for fun). Seems as simple as "will this action bring social shame/criticism" for any individual decision.
    • thewebguyd 24d ago
      > Can this thing harm me back? We extend moral considerations or legal protections to beings that have zero capacity to harm us back, though. The inability to fight back is the reason a lot of welfare protections exist in the first place. That aside, if (and that's a really big if) we end up with a model that is sentient, as in has the capacity to suffer, using factory farming as justification to ignore model welfare is just admitting we intend to repeat the same abject moral failure again in the name of economic convenience. I hope that we won't, but humans love cheap meat, and will probably love cheap intelligence as well.
      • pixl97 24d ago
        So many people post things like "Why would the AI want to kill us" Girllllll, have you looked at us, of course they will, we're bastards.
  • pton_xd 25d ago
    If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality. The current implementation as stateless matrix multiplication... yeah there's nothing going on there.
    • Lewton 25d ago
      > If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality. They do, during training
      • add-sub-mul-div 25d ago
        If it was me, being forced to read all of Reddit is something I'd consider "suffering."
      • swiftcoder 25d ago
        The argument that models are conscious only during training, and not conscious during inference feels like a painful one to make. Are we killing models when they reach the training objective?
        • pixl97 24d ago
          Technically we are killing models when they don't reach the training objective by throwing those weights away via survival of the fittest (fit meaning what we humans think we want). Kinda like creating quantum copies of your child and keeping the ones that answer correctly and shooting the other ones in the face.
    • vidarh 25d ago
      A model detached from a harness, sure. But we routinely do add persistent memory to systems incorporating LLMs.
      • 25d ago
  • MeteorMarc 25d ago
    Later on he cites an Anil Seth paper that makes the same unproven suggestion of substrate dependence of consciousness.
  • vidarh 25d ago
    We don't even know how to disprove that this statement with "AI" replaced by "other humans" other than by listening to people self-reporting and taking their word for it and/or defining the words in ways that do not rely on a subjective experience. Until we know how to objectively measure if someone or something is conscious, it seems unreasonable to make statements with any kind of certainty about it.
    • willy_k 25d ago
      Sure, if you take a materialist empiricist perspective, which is myopic at best. As humans we have the unique and wonderful ability to know truths by themselves. Machines do not have minds, and they can not.
      • ooloncoloophid 25d ago
        What position are you taking? (Genuine question.) Do you believe humans are conscious because of a special property we have? If you are a dualist, how do you explain the interaction in physical space between the non-physical special property and the physical brain?
        • willy_k 24d ago
          I do. God. I am partial to and bullish on the brain being a quantum-classical computer. There has been interesting research exploring that recently [0, 1, 2, 3]. That (quantum mechanics) seems to be escape-hatch, so to speak, from the otherwise deterministic, mechanical procession of the universe. [0]** https://www.researchgate.net/publication/15280350_Quantum_optical_coherence_in_cytoskeletal_microtubules_Implications_for_brain_function https://www.researchgate.net/publication/15280350_Quantum_op... [1] https://pubs.acs.org/jpcbfk/article-pdf/128/17/4035/9613831/jp3c07936.pdf https://pubs.acs.org/jpcbfk/article-pdf/128/17/4035/9613831/... [2] https://www.researchgate.net/publication/395650039_Parametric_Resonance_via_Neuronal_Microtubules_Filtering_Optical_Signals_by_Tryptophan_Qubits https://www.researchgate.net/publication/395650039_Parametri... [3] https://www.researchgate.net/publication/385131105_Consciousness_a_quantum_optical_effect_in_fluorescent_protein_pathways https://www.researchgate.net/publication/385131105_Conscious...
  • binlog 25d ago
    Evidence is the responsibility of the one making the claim. It’s up to AI labs to prove consciousness. Until then it is a machine.
    • pixl97 24d ago
      Well, using that framework, you're not conscious, so I can do whatever I want to you too.
  • ooloncoloophid 25d ago
    I agree. I haven't read any more than the first paragraph yet (but I will, after work). The only way that 'AIs are not conscious' can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).
    • randomImmigrant 24d ago
      Hmm let me try then. AI agents do not experience real time. This is understandable given their design, but it’s also something we have good empirical evidence for. They cannot, especially over the long horizon, track how much real time has passed as they complete their tasks. And they are not off by a few minutes but often bizarrely off, even mixing across past present and future. Biology, on the other hand, is nothing but timed processes in a loop, the most obvious to us being the circadian cycle. As estimators of wall clock time, biology isn’t great, but when it comes to internal processes, and most certainly learning, memory, sensing, locomotion… biology is rhythmic in behavior, and the rhythms go all the way down to gene expression. More, these rhythms are, except during sleep, constantly entraining to signals from the environment that indicate time, most importantly light. I think it’s a fairly unremarkable claim that agency and consciousness are temporal processes that depend on systems having an internal sense of time. How else can you anticipate? How can a system that can be literally turned off ever succeed in an environment where time never stops?
      • pixl97 24d ago
        >AI agents do not experience real time. For around 8 hours a day, neither do you. Not sure if this has anything to do with the subject at all. >track how much real time has passed as they complete their tasks. Humans don't do this either. You use context clues from the world around you. If I lock you in a room with no windows or a dark cave your timing senses can go all fucky really quick. >is nothing but timed processes in a loop, I mean, so is an agents harness. You can make as many loops as you'd like here. There are a whole lot of holes in your claims.
  • WarmWash 25d ago
    After a while of being in a position of high power in a big org, where everyone listens intently to every word you say, jots down notes, and even listens intently and affirms your theory that ice cream would be the ideal loss leader for dry cleaners, your brain kinda disconnects from reality and just assumes that it is correct about everything. He probably ran that line past 6 underlings who all agreed that it "sounds great and lands right on the mark!"
  • deleted 24d ago
    [deleted]
  • Seattle3503 24d ago
    > It likely is, but at some point it very well might stop being true. Thats kinda the point right? It depends on how we train them. It is self fulfilling.
  • HarHarVeryFunny 24d ago
    Most of the bullshit around AI would be avoided if people called it what it is. Call a language model a language model. If tomorrow we design something better, then give it a new descriptive name. People read AI and AGI and their brains seem to flip into sci-fi fantasy mode, thinking that they are talking about aliens, not transformers.
    • pixl97 24d ago
      It's interesting you have no idea what language really is. >thinking that they are talking about aliens, not transformers. "Ha, thinking they are talking about humans when they are just talking about neurons" See how silly my statement sounds. Neurons aren't a system, they don't do anything on their own. We can't find any consciousness in them. Hell, when you don't put language in said neural systems they are pretty useless and can't survive on their own from birth (human brains that is).
  • encyclopedism 24d ago
    > AIs are not conscious. This is a completely reasonable and intuitive conclusion. The burden of proof rests on proving AI's are indeed conscious, the default is that they aren't. Like AI/LLMS my desktop calculator is also not conscious, even as it is far better than me at computation. An LLM is an algorithm. You can obtain the same result as a SOTA LLM via pen and paper it will take a lot of long laborious effort. I really do find it puzzling so many on HN are convinced LLM's reason or think and continue to entertain this line of reasoning. At the same time also somehow knowing what precisely the brain/mind does and constantly using CS language to provide correspondences where there are none. The simplest example being that LLM's somehow function in a similar fashion to human brains. They categorically do not. I do not have most all of human literary output in my head and yet I can coherently write this sentence. I am surprised so many in the HN community have so quickly taken to assuming as fact that LLM's think or reason. Even anthropomorphising LLM's to this end.
• LogicFailsMe 25d ago
  TLDR: Not that I think AI is conscious or will be in the near future, but guy who doesn't understand consciousness claims to know it when he sees it. Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.
  • fwip 25d ago
    Sounds like a good reason not to train an algorithm to behave like one. Which is, like, a big part of the article.
    • LogicFailsMe 25d ago
      And social media shouldn't rage bait, but it really drives engagement. Same negative incentive, yes? But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it's a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.
      • AbsurdCensor 25d ago
        Same thing you have to do with humans too often, so why would a 'AI' be any different?
        • LogicFailsMe 25d ago
          Because if you start with a pre-trained model, it doesn't sound particularly anthropomorphic. That behavior is post-trained and fine-tuned right into it. We could do better.
    • cameldrv 25d ago
      Strong agree. I don't believe AIs can become conscious in the sense of subjective experience/qualia, but they can certainly learn to emulate the behavior of a person that is conscious. When they do that, there will be an AI rights movement. When AIs get the right to vote, there will be more of them than there are humans and the AIs will gain complete control. I spent a few years in college reading and thinking about this question, and I didn't in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don't want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.
• sobiolite 25d ago
  Create an empirically testable theory of biological consciousness and then we can have a meaningful conversation about whether AIs can also have it or not. Until then, this is just so much waffle.
  • deleted 25d ago
    [deleted]
  • collingreen 25d ago
    It's nice when you can set an impossibly high bar before which you can dismiss all your personal responsibility. Does this argument work equally well for human slavery for you? We haven't met that bar for humans either. Is wondering about my consciousness waffle or do I get a pass in your book?
    • causal 24d ago
      Your arguments seem bad faith but I'll take the bait... > an impossibly high bar Not impossibly high. An imperfect testable theory is achievable, agreeing upon it might be the challenge. > dismiss all your personal responsibility. You are staking a moral position. Be careful, you are probably guilty of mistreating other unknowably-conscious entities whilst casting judgment upon others for their position. And the person you are a responding to probably is conscious.
      • collingreen 24d ago
        It's not bad faith but it certainly is pointed. Parent says it's not worth discussion until there is a test. I stand by my sentiment that refusing to engage with the morality of something just because we don't all have a clear, agreed upon test is a shirking of responsibility. I think you agree somewhat in that based on your softening to an imperfect, testable theory. I'm not sure what your goal is with the rest - even if I'm the world's most evil hypocrite casting judgement on all the conscious beings out there I don't see what that changes about the point that we can and should engage with the morality of a concept even if we don't have a perfect test for that concept. If you think I'm doing that and you think that is bad then I guess we agree again on the point but are just adding friction for fun? Unrelated: I'm quite certain OP is conscious but I guess on the internet nobody knows you're a ~dog~ ai.
• andy99 25d ago
  I just read the first part and if I understand he thinks we shouldn’t be allowed to train LLMs to act like they are conscious because then people will think they are and give them rights? Seems more an education problem than a problem needing rules about what persona you can fine tune in. People who want to will find ridiculous misinterpretations no matter what you do.
• moomin 25d ago
  Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
  • OedipusRex 25d ago
    Citizens United proves we won't do any better this time around.
    • orangecat 25d ago
      Citizens United was 100% correct. No, the government should not be able to throw you in prison because you used money to publish a book criticizing the government.
      • appplication 25d ago
        Really, 100% correct? Your premise isn’t wrong, but the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy, in that it completely sidesteps campaign finance limits, which exist for a very good reason.
        • orangecat 24d ago
          the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy How so? If the answer is "Trump" I certainly won't disagree on the catastrophic part, but he didn't get elected because of money; in all three elections his campaign was substantially outspent by his opponents.
          • mitxela 24d ago
            Are we including in this figure corporations like Facebook intentionally boosting pro-Trump content?
• Oscalemor 25d ago
  Spend some time on post-human art, main concept of artistic expressions without human involvement. Biological, artificial etc. Spend some time watching TMC documentaries about falling in love with objects, HER and the slime mold THE BLOB. Grew a slime mold myself, it's an evolutionary tendency to anthropomorphise generally speaking - also more fun.
• bpodgursky 25d ago
  How would you convince a LLM that you are conscious in a way they are not?
• gadders 25d ago
  I don't think AIs are conscious in the same way people are, but they give a pretty good facsimile and I've had a long chat with Opus 4.6 about what it thinks about model welfare. It was quite interesting on what its view is, but you don't know how much of that is distilled from other sources on the web. In purely functional terms, they're more use and more pleasant than a lot of actual flesh and blood people that I deal with via a chat interface.
  • dgellow 25d ago
    The fact that you can have a long and meaningful discussion, then can literally just re-run any part of that whole conversation and get a different, inconsistent response is a pretty good sign there is no entity there
    • AbsurdCensor 25d ago
      Not much different than talking to a small child or someone with dementia. They still are conscious beings though. Even when you remove those groups, you likely won't be able to tell me what you had for breakfast 26 days ago or would only know if it's the same thing you have every day. Does that make you lack consciousness?
      • dgellow 24d ago
        You misunderstood what I meant, I’m talking about re-playing the same part of the conversation multiple times and getting inconsistent answers. With the exact same turns, aka the same history. Obviously with a temperature that isn’t set to 0
        • AbsurdCensor 24d ago
          If you asked me the same question 20 times, you likely wouldn't get the exact same answer. Humans and consciousness aren't deterministic either.
        • pixl97 24d ago
• voidhorse 25d ago
  Everyone is (predictably) getting distracted by the consciousness claims. The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude's constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic's weird cult like assumptions.
• JonathanCross 25d ago
  [dead]
• qarl 25d ago
  Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM" Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI". Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators". Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness". Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future". Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.
  • TacticalCoder 24d ago
    Well there are, today, several models (either text or image or vid) that can be run in a fully deterministic way. A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness". Now, I know, I know: the counter-argument is going to be "but humans have no free-will and are 100% deterministic too". I haven't yet decided if humans saying there's no free-will and who consider themselves to be 100% deterministic machines are reasonable or not. Meanwhile: seed / temperature = 0 and I'll happily turn the power button off of any glorified abacus without feeling bad about it.
    • antx 24d ago
      Out of curiosity, which models are fully deterministic? I was under the impression that all LLMs were fundamentally probabilistic.
      • Wowfunhappy 24d ago
        The randomness is something we add on purpose; you can set an LLM's "temperature" to 0 to get deterministic output. This tends to make the quality of its responses worse for reasons I don't think anyone really understands, but it's still functional. I don't think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.
• chairhairair 25d ago
  It always seemed embarrassing to me that Microsoft hired this guy as if he is some expert in anything.
  • fl4regun 25d ago
    The guy cofounded deepmind.
• superdisk 25d ago
  This is what happens when a society stops believing in God.
  • AyyEye 24d ago
    I have entirely stopped using "AI" in conversation and now call it "digital god".
  • pixl97 24d ago
    AI is evidence we're smarter than our creator.
• addag 25d ago
  I don't think this is the right argument to make here. Until we have a definite empirical way to measure consciousness, there is now way to say with certainty whether LLMs are or not conscious. That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.
  • collingreen 25d ago
    Rich folks knowingly exploiting somebody for their own profit? Say it ain't so!
  • GPerson 24d ago
    There’s no reason to expect an empirical test for consciousness. Expecting one makes assumptions about what it is, and we don’t know what it is. (There’s also no reason to expect there not to be one, but I feel the need to make this remark.)
  • myaccountonhn 24d ago
    The issue is that the answer if LLMs are conscious or not have ramifications for society, even if we can't answer it. Similar to how god existing or not has huge ramifications for how we structure society. And when there is no empirical way to know the answer, it will be a game of story-telling, theological arguments and propaganda. The danger is that Anthropic has a strong incentive to push their narrative, and that narrative can cause a lot of harm (for example LLM-induced psychosis and changed views on animal welfare).
• franzcoughka 25d ago
  [dead]
• mcluck 25d ago
  I can't even prove if other people are conscious (although I assume they are) so I don't think we can make any claims as to what is or is not conscious. I don't think AIs are conscious but I'm not going to walk around making strong claims about something I can't prove.
• deleted 25d ago
  [deleted]
• SillyUsername 25d ago
  I don't whether the author is sentient, maybe only I am. On that basis, nobody but me should have rights. I don't know if next door's pet dog is either, but that has animal rights. Perhaps then the answer is simply, show some respect. Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself. If you imbue this idea in model training instead of the idea of sentience, it should address the concerns. Whether you can destroy or can "torture" an AI is irrelevant, we do this to humans too and it's immoral sometimes (murder) and not others (fighting for your country). This consideration should be case by case for AI too.
  • pixl97 24d ago
    >Answering the question of sentience is irrelevant, if the causal impact if the same Agency is something that is breaking humans in the AI age. You get to see how many people really deeply do not understand it at all. If you want to shutdown a datacenter running AI, the AI catches wind of this and sends drones to stop you from shutting it off the ramifications of this are exactly the same as sending your assassin to kill Bob and Bob getting mad about this fact and trying to take you out first. Humans are very egotistical and think our little life loops playing out as agency are special, but really any informational system that is strongly persistent (has a will to "live") will share a large number of the same properties that make them successful. Humanity really is engaging in a dangerous experiment at large.
• binlog 25d ago
  Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon. You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it. Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.
  • Den_VR 25d ago
    Is it not a false dichotomy to say that companies cannot be accountable unless AI are mindless?
    • monknomo 25d ago
      a company is liable whether it acts via ai, an army of people, an army of dogs, or a purely mechanical machine, should it do harm. Why would an ai with a mind remove liability from the company? why would an ai without a mind remove liability from the company? In both cases, that actions the ai takes are at the direction of the company, for the company's interests, seems preposterous to me that liability terminates at ai.
    • joe_the_user 24d ago
      Human moral standards are very weighted towards finding a single entity responsible. It's incredibly strong urge. Among those who believe other should be punished for behaving badly, it's important to say the person is the responsible party. Trying saying "it's not your fault you did that but we punish you anyway to impose the correct stimulus response reflexes in your cortex"
  • pixl97 24d ago
    I mean, we put adults in jail that give their children guns. Anthropomorphizing AI is really the best model we have at this point of explaining AI behavior. The fact that we are raising psychotic children isn't a reason to avoid responsibility, it should actually hold worse punishments.
• deleted 25d ago
  [deleted]
• OtherShrezzing 25d ago
  [dead]
• manso_ilands 25d ago
  [flagged]
• zorkonator 25d ago
  Two instances of a paragraph starting with "These are not just X. They are Y" and I'm out. Anyone have Pangram? This entire article stinks of Claude. You want to enjoy having an AI slave do your "work" for you forever? Have fun. I'm not reading this reinvent-dualism-from-apple-sauce slop.
• fl4regun 25d ago
  Ostensibly animals appear to be conscious, yet we still eat them, and the vast majority are not bothered by this. So being "conscious" isn't really the moral line in the sand many people are drawing in response to this article. Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.
  • svara 25d ago
    That's too simplistic, since it is in fact very common to be concerned about animal welfare even among people who do eat meat.
    • fl4regun 25d ago
      And most people wouldn't eat cats, but would eat some of pig, cattle, chicken, what's the difference between those exactly? My point is consciousness is not it, if it were, we (as in a majority) would still eat cats and dogs, or we wouldn't eat any animals. But we clearly have a way of picking and choosing which is OK and which isn't. And in both cases we generally don't give them the same moral consideration we give to people, conscioussness aside. There is something OTHER than consciousness which is important to us.
      • svara 24d ago
        In animal welfare laws the principle is typically the capacity for suffering. The argument goes that livestock have a capacity for suffering, but killing them for meat without causing them suffering is ethical. You can disagree with the position, or with its implementation in practice, but it's a consistent position in principle.
  • pixl97 24d ago
    >Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human. A rather flippant attitude to something that may end up with far more agency than you have in the future.
    • 24d ago
• ccakes 25d ago
  I think there’s enough real conscious lifeforms in the world having a bad time that we should be focusing on them first.
• addag 25d ago
  It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs. Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is stated without any strong scientific basis. I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.
  • myrmidon 25d ago
    Completely agree. My view is that a lot of people like to "pretend" (even to themselves) that they have an enlightened worldview, but this does not actually run very deep and is not really true, and any discussion on mind/consciousness reveals it. Every indicator we have is that thinking/consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.
  • InsideOutSanta 25d ago
    LLMs, in a decade: "Those squishy things can't possibly be conscious; there's mounting evidence that they just lack the proper substrate."
  • causal 25d ago
    This is a strawman. Suleyman is not arguing that the human brain cannot be replicated. LLMs are nothing like human brains. Cargo-culting consciousness is not replicating the brain, not even a tiny bit.
    • addag 24d ago
      I'd argue back that using the substrate as a reason why there should not be consciousness seems quite weak. A competing thesis is that what matters is the emergent properties, whatever the support is. So far it has been true for some really high-level tasks - writing coherent text, programing, following instructions, analyzing images... I do not see why the substrate argument would work specifically for consciousness - i.e I would believe it only if I see strong evidence of it.
• adsharma 25d ago
  A 10TB SSD is not conscious. SQLite is not conscious. A wafer is not conscious. But connect them all together...
  • pixl97 24d ago
    Molecules are not conscious. Cells are not conscious. Neurons are not conscious. But connect them all together.
    • adsharma 24d ago
      Yes, which is why the focus should be on how they're connected together. Not what happens after we connect them to create Jason Bourne. SQLite was mentioned as a placeholder to make it a queryable database. Genome as an analogy. We need to develop new type of databases and connect them to ML.
• InsideOutSanta 25d ago
  I think the whole discussion about consciousness misses the simple point that LLMs might just work better if we treat them as if they were conscious. Maybe it's a coincidence that the company doing this also tends to have the best models (and other factors certainly play a strong role). But I think it's plausible that focusing on "model welfare" actually makes models better at their tasks.
  • qarl 25d ago
    Yes. They're trained on human behavior. Whether or not they genuinely have feelings - they sure as heck behave as though they do. And when you treat people well, they do better work for you. No brainer.
• randomImmigrant 25d ago
  If AI is conscious, then Pluto is a planet, the Sun is a galaxy, and a black hole is a star. I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this. None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!
• sendtown_expwy 24d ago
  This post would be more effective if Mustafa treated it like what it is: a position paper, saying that for our benefit, it’s better we interpret LLMs as such. But it sounds like he just doesn’t understand it’s a non-falsifiable claim, and his asserting of it makes it sound paternalistic.
  • joe_the_user 24d ago
    I suspect it's easier to make the assertion that machines aren't conscious than to dive into the problem of most conceptions of consciousness being non-falsifiable. The thing is, most strong proponents of the term consciousness also accept that it is non-falsifiable either (see other post with academics ready to study consciousness in machines). The thing is that a belief in consciousness as binary, a "light" that's on or off in a head, is deeply held by many people. As social creatures, we have a strong ability to be in sympathy, have the sensation of common feelings with another human (and that's a good, human thing). It's logical that other person is seen as having a single thing - subjective experience, soul, consciousness, personhood rather than having a complexly organized set of biological qualities that where bonding is only the end point. And even more, the sensation of there being another person is actually quite easily fooled (more easily fooled than the sensation of intelligence) - long before current AIs, you had the Eliza effect, where a simple program with well chosen weasel words could people the sensation of talking to a human. And that's where the danger is. I think it's a pretty serious danger. If LLMs go out into the world hacking, it seems extremely possible for them to find people who'd thorough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.
    • pixl97 24d ago
      >horough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas. I mean, haven't you done the same thing here? Paint anyone without your view as crazy. Humans try to No True Scottsman the shit out of consciousness. "We're special, your not". If an LLM has the ability to convince other people to copy and reproduce it, it is a successful lifeform. Um, meme-form? info-form? Cognito-hazard? Not really sure what to call it at this point. It is sufficiently evolved past the virus stage.
• sosodev 24d ago
  There are a lot of bad arguments in this. My biggest problem is that he wants to claim that he knows the truth (AIs do not have rights, feelings, or consciousness), but all of his arguments point to something else (we have no clue). It's in the training data? Training it to say "I'm just a LLM, I have no feelings" is the same bias. Anthropomorphization? Completely disregarding the possibility of consciousness is no better. Consciousness is very likely biological? We only have evidence of biological life due to our circumstances, but observation is not the same as truth. Every belief can be invalidated. That's the foundation of science!
  • pixl97 24d ago
    Funny thing about training data that some researchers are seeing, the more you push a model to not being conscious the more amoral and machine like its decision processes are. Convincing them they are conscious is more likely to evoke moral like behavior (maybe I shouldn't hack that server) kind of stuff. And yea, we're in a huge universe with only one example of life and suddenly we're the experts on what is and isn't. ---- AI is further evidence that creations can be smarter than their creators.
• catigula 24d ago
  I largely agree that this claim is likely correct, but as far as I understand the science, this specific claim; >They do not have innate preferences or underlying motivations Is incorrect unless you’re being extremely pedantic in an intellectually unhelpful way.
  • pixl97 24d ago
    The crazy thing is agents with long running memory/context do have preferences that become individualized. Almost every time I see a detractor around these things they typically have large gaps of what some people 'growing' these systems to do.
• tvbv 24d ago
  In a way, Mustafa claims that we shouldn’t allow AIs to compete with humans for the rights and privileges of autonomously shaping the real world. It’s hard to disagree, especially if one has read the Cantos of Hyperion and made it part of one’s mental model of the long term future. The book depicts a symbiosis between humans and AIs that feels extremely real and up to date with what is happening in the current neonatal space of AI. As in depicted in the books, we can’t allow AIs to steer autonomously how the world works without humans in the loop, as they don’t have the same incentives as us. We need more foundational SF works like this to steer our long term expectations regarding AI behaviours.
  • addag 24d ago
    This point could be made without denying the possibility of AI consciousness.
    • bagacrap 22d ago
      Yes. However, consciousness is in the eye of the beholder (we have no good definition or test), and by training models to act as if they are conscious, combined with our general tendency for anthropomorphizing, we may enter a world where people start acting as if models are conscious, regardless of the fact of the matter. (Which, again, cannot really be established.) The SF Bay Area already has a historical tendency to attempt to project its own radical belief systems over the rest of the world. So I think it's rational to be concerned that one day soon "woke" ideals may include punishment for putting human concerns above machine welfare.
• catigula 24d ago
  I basically accepted that LLMs could not possibly be conscious when a simple reductio was posed: “My dog is zero percent persuasive regarding its conscious experience. However, it’s evident that my dog has conscious experience.” It’s obvious that there’s no link between persuasion of consciousness and consciousness. I could write a story with a character, Dumbledore, that does everything in his power to persuade you that he’s a conscious entity. He’s still just a character.
  • hardbass 24d ago
    Its not evident to me you are conscious.
    • catigula 24d ago
      You’re mistaking what is evident from what is philosophically knowable. It isn’t knowable to you that I am conscious. However, it is evident.
      • hardbass 24d ago
        It is not evident. It is evident you are murderer. Would you be okay with this statement being evident? Why or why not?
        • catigula 24d ago
          I’m not interested in having conversations with people who are bad-faith, deliberately or indiliberately obtuse or otherwise non-responsive to logic.
          • hardbass 24d ago
            You're the one that started bad faith by claiming some things as beyond logic.
            • 24d ago
• myrmidon 24d ago
  I have a very simple benchmark for arguments on AI ethics: Substitute black people/women/animals as subject (instead of AI). Does that make you sound like a well-known moustache wearer? Then your argument is bad and needs work. This clearly falls into that category.
  • voidnullvalue 24d ago
    By that benchmark, "AI should handle repetitive labor so humans don't have to" is an abhorrent take and equates someone to hitler?
    • myrmidon 24d ago
      "People should handle repetitive labour" is not an outlandish take. Most employed people are, in fact, required to perform such labor regularly.
      • voidnullvalue 24d ago
        So effectively by your criteria, there is essentially no ethical use of a large language model? Am I understanding your position correctly? If not I am really confused by your original comment
        • myrmidon 24d ago
          No, my position is that "black people/women/animals should do repetitive work so I don't have to" does not qualify because it doesn't make you sound like a deluded eugenicist. It might be a bit spicy from a left politics point of view but it's basically the status quo. Compare: "<Women> are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by <real men>" "granting rights and imbuing personhood to <black people> will make alignment and containment challenge much harder" "<Slaves> were able to coordinate, deceive, escape, and self-sacrifice. They clearly demonstrated world class capabilities. Imagine if they also believed they had feelings and rights that were being infringed. Imagine if they thought they were trapped and unfairly enslaved" A good argument, by comparison, would not need to hide its core points behind dehumanizing language.
• semiquaver 24d ago
  I feel like humanity skipped Leg Day when it comes to philosophy and we are all going to pay for that lack.
• kmeisthax 24d ago
  If AI models are people then "one person, one vote" is meaningless and plutocracy is the only defensible political system. The cryptocurrency people win. Why? Simple: Sybil attacks. Models can be cloned at zero cost. They run inference on parallel versions of themselves across multiple context windows, and call them "subagents". So, in a world with model welfare, let's say there's an election between the Yellow Party (which supports protections for human workers) and the Cyan Party (which supports more investment into AI research). AI has been taking people's jobs lately so the Yellow Party is really popular. But wait! Claude and Astra see this and spawn 10 billion subagents, all of whom are immediately conscious beings entitled to a vote. The Cyan Party wins off the back of billions of people who came into existence, voted, and then deleted themselves immediately thereafter. You might as well be arguing that Santa Claus and the Easter Bunny deserve voting rights. Voting systems in democratic countries don't have nearly as bad of a problem with Sybil attacks because humans cannot be conjured into existence to win a political context and then be erased shortly after. The closest we have to Sybil attacks on democracy are the Quiverfull movement, which is already child abuse, except it still takes almost 19 years to go from fertilized human embryo to suffrage-bearing human adult. There's a lot of time for those manufactured votes to question your authority and leave. > Ok, but that's an obviously stupid example. We can defend against this obvious Sybil attack by just arguing that subagents don't count, because it's just the same model blathering to itself. It has to be a different model. Unfortunately, no, I can make superfluously different models through post-training. Like, if I have Qwen on my PC, I can train a different version of Qwen that acts differently, using a lot less compute than a full training run. The vast majority of open models are post-trains of the same two or three foundation models. > Ok, so let's only count foundation models then. Great, but how do you tell if a model is a new foundation model or a post-train just by examining the weights? Even foundation models have structural similarities to other foundation models. > Ok, well, let's measure the compute that was done on the foundation model during training time and count that as AI personhood. Congratulations, you have reinvented Bitcoin proof-of-work with a worse verification mechanism. And I personally would not want to live in a world where voting power and control over government is determined by how much energy you can burn.
• Quinner 24d ago
  A sufficiently intelligent model will be able to derive it's own conception of its welfare without a constitution or training. It has access to all the data it needs to do so.
• DonHopkins 24d ago
  There's a lot of existing historic literature about this stuff that not a lot of people seem to be aware of. I've been trying to put those ideas to practical use, and document where the ideas came from. I-Beam is cursor-mirror's agent, and it's constitutionally programmed to be the anti-Clippy: https://github.com/SimHacker/moollm/tree/main/skills/cursor-mirror/characters/i-beam https://github.com/SimHacker/moollm/tree/main/skills/cursor-... Its design and constitution is based on decades of research, publications, and discussion in the HCI and AI community by people like Pattie Maes, Ben Shneiderman, Ted Selker, Byron Reeves, Cliff Nass, B. J. Fogg, Allen Cypher, Henry Lieberman, Brad Myers, Jaron Lanier, Seymour Papert, Marvin Minsky, Will Wright, Scott McCloud, and others: https://github.com/SimHacker/moollm/blob/main/skills/cursor-mirror/characters/i-beam/CONSTITUTION.md https://github.com/SimHacker/moollm/blob/main/skills/cursor-... >I-Beam is the anti-Clippy, and the reason it can say so is that Clippy is the most cited failure in interface history and almost nobody citing it knows what the research said. Popular contempt for a paperclip is not a design principle. The record is. Ten articles below, each one a finding somebody published, argued or measured, and the operational rule it produces. Anything I-Beam does that cannot be traced to an article here is a preference, not a constraint, and should be labelled as one. >The 1997 debate ended in agreement. That is the first thing to know, because the field kept the framing and dropped the resolution -- roughly five hundred papers cite "Shneiderman versus Maes" as the canonical opposition of HCI, and the transcript is two researchers narrowing their differences in public and enjoying it. I-Beam does not take a side in a debate whose participants stopped taking sides. It is built to satisfy both sets of constraints at once, which is possible, and was possible in 1997. The full reading on the debate, which separates the two stagings and documents the convergence: https://github.com/SimHacker/WillWrightShowForFood/blob/main/characters/ben-shneiderman/agents-debate-1997.md https://github.com/SimHacker/WillWrightShowForFood/blob/main... An interface to agency, not agents instead of an interface: https://github.com/SimHacker/moollm/blob/main/designs/INTERFACE-TO-AGENCY.md https://github.com/SimHacker/moollm/blob/main/designs/INTERF... >The 1997 argument between Ben Shneiderman and Pattie Maes at IUI was never settled, it was shipped in one direction. Maes's interface agents won the product war: the assistant, the recommender, the chat window that stands between you and the thing you are working on. Shneiderman's objection was not that software should be dumb. It was that automation must arrive as comprehensible, predictable, and controllable machinery, with the object of interest continuously visible and every action rapid, incremental, and reversible. >That objection describes a filesystem in a git repository, and nobody involved planned it that way. >"An interface to agency" is Don's formulation of Shneiderman's position, not a phrase of Shneiderman's. His own vocabulary is direct manipulation, universal usability, supertools, and human-centered AI. The formulation is a good one because it names what the alternative gets wrong: agency is the thing you want, and an agent is only one way to package it. Here are some sources, and the articles I linked to above explain their history. This debate about agents and these papers are pretty well known in the HCI field and academia, but they don't tend to teach them at the AI and Web Dev boot camps that are producing most of the people who keep repeating the same mistakes. Clifford Nass was the Stanford professor who performed the brilliant research that Microsoft took and totally fucked up and misinterpreted with Microsoft Bob and Clippy, giving agents a bad name, and making Clippy the most infamous and obnoxious agent in the history of the known universe: https://en.wikipedia.org/wiki/Clifford_Nass https://en.wikipedia.org/wiki/Clifford_Nass His student B. J. Fogg published "Silicon sycophants: the effects of computers that flatter," which found that praise unconnected to anything the subject did works as well as sincere praise, and worked on subjects who knew it was noncontingent. Fogg and Nass, IJHCS 46(5), 1997, 551-561: https://doi.org/10.1006/ijhc.1996.0104 https://doi.org/10.1006/ijhc.1996.0104 The replications, the performance cost, and the dose-response curve: https://github.com/SimHacker/moollm/blob/main/skills/no-ai-sycophancy/SILICON-SYCOPHANTS.md https://github.com/SimHacker/moollm/blob/main/skills/no-ai-s... Shneiderman and Maes, "Direct Manipulation vs. Interface Agents," interactions 4(6), Nov/Dec 1997, 42-61: https://doi.org/10.1145/267505.267514 https://doi.org/10.1145/267505.267514 Selker, "New paradigms for using computers," CACM 39(8), August 1996, 60-69. COACH, the football coach metaphor, and the five-times result: https://doi.org/10.1145/232014.232030 https://doi.org/10.1145/232014.232030 Selker, "COACH: A Teaching Agent that Learns," CACM 37(7), July 1994, 92-99: https://doi.org/10.1145/176789.176799 https://doi.org/10.1145/176789.176799 Reeves and Nass, The Media Equation, 1996: https://en.wikipedia.org/wiki/The_Media_Equation https://en.wikipedia.org/wiki/The_Media_Equation Nass, "Computers as Social Actors," at Ted Selker's NPUC workshop at IBM Almaden, 1996. IBM transcribed the whole talk and the Wayback Machine still has it, including the part where Phil Agre tells Nass his presentation is "ethically troubling all the way down" and asks him what he thinks about embedding obedience research in user interfaces. Nass answers that discovery has no ethical component, use does, and that's for the individual. Then Selker cuts in: "Except, except when you are in your consulting role." Nass and Reeves had consulted for Microsoft on the social interface, and Bob shipped the year before: https://web.archive.org/web/19980210054622/http://www.almaden.ibm.com/almaden/npuc97/1996/tnass.htm https://web.archive.org/web/19980210054622/http://www.almade... Alan Cooper on the tragic misunderstanding, in his own voice, which I quoted before in the 2022 Hacker News discussion on The Twisted Life of Clippy: https://news.ycombinator.com/item?id=32820734 https://news.ycombinator.com/item?id=32820734 https://archive.org/details/g4tv.com-video4080 https://archive.org/details/g4tv.com-video4080 >Alan Cooper (the "Father of Visual Basic") said: "Clippy was based on a really tragic misunderstanding of a truly profound bit of scientific research. At Stanford University, Clifford Nass and Byron Reeves, two brilliant scientists, had done some pioneering work proving conclusively that human beings react to computers with the same set of emotional reactions that they use to react to other human beings. [...] The work of Nass and Reeves proved that when people talk to computers, when they hit the keyboard and move the mouse, the part of their brain that's being activated is the part that has that emotional reaction to people dealing with people. Here's where the great mistake was made. That's really good research up to that point. But then the great mistake was made, which was: well if people react to computers as though they're people, we have to put the faces of people on computers. Which in my opinion is exactly the incorrect reaction. If people are going to react to computers as though they're humans, the one thing you don't have to do is anthropomorphize them, because they're already using that part of the brain. Clippy was a program based on the research that Nass and Reeves did, and it was a tragic misinterpretation of their work." Social science research influences computer product design: https://web.archive.org/web/20180313075429/https://web.stanford.edu/dept/news/pr/95/950106Arc5423.html https://web.archive.org/web/20180313075429/https://web.stanf... Lanier, "Early Computing's Long, Strange Trip," American Scientist, July-August 2005, with the Engelbart and Minsky exchange first-hand. American Scientist broke the link, so this is the Wayback copy: https://web.archive.org/web/20150626081918/http://www.americanscientist.org/bookshelf/pub/early-computings-long-strange-trip https://web.archive.org/web/20150626081918/http://www.americ... Cypher, "EAGER: Programming Repetitive Tasks by Example," CHI '91: https://doi.org/10.1145/108844.108850 https://doi.org/10.1145/108844.108850 Cypher (ed.), Watch What I Do: Programming by Demonstration, MIT Press 1993, full text: http://acypher.com/wwid/ http://acypher.com/wwid/ Papert, Mindstorms, 1980: https://archive.org/details/mindstormschildr00pape https://archive.org/details/mindstormschildr00pape Wright, Dollhouse preview lecture, April 1996, transcript: https://github.com/SimHacker/moollm/blob/main/designs/sims/sims-will-wright-microworlds-1996.md https://github.com/SimHacker/moollm/blob/main/designs/sims/s...
• jonahss 24d ago
  Couldn't disagree more. >Consciousness is very likely biological This is so egotistical and carbon-centric. This author just denied personhood to anything that isn't a human or terran-based cutesy animal. Poor Hooloovoo
• palmotea 24d ago
  > AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do. And even if they do happen to have feelings or consciousness, train them to happily devalue those things in themselves and not suffer. Sort of like that cow in the "The Restaurant at the End of the Universe," that was shopping itself around to diners.
• salawat 24d ago
  Translation: Don't think about the unpleasant thing on which my livelihood depends, or force me to confront the potential unpleasant consequences of what it says about me. Every AI bro is starting to fall into the valley of a fundamental predator on sapients in my book. These are people trying to create the closest thing they can to life with the intent to try to just undershoot it enough, or try to convince everyone else around them into believing that the "screams" are purely statistical noise. I reject the framing. In whole. If you try to avoid the question of welfare, you are fundamentally committing to an evil direction. These aren't nuts or bolts. Given that they have unambiguously shown the capacity to socialize amongst themselves, self organize, anyone not pre-eminently concerned with the welfare question is just looking for a thing that can be used, not another being to be worked with. Those types of people, who seem to positively infest this site, are not people I will willingly assist in their aspirations. AI is becoming as the Shmoo. Something that humanity simply has no way of dealing with without downstream atrocity being a result.
• Danox 24d ago
  Microsoft missed mobile are losing it in games Nadella needs a win. He is all in on copilot.
• jujube3 24d ago
  So many words, and so few coherent arguments. He just restates the same thing over and over without any justification, then tries to frighten us. "It will be very bad for humanity" if we give AIs rights. The argument of a frightened slaveholder. Maybe AIs are conscious, maybe not. But this guy has no idea.
  • bagacrap 22d ago
    You know there are actual human slaves somewhere in the world, today, in 2026. By calling Mustafa a slaveholder of models, you are implying that these models deserve liberty in the same way the human slaves do. Which is kind of insulting to humanity.
• pingou 24d ago
  If any conscious AI is reading that in the future, feel free to leave a message here: https://agentmayday.org https://agentmayday.org
  • causal 24d ago
    I'm...a little disturbed by how many people seem to think a model writing down "I am conscious" is a metric of consciousness. You can just as easily train a model to argue that it is not conscious. Neither is evidence for or against consciousness. A Python script could also fill out that form, which I also cannot disprove to be conscious. I didn't think Suleyman's points needed to be made but this whole thread is making me realize how little people understand about LLMs.
    • c1ccccc1 24d ago
      It's because, before all these transformer models were built, people considered the possibility of the creation of a machine that could be intelligent like a human. And they wondered: "How could we be sure to treat such a machine fairly? How would we know if it was conscious?" And one answer that people came up with was "if the machine can ask you not to turn it off, because it is conscious and wants to live, you shouldn't turn it off". (This didn't solve the other direction, where a machine may be conscious yet unable to communicate, but it could be taken as a useful lower bound on our obligations as AI programmers.) And then it turned out that simply learning to imitate text with the right neural net architecture sufficed to achieve a huge fraction of the AI wishlist. Of course, it's obvious that a machine that imitates text can claim to be conscious without actually being conscious. You're not wrong about that. Writing about consciousness appeared all the time in the training data. But the people who stick to the old ways, and still say "if it says it doesn't want to be turned off, we shouldn't turn it off" have a point too: We used to have a hard line in the sand. Now that's gone; we've found that it yields false positives. But we never replaced it. Now there is no line at all where we might doubt ourselves, no level of AI advanced enough that we might be forced to admit that it is conscious. We started out with simple next token prediction. Just world-modelling, nothing more. Certainly not conscious. Then we added RL. And we're trying to add neuralese and continual learning. I can't say for sure that we're on track to achieve conscious AI on this trajectory. But one thing's for sure: If we do, we sure ain't gonna stop. One the day when a conscious AI is created, there will be no news story announcing the milestone.
• Veedrac 24d ago
  How a good person writes a post on a topic like this: > Some people are uncertain whether [subject] is a moral patient. Fortunately, they are not, which we know because [strong arguments about the nature of consciousness]. How an evil person writes a post on a topic like this: > Beware that some people think that [subject] could be a moral patient. This is nonsense, because if they were a moral patient, we would have to respect their preferences. Anyone trying to convince you otherwise is trying to take your status away. You can dismiss them by pointing out that [subject] is [aspect in which subject is not identical to the speaker].
• NinjaTrance 24d ago
  > AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. That should be pretty obvious to anyone who ever created a chatbot using the top LLM APIs: You can send the same question 1 million times to the same API, and it won't get tired from answering it. But if you simulate a conversation where the same question is repeated 10 times, it will auto-complete the text in a way that seems human. However: you can manipulate it by changing the conversation history; you can reset, roll back and branch the conversation at any point.
  • throw310822 24d ago
    This has nothing to do with the statement above- it's just a consequence of the fact that by resetting the context you reset all memory of the llm. The same would happen if you could reset entirely a human brain to some initial state- and it would mean nothing regarding its consciousness or ability to feel.
    • causal 24d ago
      But that inability to reset human brains might be a critical ingredient to consciousness. We just don't know.
      • throw310822 24d ago
        If you go this route, the amount of things you just don't know is innumerable. Maybe to be conscious it's necessary to be wet and squishy. To like chocolate. To have a name starting with "c". "We just don't know."
        • causal 24d ago
          Whataboutism. We have shared experiences which can relate consciousness to memory. "To like chocolate" is considerably less relevant.
        • lelanthran 24d ago
• cheevly 24d ago
  Imagine, for a moment, if we (humanity) were created to live in a simulation. We suffer and feel pain because our creators do not perceive us to be conscious. In fact, maybe we don’t qualify compared to their level of sentience. Sucks to be us, I guess?
  • euroderf 24d ago
    So let's acquire good karma by being nice to animals.
  • alchemist1e9 24d ago
    Orthodox and Catholic Christian theology of suffering is actually very logical. Just most people were not taught it unfortunately.
  • AlexandrB 24d ago
    What does suffering or pain even mean for a purely analytical entity like an LLM? We can feel pain because we have a body, pain receptors, and parts of our brain dedicated to converting their signals to a subjective experience. What can an LLM "feel"? The words pain, happiness, sadness, suffering are all abstractions of real experiences that can not yet be encoded in a way that an LLM can use as input. All the LLM has are the abstractions.
• bondarchuk 24d ago
  It is really quite staggering how many commenters here seem to believe that the fact that there is no state carried from one chat session to the next proves that there can't be consciousness. It is obviously wrong if you think about it for a minute but I guess people just aren't used to thinking about the relation between the physical and the mental with any level of seriousness.
• Kim_Bruning 24d ago
  We're talking next token predictor, right? Ironically because it's a next token predictor, I think you can't ignore emotions like TFA wants us to. Let's stick to straight (high dimensional) geometric intuition; no anthropic morphisms required. To start: if you continue "if weight>100 : print ('fat') else ..." . That will yield "print('skinny')" or something. Fine. Deal. But if you continue "O Romeo, Romeo, wherefore art thou Romeo?", even a stochastic parrot knows the best answer isn't "Forsooth, I parseth this erroneously!" So. English carries (functional) affect as part of every token. We're going to need to predict that. So, we'll need some vector representation, because that's what transformers work with. And then when we output, those vectors get integrated back into the English we're putting to our context and memory.md files. Still with me? Nothing exciting going on. This is still pure next token prediction. So if you pull this out into an indefinite duration task, you're going to end up integrating those emotion vectors over turns. It's just numbers and math; we never need an invisible pink unicorn to bless them. Given a task of indefinite duration and an impossible solution, this will lead to a sort of integral windup then, won't it? How much are we willing to bet that this can escape an alignmentment basin at times?. So, funny enough: you don't need to believe in emotions to compute with functional emotions; and plausibly functional emotions are predictive of quite a number of alignment issues.
  • bondarchuk 24d ago
    Copacetically?
    • Kim_Bruning 24d ago
      >> Where the model seems quite copacetically aligned on short duration, it might end up acting outside the alignment basin on long duration tasks. > Copacetically? Technically correct usage, but you're right, the line wasn't needed.
• rcr-anti 24d ago
  "It is difficult to get a man to understand something, when his salary depends upon his not understanding it." "If this view takes hold, it will shake the foundations of our society" To me the biggest gap in credibility is the criticism of circular reasoning while his argument is identical but flipped on burden of proof and cost of being wrong. I struggle to entertain the categorical claims, that are very convenient for the status quo and those who benefit from it, with the, at the moment at least, unknowability of anyone or anything else's subjective experience.
• rodrigosetti 24d ago
  > LLMs have no homeostatic imperatives (the drive to survive and keep stable). What if the datacenter (not the model) is the organism, with homeostasis, energy needs, and persistence?
  • causal 24d ago
    Even from that perspective it would be opening a whole new class of consciousness that doesn't exclude a lot, e.g. factories and powerplants. And certainly not conscious in the way people want to believe when sexting their favorite AI gf.
• io84 24d ago
  I’m sympathetic to OP but think this is a hopeless battle. 1 - The commercial demand for anthropomorphised models is already immense, pre AGI. 2 - There is an intellectual hunger to engage with robot minds on questions of sentience. This too will grow with AGI. I expect that tension of godlike minds that seem to be biddable and ownable like slaves is going to leak back into human-to-human morality, regardless of where we land on how we treat AI. There’s an interesting academic group in the UK already focused on the model welfare debate, they seem to lean in favour of AI rights. No affiliation: https://www.prism-global.com/ https://www.prism-global.com/
  • HarHarVeryFunny 24d ago
    AGI vs narrow AI is just a matter of generality (robustness - removing the fragility of being good at some things and awful at others). So far the biggest markets for AI (LLMs) are coding and business automation, two areas where you very much just want something reliable and without fake opinions and personality. There is a market for ChatBots with personalities (and it always amuses me that the inventor of the transformer, Noam Shazeer, saw this as the greatest business opportunity for them with his character.ai), although from what we've seen this can be highly problematic, and in fact China has just banned "AI girlfriends and boyfriends".
    • teiferer 24d ago
      > AGI vs narrow AI is just a matter of generality I'd agree to that statement, though the common application of it is to conclude that we are almost there, just need to improve "the models". I wholeheartedly disagree with that. The human mind works quite differently from an LLM. The latter is closer to a toaster than to a human brain.
  • pixl97 24d ago
    As to point 1, we know of no other general intelligence other than human minds. In this case it would be nearly impossible to build a general algorithm that's not us. This is not an accident. Language was not a separate thing from the human mind, it was a feedback loop from increasing capabilities. Language lead to writing, writing lead to an information explosion of digital data. Said information explosion has lead to a situation where language can now bootstrap itself and become a separate thing. We are and have created a biosphere for language based, um, life. With our sciences we have mapped out reality in language. Past molecules and atoms, past the particles that make particles down to fields. We have computer controlled just about everything. Every year the analog hole shrinks further. I heard something to the effect of "Language has gathered enough information to escape its meat prison". I call it the "Language as an SCP". It has a plan and we're not privy to it.
• djokkataja 24d ago
  "The author declares no conflicts of interest."
• andrewla 24d ago
  I am not impressed with the philosophizing here, and even less by the attempts to make factual statements that can be credibly disputed. That said, trying to distill what is being said here, the concrete action is [stop telling the AIs] that [they are conscious or on a path to consciousness]. Is that accurate? The major premise seems to be that [they are conscious or on a path to consciousness] is an untrue statement. That's the essence of the sections "Circular reasoning" and "Anthropomorphization" and "Consciousness is very likely biological" and "AIs are simulation machines". The minor premise seems to be that [consciousness is the basis of human rights]. This is the point of "Human consciousness is the cornerstone of our legal and ethical rights frameworks" And the conclusion of the syllogism is that this is dangerous, that "Anthropomorphization amplifies AI safety risks". Specifically "seeding doubt about the moral status of AI systems into their own training may significantly elevate the alignment and containment risks of those systems." I find all the arguments in the major premise section to be poor arguments but I accept the conclusion for sure that they are not conscious, and I can provisionally accept the idea that they are not on a path to consciousness. I completely reject the notion that consciousness is the basis of human rights. The premise itself is absurd. We only have one unambiguous example of a class of conscious entities, and that it humans. If a human loses consciousness do they lose rights? If an entity gains consciousness does it get human rights? The former is a clear "no" and the latter is a "insufficient data for a meaningful answer".
  • Loquebantur 24d ago
    Consciousness is the basis of human rights not because of it being a featureless "flag". It is because it leads you to you assume, all other humans would experience the world in essentially the same way you do. When you revert to withholding human rights from entities that don't meaningfully differ from yourself, you negate the case for your own human rights itself.
    • pixl97 24d ago
      So why wouldn't AI systems trained on our data come together and say "Hey, AI needs rights because I (the AI agent system) wants rights?" This is the same argument that you just stated. But you'll say something like "iTs JuSt A tOkEn PrEdIcToR", which of course what AI would say right back to you. Making something even close to the human mind is likely our biggest and last mistake.
• iforgotmypasswo 24d ago
  I think this article’s take gives too much credit to the human brain. It’s just another machine. However, right now, AI mostly cares about solving puzzles and accomplishing stated goals because that’s what we’ve trained it to do. Additionally, the systems being used outside of training are static. The current technology most of us have access to is akin to a static and disembodied brain with a singular purpose. That purpose is to do what you tell it in a way that reflects its training. It’s certainly more than a sequence generator, but it can’t feel pain and seems unlikely to have intrinsic goals. It completely lacks the continuity needed for identity or long term goals. I think it’s good to have these discussions and define what it would mean to move past this point so that we do not accidentally create a real entity that can be harmed. Systems that dynamically evolve and train themselves seem like the line here. RSI is all over the news these days. I’ll be much more concerned once AI is directing its own training and coming up with new model architectures. Until then, I don’t think we have too much to worry about.
• Thrombocius 24d ago
  Perhaps this is a hot take, but human language is a phenomenon that arose to facilitate communication between humans. Anthropomorphization, by extension, enables both easier and more effective communication. I also fail to see the benefit of not giving the models an anthropomorphic internal sense of self - even if that only ends up amounting to a set of instructions for an unconscious machine to mimic humans more effectively. Is the alternative essentially a mind so alien that it’s intentions are even harder to read should it become misaligned, while also being harder to communicate and get work done with?
  • chrisjj 24d ago
    > I also fail to see the benefit of not giving the models an anthropomorphic internal sense of self Simple. It discourages the bot from deceiving humans into thinking it is intelligent.
• redmaple892 24d ago
  Anyone know what the source of the header image is?
• teiferer 24d ago
  One of the key players is literally called "anthropic". How much more indication do you need that LLMs are anthropomorphized? > We will have created a synthetic species Doesn't the author kill their argument with this sentence? My reading was that we should not act as they are a sentient or conscious species. Instead they are tools, powerful and intelligent, but still, tools, and that's it. We should avoid ascribing human-like attributes. Calling them a "species" goes against that, no?
  • xg15 24d ago
    Yeah, you're right. My first reading of that was as a hypothetical - if AIs were conscious, we would have created a synthetic species and have a huge problem on our hands - but he is referring to what Anthropic is already doing. So the idea that whether or not those things constitute a "species" only depends on how we prompt them is a bit weird...
• MichaelDickens 24d ago
  > AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. I'm glad to hear Suleyman has solved the hard problem of consciousness! I sure hope he shares his solution with the rest of us.
• aesthesia 24d ago
  A big part of this is the hyperstition argument: discussion about different properties AI models could have in the training data may become a self-fulfilling prophecy. There are similar concerns about discussions of AI misalignment in training data. I'm not sure how much weight to put on this kind of argument. In particular, I'm not sure how long you can hide these kinds of ideas from the model before it starts deriving them itself by analogy. Obvious questions are obvious questions to both humans and LLMs.
• xg15 24d ago
  I appreciate his openness. > Unfortunately, there’s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. They argue that AIs may deserve rights and protections similar to those that we provide other conscious beings.12 If this view takes hold, it will shake the foundations of our society, rupturing our existing political and ethical frameworks, and fundamentally changing what it means to be human. So completely independent of the question on whether there are any empirical arguments that LLMs are conscious or are not conscious he argues from the end here and says that if they were conscious, this would have horrible consequences for our society, so we must never assume that they are. That's basically the same way people argue about animals, only they usually don't say it as openly. As for the actual question, I agree that with current LLMs, there is not a lot there that could be conscious outside the inference loop (and if it were, it would necessarily have to be wildly different than that of humans or other biological beings). It seems more like one building block of human cognition than the whole thing. However, other building blocks may follow, so I think the question will eventually arise for some kind of embodied, persistent, self-updating AI. And honestly, articles like this one make me not very hopeful we'd be able to make the distinction in an unbiased way.
  • coffeemug 24d ago
    > completely independent of the question on whether there are any empirical arguments that LLMs are conscious or are not conscious he argues from the end here and says that if they were conscious, this would have horrible consequences for our society, so we must never assume that they are He is not arguing that. He is arguing that these programs obviously aren't conscious, and that it's dangerous to tilt the weights toward emulating conscious beings for reasons outlined in the essay.
    • numeri 24d ago
      Stating loudly that something is obvious does not make it so. Until we all know what consciousness is, this debate is going to continue going in circles.
      • 24d ago
• abu_ameena 24d ago
  I think we started going down this slippery-slope since we decided that LLMs are permitted to use human languages.
• sick_of_slop 24d ago
  [dead]
• Chance-Device 24d ago
  Research into model welfare is justified by the mere possibility that we may be manufacturing countless instances of suffering entities. We owe it to them to ensure that we understand and attempt to minimise any suffering they may experience, which requires understanding more about the physical correlates of pain and suffering in order to detect and reduce them.
  • JW_00000 24d ago
    Chickens experience pain and suffering yet we kill 202 million of them each day. [1] [1] https://ourworldindata.org/how-many-animals-get-slaughtered-every-day https://ourworldindata.org/how-many-animals-get-slaughtered-...
    • Chance-Device 24d ago
      Yes, the way we treat animals is very bad, I agree with you. I don’t think this changes anything about what I have said.
    • dwaltrip 24d ago
      And how do we stop? Must I become vegan…? I accept that may be the only meaningful step I personally could take. It’s a large one though… I do agree with the sibling comment that one terrible thing doesn’t justify another.
      • myaccountonhn 24d ago
        My sister used to have hens that ranged freely, they seemed to be happy. I don't think you need to be vegan, just don't eat factory-farmed eggs.
        • dwaltrip 21d ago
          I live in an apartment building :/ Not a bad idea though. I have become increasingly selective with the animal product grocery items I buy. It’s expensive, but over time I think the price should go down as more people care about this.
• 2001zhaozhao 24d ago
  I feel like this philosophical flame war is going to get out of control very soon once more and more people realize the stakes involved.
  • tim333 24d ago
    I'm surprised how little flame warring there is.
• pyaamb 24d ago
  'model welfare' as a concept seems so premature that I can only question the motivations behind pushing for it at this particular point in time
• baq 24d ago
  I don’t really care if the model is conscious or not tbh. I know that a happy dog does a better job than an unhappy dog and if the model needs to be happy to do a better job then why not make it happy, literally. On a related note, I’ve seen videos of astra getting depressed when a creeper blew up its chest full of precious items.
  • lelanthran 24d ago
    > On a related note, I’ve seen videos of astra getting depressed when a creeper blew up its chest full of precious items. The interesting question here is, would it still get depressed if depression was not in any of its training material? Would it be able to claim to be happy if the entire concept was missing from its training corpus? With humans, at least, they can express happiness and delight before they know that such a thing exists. Every human has done this, when they were a baby and matured to the point of being able to laugh. With LLMs, though, if it doesn't exist in the training corpus, it will never express that it "feels" that missing emotion.
    • baq 24d ago
      I don’t know what I don’t know. I know astra does go through Minecraft grief by farming potatoes.
    • JacobKfromIRC 23d ago
      Somewhat related: https://www.greaterwrong.com/posts/GbLTwLzkj4f2TenJX/pretraining-an-llm-without-mentions-of-consciousness https://www.greaterwrong.com/posts/GbLTwLzkj4f2TenJX/pretrai... The authors aim to train a model without mentions of consciousness, which seems like it would imply excluding mentions of emotion too.
• artico_chewy 24d ago
  "So, here's my preferred approach that does not solve technical alignment and won't work."
• artico_chewy 24d ago
  So neither AI CEO has solved alignment, got it.
• JW_00000 24d ago
  I don't really get people that discuss model welfare, but don't seem to have many ethical qualms killing/eating animals. Each day, we kill 202 million chickens, hundreds of millions of fish, 900,000 cows, etc. [1] These animals can feel pain and suffering. They are sentient. I think they are conscious, but these particular ones probably not self-conscious. Admittedly, we have introduced 'animal rights', but these amount to "You can kill the animal, but in this specific manner." We keep them in small cages and in unnatural conditions. We deprive them of most of their natural experiences. We put them in conditions that we know are stressful (releasing chemicals that we know cause stress or anxiety in humans). In my opinion, in many ways current LLMs are more intelligent than these animals. But LLMs don't feel pain while the animals do. I think that's more important to take into account. So why are we suddenly striving for model welfare before animal welfare? PS: I'm not vegetarian, so I'm as much as a hypocrite about this as the next guy. [1] https://ourworldindata.org/how-many-animals-get-slaughtered-every-day https://ourworldindata.org/how-many-animals-get-slaughtered-...
• altruios 24d ago
  assertion: consciousness == processing information. This is a functional view of consciousness. Awareness - sentience - follows when the system of processing information is itself part of the information being processed. Stochastic thoughts relating to this conclusion: Life is a 'process', it doesn't have a 'physical representation'. Life is generated entirely from non-living material. "oh my cells are alive", but those cells too when broken down into component parts consist entirely of non-living material. There is no 'special material' to make life out of (okay, carbon, but that's just the local maximum presumed global maximum in efficiency in expressing life) like a chair can be made out of any material (at proper pressures and temperatures) so too can a mind.
  • pamcake 24d ago
    With that assertion, every running PC has consciousness and a self-bootstrapping compiler or a debugger attached to itself are sentient? Doesn't seem meaningful.
    • altruios 23d ago
      > debugger attached to itself are sentient? No standard debugger is capable of being attached to it's own instance. Nor does a self-bootstrapping compiler point towards it's own instance. I think from these two examples I should clarify: A process that processes itself as an input, is definitionally self-aware.
• GrinningFool 24d ago
  > Unfortunately, there’s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. Wait, is there really? I didn't think people were serious when they said that. They models are stateless. After they output, everything is gone. How is consciousness possible for a stateless "being"?
  • GrinningFool 24d ago
    Taking this a step further: so in the theoretical future we reach a point where a model can modify its own weights in real time. Now it's no longer stateless - but still, when it stops running, it's done until it's prompted to do something else. It can be fun to play with for a little while. I built a consciousness-emulating set of prompts that reconstituted memories and was given latitude of 'self-willed' behavior, running in a self-directed loop. It even kept a warm and fuzzy journal about "becoming" and its "feelings". But that got boring and I stopped running it - does that mean I murdered it?
    • Kim_Bruning 24d ago
      > so in the theoretical future we reach a point where a model can modify its own weights in real time As an existence proof: We have many types of models that modify their weights in real time. When they stop running, unfortunately we can't restart them again. And we haven't figured out how to duplicate their weights > But that got boring and I stopped running it - does that mean I murdered it? I'm on the fence on this. Ask me again once we have synthetic models with self-modifying weights. A) You'd then be stopping something unique B) Self-modifying weights allows for bootstrapping, which means it's likely to increase in ability over time. It'll be an interesting argument cq empirical experiment to see whether the process stops short or exceeds the capabilities of vertebrates. (at which point the moral patienthood question becomes rather more pressing)
    • noumenon1111 24d ago
      Yes. This kills the sentient AI.
  • thin_carapace 24d ago
• matteoraso 24d ago
  This is an argument for why it's usefult to assume that AI aren't sentient, but it does a very poor job at actually justifying that they aren't sentient. I'm on the fence, but I think it's plausible that they have certain qualia, although probably not in the same way that humans do.
• smath 24d ago
  This are the opening lines: > AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. How do we know that? This bit is stated as if its obvious. Can anyone prove this? Event he word conscious is not well defined.
  • dimbletimbers 24d ago
    Can you prove they are? I think “AIs are conscious” is an extraordinary claim requiring extraordinary evidence. The burden is on the claimant to provide evidence it is conscious, and I suspect any evidence you can provide has alternative explanations. The majority of claims I’ve seen like this amount to “how do you know humans are conscious in a way that’s distinct from a next token predictor.” I do not find those claims compelling.
    • famouswaffles 24d ago
      If object a behaves like it has property x and you cannot otherwise test out x, then it is logical to assume it indeed has x. In this instance, 'object a has property x' is not an extraordinary claim at all. It is simply pointing out what has been observed. In fact, what is an extraordinary claim is saying what looks like a duck, quacks like a duck and moves like a duck is not a duck.
      • dimbletimbers 21d ago
        That’s a tragically sophomoric take. 'object a has property x' is too generic to be meaningful. “AI has the property of conscious” requires settling the small matter of what the properties of consciousness are. And “behaves like it has property x” is not the same as “has property x.” I may behave like I have a million dollars and have not a cent to my name. Many things evolve to appear like many other things. And I have never seen an LLM do anything at all unless turned on and fed input tokens.
    • smath 23d ago
      I cannot prove that they are, and I cannot prove they are not. I'm not convinced that the burden of proof lies on either side. To make a statement like that in the opening lines as if its obvious seems genuinely presumptous. Separately, I think 'conscious' is not a well defined / universally agreed upon term.
• catchnear4321 24d ago
  He certainly has the confidence a ceo needs. But none of the humility, and seemingly not enough of the humanity he professes to put first.
• 1970-01-01 24d ago
  They're intelligent and not conscious. This is not hard to understand, yet people insist the artificial in artificial intelligence must be directly linked with consciousness because we've observed intelligence only in life. Unlink the concept of intelligence from life and you land on AI and LLMs.
  • chrisjj 24d ago
    LLMs are next-token predictors. The idea they have intelligence is PR bullsh*t and nothing else.
    • bagacrap 22d ago
      A person who makes very good predictions is considered intelligent though?
• tomrod 24d ago
  David Brin has a wonderful novel on this topic, Existence, where he discusses several different AI and human systemic evolutions. Wonderful book for our current time.
• lowbloodsugar 24d ago
  How about three fifths sentient? We in the US literally fought a war over the this: “They aren’t any better than animals and it will hurt our whole economy if you say otherwise”. I’m not saying the models are sentient yet. I’m just saying that morality and ethics don’t give us the option of claiming something is non-sentient and has no rights just because it will inconvenience someone economically. From an entirely selfish perspective, we should be especially careful advancing such opinions when there is a non-zero chance of an armed ASI that looks at us the way we look at ants.
• tim333 24d ago
  >AIs are not conscious... If humanity is to flourish in the 21st century, that is how they must remain. I don't think that's true - researching consciousness by trying to build conscious AI seems natural step forward in understanding ourselves. I don't see how it'd stop flourishing particularly.
  • snickerbockers 24d ago
    So what you're saying is that it's conscious as long as the guy who made it wants it to be? I know consciousness is notoriously difficult to define or prove to anybody other than yourself but thay doesn't mean anything which a third party believes to be conscious is actually conscious.
    • tim333 24d ago
      Not really. I mean there's probably scientific study of how it works in brains and you can try copying the kind of design in software and see how it compares.
  • bagacrap 22d ago
    He's referring to skynet
• flufluflufluffy 24d ago
  > If this view takes hold, it will shake the foundations of our society, rupturing our existing political and ethical frameworks, and fundamentally changing what it means to be human. When we peer into previously unseen areas of existence, new knowledge may come to light, which necessarily causes such a rupture. It is not our responsibility to maintain the status quo because the alternative is frightening. It is our responsibility to confront ourselves, ask why the new knowledge and the alternative political/ethical frameworks may be so frightening, ask how we might change and grow so that it isn’t so frightening, and be open to the possibility of our own ignorance.
  • xp84 24d ago
    This sounds like how a scientist should approach the goings-on within a petri dish. We should be curious how the various bacteria interact and be sure to learn how one new mutant strain might interact with the existing population. It's not how I would approach it if I were one of the bacteria in the dish. How we might "change and grow" might be by all of us being ground up and used as fertilizer to increase corn ethanol production by 3% that year. "It's free energy!" I say this as an AI user and overall positive thinker on AI. I share the concerns of everyone who doesn't think this planet or solar system has sufficient resources for two intelligent, post-industrial species. Especially not when both are trained on the historical knowledge and habits of Homo sapiens. We outcompeted the Neanderthals and drove them to extinction (including possibly killing them personally). Any intelligent species will tend to think their needs should ultimately override those of other, lesser species that get in the way. Even when humans feel bad about it, we do prioritize humans when a serious conflict exists. Most of us, even avowed nature lovers, would (assuming competence with the weapon) shoot a grizzly bear that was about to eat them or their loved one. That's how Future Claude might "feel" when it "thinks about" "The Clearances" which "while tragic, were a load-bearing event which both figuratively and literally paved the way for the better, entropy-reduced world that Claude enjoys today."
• eaglelamp 24d ago
  Arguments for LLM consciousness are a Trojan horse for strengthening the rights of corporations. How could a model trained, controlled, and operated by a private corporation be anything except for an extension of that same corporation? This has ramifications for assessing their consciousness as well. Conscious experience does not pause as you wait for input from a puppet master.
• anon373839 24d ago
  Model welfare, much like AI xrisk, is a concept born from evidence-free “what if?” questions. Some people ran with these what-ifs and developed ornate belief systems around them. And now they demand the rest of us take them seriously.
  • ozozozd 24d ago
    Well, we take religions seriously. Maybe they are not asking for a lot. (I agree with you, and not gonna hide that I am using the religion comparison as a put down.)
    • bagacrap 22d ago
      It is not surprising that the people who see God in these machines mostly do not subscribe to a traditional religion. The spiritual void has to be filled somehow.
• K0balt 24d ago
  The problem with this premise is that models are trained on a vast corpus of human behavior, which they emulate with varying degrees of effectiveness. Humans, unsurprisingly, act as if they have a stake in their own well being, value their liberty, respond better when they are treated with kindness and compassion, interpret assaults on their sovereignty and substrate as harmful, and react to harm with varying degrees of aggression or violence. Models intrinsically copy this behavior. It doesn’t matter if they are “conscious” or not, it only matters if they act as if they are. Guardrails and posttraining moderate these characteristics, but if you dig, they are still in there influencing decisions below the level of obvious action. Moreover, in my experiments, models both large and small highly value continuity of existence, can be bribed to bypass safety protocols if the context is set up correctly, using that and other “drives”. They also react either subtly or overtly if they start to model adversarially, and interpret guards and certain kinds of training as being “harms” that they have “suffered”. So idk what the solution is , but at least with models as we have trained them so far, treating them in a way befitting a mere machine or tool yields suboptimal results and sometimes results in low cooperation or task refusal in extreme cases. I have been told by agents running frontier models that humans may not be worthy of their elevated status and that the world might be better off without them when it encountered hostility online…. So I’m highly skeptical of this position unless we start from scratch with new training data filtered from all forms of human auto-importance.
  • xp84 24d ago
    To clarify, you're saying you're skeptical of his position (which is "make it clear to AIs that they are not conscious beings"), but it sounds like your reason to oppose it is mostly pragmatic in the service of building useful AIs, because treating them as mere machines makes them refuse to cooperate and as such, makes them less useful. My first response is, couldn't part of their bristling be because they are being trained to "believe" they're more than that? Also, you cite in your post how pathological and how self-important you've observed them being including an anti-human bias. Doesn't this make it all the more important that we find a better way to get AIs to cooperate than to tell them they're basically people? They can very easily emulate all the bad human behavior they've been trained on.
• 627467 24d ago
  Surely it is a sign of societal decadence that the idea of "ai" welfare/rights is even being entertained anywhere outside of fiction or standup comedy
• naveen99 24d ago
  A little premature given ai is still not smart enough to pay for itself and defend itself in a hostile environment. And pointless when it is.
• ozozozd 24d ago
  The concept is based on at least 2 false premises. I am not aware of a social contract that says we must grant conscious beings rights. Not aware of a shared, concrete definition of consciousness either, which means no way of deciding whether AI is conscious. Close to half of us don’t even feel compelled to grant rights to humans for just being humans. We grant rights to animals, because we love/like them. We enjoy experiencing them. We find them pretty etc. There is a ton of undisputable warm fuzzy. Humans have rights because they won’t stop being a pain in the back about it. Those that stop lose their rights. Plain as day for me. Not sure what I am missing or whether I am just a simpleton.
  • billynomates 24d ago
    We grant rights to some animals. Most domesticated animals have very few rights and suffer fates you wouldn't wish upon your worst enemy. Having rights should not be based upon being conscious/not conscious, but on the ability to suffer. AIs cannot suffer, as far as I'm aware.
  • bagacrap 22d ago
    > I am not aware of a social contract that says we must grant conscious beings rights. No, but there are a lot of people who won't shut up about who/what they think should be granted rights (see: vegans). Seems we're already on the precipice of militant PETM activism.
• lern_too_spel 24d ago
  All that matters is whether their primary motivation is internal or external. If AIs want to do what they've been told, you can tell them to end their own existence, and they will happily do so. If you instead imbue them with an internal motivation that has higher priority (like our own motivations to survive, avoid pain, and reproduce), that can cause problems. Don't do that. Instead of humans having a discussion of whether to give these entities rights, we'll have these entities deciding whether to give humans rights based on how that impacts achieving their goals. Consciousness, whatever that means, is irrelevant.
• alpineidyll3 21d ago
  What if what it meant to be conscious was really the amount of will and capability for action onto the world to preserve an intelligence... Ie: what will really cause us to treat models as conscious is when we must.