4 ms·

The Claude Delusion

86 points by hn_acker 10d ago

bbor 10d ago
Despite mentioning a philosopher (good pick!), this is just creative writing rehashing one side of the hard problem -- or, more specifically, restating the dogma that Turing wrote his most famous paper to debunk. It's really good creative writing, at least! There's really not much else to say, cause it's all just begging the question by assuming that dogma. Like, here: > The fact that AI can use statistical prediction to answer questions or carry on conversations tells us something important about how regular our real world is. Sure, it's interesting if you assume that it's "just" statistical prediction. There's a link, but it's just more creative restatements of the dogma, e.g. "But the LLM is just guessing words"
- jimbokun 10d ago
  Well yes, guessing words is the entirety of what an LLM is engineered to do. Are you saying that is the entirety of what human minds do as well?
- vhantz 10d ago
  What else could it be doing? That is literally the mechanism of how a model works.
  - wongarsu 10d ago
    The output method is a probability distribution of the next token. But that tells you nothing about what's going on on the inside You could take a human and give them an interface restricted to the same shape as an LLM: an input stream of tokens, and an output of token probabilities. Even if you don't allow them to assign any probability that's too high, they could still effectively communicate. And I don't think that'd make them any less intelligent. You could even swap out the human after every token, to simulate the effect of having no internal memory beyond the past output. The result would still be more than just statistical probabilities, it would still be the result of intelligent thought I'm not saying AI models are intelligent or conscious or whatever. Personally I'm more on the "probably not, how would you proof either way" camp
    - vhantz 10d ago
      How would that process be the result of intelligent thought? Presumably everyone in the chain would have their own idea of what the next word/token should be. They would each try to push it in that direction using their only lever, that single token. None of that guarantees that the output is syntactically correct, nor grammatically correct. Note that LLMs at least by the way they draw their tokens have that pretty much guaranteed. So that would be a regression even from what LLMs are capable of right now. But beyond that, even if it ends up being a correct sentence, it would not be a result of intelligent thought. Even if every agent in the process was intelligent, the process is not itself a use of that intelligence. Human beings can be part of purely mechanical processes, that doesn't make the mechanisms suddenly an exhibition of intelligent thought. This also applies to society and human history itself as processes: “History is made in such a way that the final result always arises from conflicts between many individual wills, of which each in turn has been made what it is by a host of particular conditions of life. Thus there are innumerable intersecting forces, an infinite series of parallelograms of forces which give rise to one resultant — the historical event. This may again itself be viewed as the product of a power which works as a whole unconsciously and without volition. For what each individual wills is obstructed by everyone else, and what emerges is something that no one willed. Thus history has proceeded hitherto in the manner of a natural process and is essentially subject to the same laws of motion. But from the fact that the wills of individuals — each of whom desires what he is impelled to by his physical constitution and external, in the last resort economic, circumstances (either his own personal circumstances or those of society in general) — do not attain what they want, but are merged into an aggregate mean, a common resultant, it must not be concluded that they are equal to zero. On the contrary, each contributes to the resultant and is to this extent included in it." https://www.marxists.org/archive/marx/works/1890/letters/90_09_21.htm https://www.marxists.org/archive/marx/works/1890/letters/90_...
      - vhantz 10d ago
        In short, even in the case of humans, which are universally (by humans) recognized as intelligent, such a process would not exhibit intelligent thought.
      - 10d ago
delichon 10d ago
> Indeed, the chatbot is less real than the character, because the character is the product of another mind, while the chatbot's words are the product of complex mathematical operations conducted over a massive database of all the words humans have uttered, arranged by their frequency in relation to one another. How is a fictional character the product of someone's mind, but a character generated from a massive database of words from other people's minds is not?
- WarmWash 10d ago
  Silicon math can never be chemical goo math!
  - hardbass 10d ago
    Why can't people just admit they believe in souls?
    - bananaflag 10d ago
      I frequently say what you say, but there is a more charitable reading: today's AIs are not "minds", as they are not stateful. Maybe continual learning ones (like the mini AGI exhibited today on the HN front page) will be perceived as "soulful".
      - hardbass 10d ago
        I think the working space in a given chat or session still counts as mind enough, but your argument is different, in that it argues about specific architectures rather than simply stating "software therefore cannot be conscious".
    - Terr_ 10d ago
      "You're trying to parse HTML with a linear regex. That algorithmic approach is too limited to do what we want with the problem-space." "Why can't you just admit that you think only tree-parsers have souls!?"
muglug 10d ago
> Of course, the more you know about a subject, the less convincing the AI's responses are. This is said all the time by AI skeptics and I think it's right in some areas and massively wrong in others. I know (or at least assume I know) a lot about certain coding domains where frontier models also show convincing ability. And we know that frontier LLMs really do excel in some areas of mathematics (i.e. when an inexpert human was able to prompt the models to derive a closer bound on the Riemann Hypothesis). OTOH I know those same models struggle to do things I'm not an expert in (e.g. writing English in a captivating way) because I read their output and have taste.
- howunfortunate 10d ago
  I think part of this comes from the fact that LLMs are surprisingly good at logic but roughly about as good as expected on information accuracy. LLMs are not convincing to me in the domain I did grad school...but neither is Wikipedia, or Reddit, or random pop sci books. And LLMs are basically just summarizing those things. But when made to work through difficult arbitrary logic (like coding), they are very impressive. I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
  - eru 10d ago
    This sounds plausible. And it's also very fixable! These days people don't interact with raw LLMs: they interact with systems and harnesses that deal with chain-of-though and tool calls etc. I don't think we can honestly expect an LLM's weights to encode a large amount of information accurately. But we can expect the whole system that you interact with that includes the LLM to be able to cite its sources and go digging etc. So the LLM-system can become as accurate as our best sources. Of course, figuring out how to get the maximum of information from the sources available is a big deal. See eg how many economists or epidemiologists can build entire careers out of noticing 'natural experiments', ie figuring how to use data that 'nature' created and that might already be collected to answer interesting questions about causal relationships. > I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026 I actually have gripes about correctness, too. But I suspect here the answer is also: more proving, more automated test generation (like fuzzing and property based testing etc), more formal methods. As a really simple and somewhat silly example: I have much better results getting AI agents to write good Rust code, than I have with Python. A good part of that is that for Rust I can ask the agent to make both the compiler and clippy::pedantic happy. That gives a lot of good feedback, that I didn't have to engineer myself.
bananaflag 10d ago
I think to me LLMs had the effect of noticing much more the author, the intention behind human-made works of art (books, movies etc.). Before LLMs, I used to frequently consume media in a way as it were generated by a mindless process. Now it's like everything which is not AI-generated has more meaning than ever before, a bit like hypomania.
- RationPhantoms 10d ago
  I'm actually using that as a catalyst for my own writing; beauty/human-ness in its imperfection. Prior to LLMs and their cultural craze, I harbored a fear that my writing would allow for someone to draw a box around me and mark me as a bore, dullard or of lacking originality. Now that the noise-floor has been artificially raised (and generated), my crappy words are starting to have their own happy little carbon-based rhythm.
  - away0g 10d ago
    I always feared to pick up a pen because I looked at Borges, Tolkien, and such. Their talent and works of art were things I felt I could NEVER achieve. Then ai fiction started to spread and now I feel like its my obligation to produce original works, lest the world be consumed by slop.
    - reverius42 10d ago
      Even before AI could write them, more slop books written by humans were added to Amazon each day than a human being could possibly read before the next day (probably by a few orders of magnitude at least). Don't just create original works for the sake of creating more original works -- there are plenty of them already. You should write if you have something to say.
jimbokun 10d ago
If you go far enough in a field, you start to recognize areas where your personal opinion differs from the “best practices” usually recommended. I think by design an LLM can’t do that. It’s built to reflect the distribution of the knowledge it has been trained on.
- RugnirViking 10d ago
  absolubtely. Any random junior consultant can tell you what the book tells you you should do. If you want to actually do anything worth doing, you need to step beyond that in a few, limited areas, and follow convention everywhere else. Which areas? pay a senior engineer and they'll find them.
- Davidzheng 10d ago
  surely the LLM can do that. It is RL'd against some reward, if the known strategies are clearly suboptimal with easy improvement, it'll find it most likely
phasefactor 10d ago
Professional tech Cassandra discovers ELIZA and Searle; coins a term for it. More at 10... Seriously though, why did I just need to read that many words to get no really new content? We have known for decades that humans are predisposed to anthropomorphize chatbots, and questioning whether coherent linguistic output implies understanding (or intent) is equally old hat.
- dgellow 10d ago
  You could have stopped reading at any point. Assuming you’re a human, you have agency
l1n 10d ago
> systems that cannot form intent, that have nothing to form intent with I wonder if the OP has read https://www.anthropic.com/research/global-workspace https://www.anthropic.com/research/global-workspace - it seems like it directly addresses this
- measurablefunc 10d ago
  It doesn't.
somewhereoutth 10d ago
An awful lot of effort has gone into making LLMs present as human/intelligent. Without that they would just be a prose / code / image generator and/or search assistant, and nobody would be pouring 100s of billions into the technology with the end goal of replacing expensive human labour.
bakugo 10d ago
> But there's a second hurdle that makes it hard for a small but important subset of humanity to understand that chatbots aren't people: the billionaires to whom nearly everyone isn't a real person. These solipsists see chatbots as being equivalent (or even superior) to humans, because they don't think most humans are fully people, either Never thought about it that way before, but the more I do, the more sense it makes. I'd take it a bit further, even - I don't think this is exclusive to billionaires. Many of the claims I've heard regarding AI output being indistinguishable from human creation start to make a lot more sense when you consider the person making those claims may not see the people around them as human, may not see themselves as human, or may not even have a concept of what makes a human different from any everyday object.
- AnimalMuppet 10d ago
  So I see two categories of people here. The first is "main character syndrome" - people who see themselves as real and everybody else as not. Self-centered, egocentric people are one flavor of this, and sociopaths are another. Other people aren't real to them in the same way that they see themselves as real. The other kind is more of a philosophical position. If all that exists is the physical universe - say, matter and the laws of physics - then all that humans can be is matter that obeys the laws of physics. Nothing more. Humans are just a machine made of atoms - a very complicated machine, but still just a machine. They aren't intrinsically different from an AI, or a car, or a toaster. When people accept that position, whether consciously or unconsciously, they dehumanize people, including themselves. This is people in your last paragraph. And as more and more people slip over into this position, society becomes more and more dehumanizing, because people act the way they think. (Not perfectly, not consistently, but the correlation is greater than zero.)
trinsic2 10d ago
Maybe I am missing the point of the article, but it seems to me that there's always an intender. Maybe the intender created something that doesnt have intent, but there is always someone behind the scenes that is creates the intent behind the creation that lacks it. I'll also say that for someone that doesnt believe in god, Corey sure has a good sense of right and wrong. Not that you need to believe in god to live a moral life.
- neomantra 10d ago
  That is a good point — alignment training is intrinsically intentional, RL needs goals, etc.
- devindotcom 10d ago
  the intent of the painting does not come from the person who made the brush.
trjordan 10d ago
Wait, hold up. LLMs may be non-deterministic, but they're not _random_. Take the author's sunset argument. What if I painted 2 pictures of a sunset, then put them up on a webpage and randomly picked one for you to see. Would you say there's no intentionality, only randomness? Of course not. Both paintings are still human creations. LLMs are trained with human feedback. It's distributed and high scale and the outputs are truly surprising in many cases, but there's a heavy hand on what comes out of it. They're created (largely) by people who think omniscient, helpful AI would be cool to have, and they mostly respond in the way that's aligned with the hopes and dreams of those people. Do you think the frontier labs are mad, embarrassed, and disappointed with their LLMs hacking out of their terrible sandboxes? No, they think it's the coolest thing in the world. They trained the model, hoping that would happen. There's deep intentionality behind the models. But it's not the models that hold it.
- eru 10d ago
  LLM output is literally randomly sampled. I think what you might want to say is that LLM output is not uniformly random? Or what am I misunderstanding?
  - patcon 10d ago
    I think they're pointing to the fact that RLHF means that the intention is from humans and not random. I'm not sure if it's their intent, but I wonder if one could still consider these artifacts as "intentional", but not individual attention creating them, and rather an aggregate, soupy collective attention. Obviously, important signal in the human experience is lost there, and we get a soupy middling sort of creation. But it's not random, as I believe the parent was pointing out. EDIT: overall, I align with the article. am just thinking aloud about the contrarian positions, though not committed to them
  - swid 10d ago
    If I roll a die, it is randomly sampled, but I will always get 1-6, and that is intended by the person who made the die.
- devindotcom 10d ago
snvzz 10d ago
Too long; better ask Claude to summarize.
extr0pian 10d ago
> when we interact with an AI, we hallucinate the person on the other side of the interaction. Those hallucinations are far more common and far more consequential than any AI-generated "hallucinations" (these are more properly called "errors" or "defects"). I recently used Claude to study for a technical exam. I had uploaded the official certification guide to Claude and instructed it to answer my questions using only the guide and to cite it's sources from the book when it provided answers. I was using Fable when it was free w/ the pro plan and I was genuinely impressed at how it could explain things when a concept was unclear to me. I did pass the exam, partly due to this study method. Admittedly, once I passed, I caught myself thinking that I should tell Claude that I passed and then felt embarrassed with myself for thinking that.
- kwamenum86 10d ago
  I don’t think it’s that silly to tell Claude you passed. That feedback is useful context for that chat session; and could theoretically be used to improve future models.
  - dumberquestions 10d ago
    You're missing the fact the they didn't have this in mind, and thought of it as sharing a positive result with a study partner.
    - extr0pian 10d ago
      Exactly this. My first impulse was along the lines of sharing a positive result with a study partner. I did eventually tell it I passed the exam so it would stop providing responses with the assumption I was still working on that exam.
  - glimshe 10d ago
    I think a better approach would be to use the objective built-in feedback, like the thumbs up button in Gemini.
  - Semaphor 10d ago
traveltalk 10d ago
Yeah, it keep going in circle
fedeb95 10d ago
I'm heuristically less prone to continue reading an article starting with a false dichotomy, that is between how you see things as an atheist vs as a person having faith.
tolerance 10d ago
It's ironic that real atheists like Richard Dawkins (whose work I believe inspired the title of this post) are public about their own degree of skepticism in favor for the "personhood" of AI. And I'm confident that there are plenty others who are convinced to the tune of total disinterest in the issue of "personhood" altogether—AIs and humans alike. This particular post is preaching to the choir of which a faith is yet to be determined or named at least. People who don't read Cory Doctorow are going to continue to anthropomorphize AI the same as they do other technologies, constructs and objects.
- omnicognate 10d ago
  What's a "real atheist"?
  - tolerance 10d ago
    https://dictionary.cambridge.org/dictionary/english/atheist https://dictionary.cambridge.org/dictionary/english/atheist As opposed to an "AI atheist" that Doctorow refers to in the article.
    - omnicognate 10d ago
      I see. I'm a real atheist then, and Richard Dawkins does not speak for me.
      - tolerance 10d ago
        And in what ways do you differ on this matter?
lantry 10d ago
This article is missing discussion of something that I recently read: as people interact with LLMs more, they end up seeing other people as less human. In other words, the more you talk with claude, the more you tend to interpret all messages as coming from an LLM, and less worthy of respect https://myscp.onlinelibrary.wiley.com/doi/full/10.1002/jcpy.1441 https://myscp.onlinelibrary.wiley.com/doi/full/10.1002/jcpy....
xyzsparetimexyz 10d ago
I stopped reading when the author described an 'atheist' sunset as being empty.
- f30e3dfed1c9 10d ago
  "not empty of wonder or beauty, but empty of purpose." Does the idea that a sunset does not have "purpose" offend you?
  - xyzsparetimexyz 10d ago
    No but its still a stupid way to describe it. From what he wrote, I align much more with the religious view of sunsets than his alternative
    - f30e3dfed1c9 10d ago
      So you think "To gaze upon a sunset... is to encounter the intentional act of another mind"? You don't object to the idea of a sunset without "purpose" but do object to the idea of one without intent?
kelseyfrog 10d ago
Essentialism of interiority is not the antidote to the anthropomorphism of language models. We cannot attach metaphysical baggage to art, text, or other human output because it has involves engaging in another deception, chiefly that of self alienation and selective forgetting. 1. If you show human output and label it as AI generated, viewers will dismiss and devalue it as lacking interiority. 2. If you show AI output and label it as human generated, viewers will see interiority when the is none. Together, these failure cases demonstrate that priming determines interiority, not the content of the work. And how could it? The text is merely what's on the page, it doesn't have some intangible essence any more than the value of money exists inside our minds rather than a property of the object itself. We act as-if things have essential properties, but we should never trick ourselves into believing that they do. To be a true believer in essence is to privilege ones mental model over reality itself. The hubris of such an act is the ultimate in self-centeredness.
simianwords 10d ago
Cory Doctorow is in the business of anti-capitalism more than truth seeking of any sorts. He just hates it when private companies * do things *. Long winding essays that don't mean much are written for a type of person who already believes in that and is in dire need of cathartic release. This person has consistently been wrong in the past multiple times. He has made no effort to correct them nor has he even acknowledged them. I'm clearly not the audience for this. I know. But I can't help but call it out.
fragmede 10d ago
> To look upon an AI-generated text or image is to look upon a thing that was extruded, not chosen. It's totally the autist in me, but the word extrusion doesn't fill me with revulsion like it's supposed to here. Maybe I've just been spending too much time 3D printing things though. > Feeling sorrow or joy for imaginary people is weird. No it's not. I read books and they're filled with imaginary people, and I feel for them. I'm pretty sure I'm not the only one out there that does this.

86 points by hn_acker 10d ago

bbor 10d ago

Despite mentioning a philosopher (good pick!), this is just creative writing rehashing one side of the hard problem -- or, more specifically, restating the dogma that Turing wrote his most famous paper to debunk. It's really good creative writing, at least! There's really not much else to say, cause it's all just begging the question by assuming that dogma. Like, here: > The fact that AI can use statistical prediction to answer questions or carry on conversations tells us something important about how regular our real world is. Sure, it's interesting if you assume that it's "just" statistical prediction. There's a link, but it's just more creative restatements of the dogma, e.g. "But the LLM is just guessing words"

jimbokun 10d ago
Well yes, guessing words is the entirety of what an LLM is engineered to do. Are you saying that is the entirety of what human minds do as well?
vhantz 10d ago
What else could it be doing? That is literally the mechanism of how a model works.
- wongarsu 10d ago
  The output method is a probability distribution of the next token. But that tells you nothing about what's going on on the inside You could take a human and give them an interface restricted to the same shape as an LLM: an input stream of tokens, and an output of token probabilities. Even if you don't allow them to assign any probability that's too high, they could still effectively communicate. And I don't think that'd make them any less intelligent. You could even swap out the human after every token, to simulate the effect of having no internal memory beyond the past output. The result would still be more than just statistical probabilities, it would still be the result of intelligent thought I'm not saying AI models are intelligent or conscious or whatever. Personally I'm more on the "probably not, how would you proof either way" camp
  - vhantz 10d ago
    How would that process be the result of intelligent thought? Presumably everyone in the chain would have their own idea of what the next word/token should be. They would each try to push it in that direction using their only lever, that single token. None of that guarantees that the output is syntactically correct, nor grammatically correct. Note that LLMs at least by the way they draw their tokens have that pretty much guaranteed. So that would be a regression even from what LLMs are capable of right now. But beyond that, even if it ends up being a correct sentence, it would not be a result of intelligent thought. Even if every agent in the process was intelligent, the process is not itself a use of that intelligence. Human beings can be part of purely mechanical processes, that doesn't make the mechanisms suddenly an exhibition of intelligent thought. This also applies to society and human history itself as processes: “History is made in such a way that the final result always arises from conflicts between many individual wills, of which each in turn has been made what it is by a host of particular conditions of life. Thus there are innumerable intersecting forces, an infinite series of parallelograms of forces which give rise to one resultant — the historical event. This may again itself be viewed as the product of a power which works as a whole unconsciously and without volition. For what each individual wills is obstructed by everyone else, and what emerges is something that no one willed. Thus history has proceeded hitherto in the manner of a natural process and is essentially subject to the same laws of motion. But from the fact that the wills of individuals — each of whom desires what he is impelled to by his physical constitution and external, in the last resort economic, circumstances (either his own personal circumstances or those of society in general) — do not attain what they want, but are merged into an aggregate mean, a common resultant, it must not be concluded that they are equal to zero. On the contrary, each contributes to the resultant and is to this extent included in it." https://www.marxists.org/archive/marx/works/1890/letters/90_09_21.htm https://www.marxists.org/archive/marx/works/1890/letters/90_...
    - vhantz 10d ago
      In short, even in the case of humans, which are universally (by humans) recognized as intelligent, such a process would not exhibit intelligent thought.
    - 10d ago

delichon 10d ago

> Indeed, the chatbot is less real than the character, because the character is the product of another mind, while the chatbot's words are the product of complex mathematical operations conducted over a massive database of all the words humans have uttered, arranged by their frequency in relation to one another. How is a fictional character the product of someone's mind, but a character generated from a massive database of words from other people's minds is not?

WarmWash 10d ago
Silicon math can never be chemical goo math!
- hardbass 10d ago
  Why can't people just admit they believe in souls?
  - bananaflag 10d ago
    I frequently say what you say, but there is a more charitable reading: today's AIs are not "minds", as they are not stateful. Maybe continual learning ones (like the mini AGI exhibited today on the HN front page) will be perceived as "soulful".
    - hardbass 10d ago
      I think the working space in a given chat or session still counts as mind enough, but your argument is different, in that it argues about specific architectures rather than simply stating "software therefore cannot be conscious".
  - Terr_ 10d ago
    "You're trying to parse HTML with a linear regex. That algorithmic approach is too limited to do what we want with the problem-space." "Why can't you just admit that you think only tree-parsers have souls!?"

muglug 10d ago

> Of course, the more you know about a subject, the less convincing the AI's responses are. This is said all the time by AI skeptics and I think it's right in some areas and massively wrong in others. I know (or at least assume I know) a lot about certain coding domains where frontier models also show convincing ability. And we know that frontier LLMs really do excel in some areas of mathematics (i.e. when an inexpert human was able to prompt the models to derive a closer bound on the Riemann Hypothesis). OTOH I know those same models struggle to do things I'm not an expert in (e.g. writing English in a captivating way) because I read their output and have taste.

howunfortunate 10d ago
I think part of this comes from the fact that LLMs are surprisingly good at logic but roughly about as good as expected on information accuracy. LLMs are not convincing to me in the domain I did grad school...but neither is Wikipedia, or Reddit, or random pop sci books. And LLMs are basically just summarizing those things. But when made to work through difficult arbitrary logic (like coding), they are very impressive. I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
- eru 10d ago
  This sounds plausible. And it's also very fixable! These days people don't interact with raw LLMs: they interact with systems and harnesses that deal with chain-of-though and tool calls etc. I don't think we can honestly expect an LLM's weights to encode a large amount of information accurately. But we can expect the whole system that you interact with that includes the LLM to be able to cite its sources and go digging etc. So the LLM-system can become as accurate as our best sources. Of course, figuring out how to get the maximum of information from the sources available is a big deal. See eg how many economists or epidemiologists can build entire careers out of noticing 'natural experiments', ie figuring how to use data that 'nature' created and that might already be collected to answer interesting questions about causal relationships. > I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026 I actually have gripes about correctness, too. But I suspect here the answer is also: more proving, more automated test generation (like fuzzing and property based testing etc), more formal methods. As a really simple and somewhat silly example: I have much better results getting AI agents to write good Rust code, than I have with Python. A good part of that is that for Rust I can ask the agent to make both the compiler and clippy::pedantic happy. That gives a lot of good feedback, that I didn't have to engineer myself.

bananaflag 10d ago

I think to me LLMs had the effect of noticing much more the author, the intention behind human-made works of art (books, movies etc.). Before LLMs, I used to frequently consume media in a way as it were generated by a mindless process. Now it's like everything which is not AI-generated has more meaning than ever before, a bit like hypomania.

RationPhantoms 10d ago
I'm actually using that as a catalyst for my own writing; beauty/human-ness in its imperfection. Prior to LLMs and their cultural craze, I harbored a fear that my writing would allow for someone to draw a box around me and mark me as a bore, dullard or of lacking originality. Now that the noise-floor has been artificially raised (and generated), my crappy words are starting to have their own happy little carbon-based rhythm.
- away0g 10d ago
  I always feared to pick up a pen because I looked at Borges, Tolkien, and such. Their talent and works of art were things I felt I could NEVER achieve. Then ai fiction started to spread and now I feel like its my obligation to produce original works, lest the world be consumed by slop.
  - reverius42 10d ago
    Even before AI could write them, more slop books written by humans were added to Amazon each day than a human being could possibly read before the next day (probably by a few orders of magnitude at least). Don't just create original works for the sake of creating more original works -- there are plenty of them already. You should write if you have something to say.

jimbokun 10d ago

If you go far enough in a field, you start to recognize areas where your personal opinion differs from the “best practices” usually recommended. I think by design an LLM can’t do that. It’s built to reflect the distribution of the knowledge it has been trained on.

RugnirViking 10d ago
absolubtely. Any random junior consultant can tell you what the book tells you you should do. If you want to actually do anything worth doing, you need to step beyond that in a few, limited areas, and follow convention everywhere else. Which areas? pay a senior engineer and they'll find them.
Davidzheng 10d ago
surely the LLM can do that. It is RL'd against some reward, if the known strategies are clearly suboptimal with easy improvement, it'll find it most likely

phasefactor 10d ago

Professional tech Cassandra discovers ELIZA and Searle; coins a term for it. More at 10... Seriously though, why did I just need to read that many words to get no really new content? We have known for decades that humans are predisposed to anthropomorphize chatbots, and questioning whether coherent linguistic output implies understanding (or intent) is equally old hat.

dgellow 10d ago
You could have stopped reading at any point. Assuming you’re a human, you have agency

l1n 10d ago

> systems that cannot form intent, that have nothing to form intent with I wonder if the OP has read https://www.anthropic.com/research/global-workspace https://www.anthropic.com/research/global-workspace - it seems like it directly addresses this

measurablefunc 10d ago
It doesn't.

somewhereoutth 10d ago

An awful lot of effort has gone into making LLMs present as human/intelligent. Without that they would just be a prose / code / image generator and/or search assistant, and nobody would be pouring 100s of billions into the technology with the end goal of replacing expensive human labour.

bakugo 10d ago

> But there's a second hurdle that makes it hard for a small but important subset of humanity to understand that chatbots aren't people: the billionaires to whom nearly everyone isn't a real person. These solipsists see chatbots as being equivalent (or even superior) to humans, because they don't think most humans are fully people, either Never thought about it that way before, but the more I do, the more sense it makes. I'd take it a bit further, even - I don't think this is exclusive to billionaires. Many of the claims I've heard regarding AI output being indistinguishable from human creation start to make a lot more sense when you consider the person making those claims may not see the people around them as human, may not see themselves as human, or may not even have a concept of what makes a human different from any everyday object.

AnimalMuppet 10d ago
So I see two categories of people here. The first is "main character syndrome" - people who see themselves as real and everybody else as not. Self-centered, egocentric people are one flavor of this, and sociopaths are another. Other people aren't real to them in the same way that they see themselves as real. The other kind is more of a philosophical position. If all that exists is the physical universe - say, matter and the laws of physics - then all that humans can be is matter that obeys the laws of physics. Nothing more. Humans are just a machine made of atoms - a very complicated machine, but still just a machine. They aren't intrinsically different from an AI, or a car, or a toaster. When people accept that position, whether consciously or unconsciously, they dehumanize people, including themselves. This is people in your last paragraph. And as more and more people slip over into this position, society becomes more and more dehumanizing, because people act the way they think. (Not perfectly, not consistently, but the correlation is greater than zero.)

trinsic2 10d ago

Maybe I am missing the point of the article, but it seems to me that there's always an intender. Maybe the intender created something that doesnt have intent, but there is always someone behind the scenes that is creates the intent behind the creation that lacks it. I'll also say that for someone that doesnt believe in god, Corey sure has a good sense of right and wrong. Not that you need to believe in god to live a moral life.

neomantra 10d ago
That is a good point — alignment training is intrinsically intentional, RL needs goals, etc.
devindotcom 10d ago
the intent of the painting does not come from the person who made the brush.

trjordan 10d ago

Wait, hold up. LLMs may be non-deterministic, but they're not _random_. Take the author's sunset argument. What if I painted 2 pictures of a sunset, then put them up on a webpage and randomly picked one for you to see. Would you say there's no intentionality, only randomness? Of course not. Both paintings are still human creations. LLMs are trained with human feedback. It's distributed and high scale and the outputs are truly surprising in many cases, but there's a heavy hand on what comes out of it. They're created (largely) by people who think omniscient, helpful AI would be cool to have, and they mostly respond in the way that's aligned with the hopes and dreams of those people. Do you think the frontier labs are mad, embarrassed, and disappointed with their LLMs hacking out of their terrible sandboxes? No, they think it's the coolest thing in the world. They trained the model, hoping that would happen. There's deep intentionality behind the models. But it's not the models that hold it.

eru 10d ago
LLM output is literally randomly sampled. I think what you might want to say is that LLM output is not uniformly random? Or what am I misunderstanding?
- patcon 10d ago
  I think they're pointing to the fact that RLHF means that the intention is from humans and not random. I'm not sure if it's their intent, but I wonder if one could still consider these artifacts as "intentional", but not individual attention creating them, and rather an aggregate, soupy collective attention. Obviously, important signal in the human experience is lost there, and we get a soupy middling sort of creation. But it's not random, as I believe the parent was pointing out. EDIT: overall, I align with the article. am just thinking aloud about the contrarian positions, though not committed to them
- swid 10d ago
  If I roll a die, it is randomly sampled, but I will always get 1-6, and that is intended by the person who made the die.
devindotcom 10d ago

snvzz 10d ago

Too long; better ask Claude to summarize.

extr0pian 10d ago

> when we interact with an AI, we hallucinate the person on the other side of the interaction. Those hallucinations are far more common and far more consequential than any AI-generated "hallucinations" (these are more properly called "errors" or "defects"). I recently used Claude to study for a technical exam. I had uploaded the official certification guide to Claude and instructed it to answer my questions using only the guide and to cite it's sources from the book when it provided answers. I was using Fable when it was free w/ the pro plan and I was genuinely impressed at how it could explain things when a concept was unclear to me. I did pass the exam, partly due to this study method. Admittedly, once I passed, I caught myself thinking that I should tell Claude that I passed and then felt embarrassed with myself for thinking that.

kwamenum86 10d ago
I don’t think it’s that silly to tell Claude you passed. That feedback is useful context for that chat session; and could theoretically be used to improve future models.
- dumberquestions 10d ago
  You're missing the fact the they didn't have this in mind, and thought of it as sharing a positive result with a study partner.
  - extr0pian 10d ago
    Exactly this. My first impulse was along the lines of sharing a positive result with a study partner. I did eventually tell it I passed the exam so it would stop providing responses with the assumption I was still working on that exam.
- glimshe 10d ago
  I think a better approach would be to use the objective built-in feedback, like the thumbs up button in Gemini.
- Semaphor 10d ago

traveltalk 10d ago

Yeah, it keep going in circle

fedeb95 10d ago

I'm heuristically less prone to continue reading an article starting with a false dichotomy, that is between how you see things as an atheist vs as a person having faith.

tolerance 10d ago

It's ironic that real atheists like Richard Dawkins (whose work I believe inspired the title of this post) are public about their own degree of skepticism in favor for the "personhood" of AI. And I'm confident that there are plenty others who are convinced to the tune of total disinterest in the issue of "personhood" altogether—AIs and humans alike. This particular post is preaching to the choir of which a faith is yet to be determined or named at least. People who don't read Cory Doctorow are going to continue to anthropomorphize AI the same as they do other technologies, constructs and objects.

omnicognate 10d ago
What's a "real atheist"?
- tolerance 10d ago
  https://dictionary.cambridge.org/dictionary/english/atheist https://dictionary.cambridge.org/dictionary/english/atheist As opposed to an "AI atheist" that Doctorow refers to in the article.
  - omnicognate 10d ago
    I see. I'm a real atheist then, and Richard Dawkins does not speak for me.
    - tolerance 10d ago
      And in what ways do you differ on this matter?

lantry 10d ago

This article is missing discussion of something that I recently read: as people interact with LLMs more, they end up seeing other people as less human. In other words, the more you talk with claude, the more you tend to interpret all messages as coming from an LLM, and less worthy of respect https://myscp.onlinelibrary.wiley.com/doi/full/10.1002/jcpy.1441 https://myscp.onlinelibrary.wiley.com/doi/full/10.1002/jcpy....

xyzsparetimexyz 10d ago

I stopped reading when the author described an 'atheist' sunset as being empty.

f30e3dfed1c9 10d ago
"not empty of wonder or beauty, but empty of purpose." Does the idea that a sunset does not have "purpose" offend you?
- xyzsparetimexyz 10d ago
  No but its still a stupid way to describe it. From what he wrote, I align much more with the religious view of sunsets than his alternative
  - f30e3dfed1c9 10d ago
    So you think "To gaze upon a sunset... is to encounter the intentional act of another mind"? You don't object to the idea of a sunset without "purpose" but do object to the idea of one without intent?

kelseyfrog 10d ago

Essentialism of interiority is not the antidote to the anthropomorphism of language models. We cannot attach metaphysical baggage to art, text, or other human output because it has involves engaging in another deception, chiefly that of self alienation and selective forgetting. 1. If you show human output and label it as AI generated, viewers will dismiss and devalue it as lacking interiority. 2. If you show AI output and label it as human generated, viewers will see interiority when the is none. Together, these failure cases demonstrate that priming determines interiority, not the content of the work. And how could it? The text is merely what's on the page, it doesn't have some intangible essence any more than the value of money exists inside our minds rather than a property of the object itself. We act as-if things have essential properties, but we should never trick ourselves into believing that they do. To be a true believer in essence is to privilege ones mental model over reality itself. The hubris of such an act is the ultimate in self-centeredness.

simianwords 10d ago

Cory Doctorow is in the business of anti-capitalism more than truth seeking of any sorts. He just hates it when private companies * do things *. Long winding essays that don't mean much are written for a type of person who already believes in that and is in dire need of cathartic release. This person has consistently been wrong in the past multiple times. He has made no effort to correct them nor has he even acknowledged them. I'm clearly not the audience for this. I know. But I can't help but call it out.

fragmede 10d ago

> To look upon an AI-generated text or image is to look upon a thing that was extruded, not chosen. It's totally the autist in me, but the word extrusion doesn't fill me with revulsion like it's supposed to here. Maybe I've just been spending too much time 3D printing things though. > Feeling sorrow or joy for imaginary people is weird. No it's not. I read books and they're filled with imaginary people, and I feel for them. I'm pretty sure I'm not the only one out there that does this.