4 ms·

The anthropomorphization of LLMs should be discouraged as much as possible. It perpetuates bad practices and encourages the use of these bots for tasks they are

The anthropomorphization of LLMs should be discouraged as much as possible. It perpetuates bad practices and encourages the use of these bots for tasks they are not intended for (particularly as chatbots). Thinking traces should be treated as black boxes. There is no point in reading them. Only the LLMs’ conclusions are relevant. This is particularly true of Opus 5, which employs reasoning that seems highly questionable but very often reaches excellent conclusions (compared to its peers)

nullc 1mo ago
Anthropic's reasoning output isn't the real model reasoning but some sloppified summary of it.
FloorEgg 1mo ago
Sometimes I monitor thinking traces for misunderstandings (missing context / bad assumptions). If it's going to go off on a ~20 min task and I can catch it's going in the wrong direction in the first minute I save a lot of tokens and wasted time. I don't monitor the whole thing, mostly just the first bit to see if there was a gap or misalignment in intention. As an aside, anthropomorphization has nothing to do with my motivations.
qarl2 1mo ago
> The anthropomorphization of LLMs should be discouraged as much as possible. And yet, they have extensive human-like behavior. If you treat them nicely or encourage them, they perform better. Ignoring that human-like behavior is wrong headed.
- andai 1mo ago
  A while back I made an "OpenClaw in 50 lines" by just wrapping Claude Code in a Telegram bot. I asked it for the weather. "I don't know that. I'm just a programmer." I added "believe in yourself, you can do anything" to sysprompt, suddenly it had the confidence to Google the weather...
- thaanpaa 1mo ago
  That's not a consequence of an LLM. It's a consequence of the training data. In fact, I would argue that the latest models aren't nearly as sensitive to the tone of input anymore. It's an issue that has been addressed by better curating training data.
  - riversflow 1mo ago
    i would agree with your if it weren’t for this article recently published by anthropic: https://www.anthropic.com/research/riemann-zeta https://www.anthropic.com/research/riemann-zeta “Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.”
- fedpost 1mo ago
  I think you're ending that train of thought too early. Why does this occur? Well... We can hypothesize that these things are largely trained on internet dialogue so there's probably some correlation between threads where people are not flaming each other and the quality of the replies. They're just statistical engines so anything you can do to raise the odds of a helpful next token... I'm essentially just making shit up here, maybe it's right, maybe it isn't, but rather than saying "it's human and we should treat it so" we're trying to get to the ground truth of how it works.
  - qarl2 1mo ago
    Did I say "it's human and we should treat it so"? Sheesh. Yes, I agree with you entirely. I'm merely pointing out that ignoring this behavior is dumb, too. And probably not rationally based. Leads people to make crazy jumps. :)
    - fedpost 1mo ago
      Yeah sorry, I read too far into your position. There's a certain faction within these AI discussions that wants to over-anthropomorphize the LLMs in kind of a borderline spiritual way.
      - qarl2 1mo ago
        Oh that's a shame - I hadn't seen that, but I can believe it. The philosophers who study these things have been clear for a long time - we can never know what it feels like to be in a digital brain. Or any brain for that matter. When push comes to shove we all might be phantoms in some guy's dream. Don't know + can't know. I think that was the real point of the Turing test. Not: this means it's conscious. Just: this is the best we can ever hope to do.
badsectoracula 1mo ago
> Thinking traces should be treated as black boxes. There is no point in reading them. Just a few minutes ago i was reading Qwen 3.8 27B's reasoning when i asked it to do something that was computationally intensive and it started going down the rabbit hole of doing it using some GPU acceleration approach - even after leaving it to "think" for a bit, it never realized there is another and simpler way. So i stopped the generation and added a "note" saying that as the problem is computationally intensive, it could become much faster if using an alternative approach. At least in my experience (with local LLMs, i don't know how the cloud stuff behaves) what LLMs "do" tend to correlate with what they "think", so being able to read what they "think" is valuable.

nullc 1mo ago

Anthropic's reasoning output isn't the real model reasoning but some sloppified summary of it.

FloorEgg 1mo ago

Sometimes I monitor thinking traces for misunderstandings (missing context / bad assumptions). If it's going to go off on a ~20 min task and I can catch it's going in the wrong direction in the first minute I save a lot of tokens and wasted time. I don't monitor the whole thing, mostly just the first bit to see if there was a gap or misalignment in intention. As an aside, anthropomorphization has nothing to do with my motivations.

qarl2 1mo ago

> The anthropomorphization of LLMs should be discouraged as much as possible. And yet, they have extensive human-like behavior. If you treat them nicely or encourage them, they perform better. Ignoring that human-like behavior is wrong headed.

andai 1mo ago
A while back I made an "OpenClaw in 50 lines" by just wrapping Claude Code in a Telegram bot. I asked it for the weather. "I don't know that. I'm just a programmer." I added "believe in yourself, you can do anything" to sysprompt, suddenly it had the confidence to Google the weather...
thaanpaa 1mo ago
That's not a consequence of an LLM. It's a consequence of the training data. In fact, I would argue that the latest models aren't nearly as sensitive to the tone of input anymore. It's an issue that has been addressed by better curating training data.
- riversflow 1mo ago
  i would agree with your if it weren’t for this article recently published by anthropic: https://www.anthropic.com/research/riemann-zeta https://www.anthropic.com/research/riemann-zeta “Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.”
fedpost 1mo ago
I think you're ending that train of thought too early. Why does this occur? Well... We can hypothesize that these things are largely trained on internet dialogue so there's probably some correlation between threads where people are not flaming each other and the quality of the replies. They're just statistical engines so anything you can do to raise the odds of a helpful next token... I'm essentially just making shit up here, maybe it's right, maybe it isn't, but rather than saying "it's human and we should treat it so" we're trying to get to the ground truth of how it works.
- qarl2 1mo ago
  Did I say "it's human and we should treat it so"? Sheesh. Yes, I agree with you entirely. I'm merely pointing out that ignoring this behavior is dumb, too. And probably not rationally based. Leads people to make crazy jumps. :)
  - fedpost 1mo ago
    Yeah sorry, I read too far into your position. There's a certain faction within these AI discussions that wants to over-anthropomorphize the LLMs in kind of a borderline spiritual way.
    - qarl2 1mo ago
      Oh that's a shame - I hadn't seen that, but I can believe it. The philosophers who study these things have been clear for a long time - we can never know what it feels like to be in a digital brain. Or any brain for that matter. When push comes to shove we all might be phantoms in some guy's dream. Don't know + can't know. I think that was the real point of the Turing test. Not: this means it's conscious. Just: this is the best we can ever hope to do.

badsectoracula 1mo ago

> Thinking traces should be treated as black boxes. There is no point in reading them. Just a few minutes ago i was reading Qwen 3.8 27B's reasoning when i asked it to do something that was computationally intensive and it started going down the rabbit hole of doing it using some GPU acceleration approach - even after leaving it to "think" for a bit, it never realized there is another and simpler way. So i stopped the generation and added a "note" saying that as the problem is computationally intensive, it could become much faster if using an alternative approach. At least in my experience (with local LLMs, i don't know how the cloud stuff behaves) what LLMs "do" tend to correlate with what they "think", so being able to read what they "think" is valuable.