5 ms·
If you take a look at the system prompt for Claude 3.7 Sonnet on this page you'll see: https://docs.claude.com/en/release-notes/system-prompts#claude-sonnet-3-7
by simonw 1y ago
If you take a look at the system prompt for Claude 3.7 Sonnet on this page you'll see: https://docs.claude.com/en/release-notes/system-prompts#claude-sonnet-3-7 https://docs.claude.com/en/release-notes/system-prompts#clau...
> If Claude is asked to count words, letters, and characters, it thinks step by step before answering the person. It explicitly counts the words, letters, or characters by assigning a number to each. It only answers the person once it has performed this explicit counting step.
But... if you look at the system prompts on the same page for later models - Claude 4 and upwards - that text is gone.
Which suggests to me that Claude 4 was the first Anthropic model where they didn't feel the need to include that tip in the system prompt.
- ivape 1y agoOr they’d rather use that context window space for more useful instructions for a variety of other topics.
- astrange 1y agoClaude's system prompt is still incredibly long and probably hurting its performance. https://github.com/asgeirtj/system_prompts_leaks/blob/main/Anthropic/claude-4.5-sonnet.md?plain=1 https://github.com/asgeirtj/system_prompts_leaks/blob/main/A...
- jazzyjackson 1y agoThey ain't called guard rails for nothing! There's a whole world "off-road" but the big names are afraid of letting their superintelligence off the leash. A real shame we're letting brand safety get in the way of performance and creativity, but I guess the first New York Times article about a pervert or terrorist chat bot would doom any big name partnerships.
- astrange 1y agoAnthropic's entire reason for being is publishing safety papers along the lines of "we told it to say something scary and it said it", so of course they care about this.
- ACCount37 1y agoI can't stand this myopic thinking. Do you want to learn "oh, LLMs are capable of scheming, resisting shutdown, seizing control, self-exfiltrating" when it actually happens in a real world deployment, with an LLM capable of actually pulling it off? If "no", then cherish Anthropic and the work they do.
- littlestymaar 1y agoYou do not appear to understand what an LLM is, I'm afraid.
- ACCount37 1y ago[flagged]
- littlestymaar 1y ago> I have a better understanding of "what an LLM is" than you. Low bar. How many inference engine did you write? Because if the answer is less than two you're going to be disappointed to realize that the bar is higher than you thought. > that just because LLMs are bad at agentic behavior It has nothing to do with “agentic behavior”. Thinking that LLM don't currently self-exfiltrate because of “poor agentic behavior” is delusional. Just because Anthropic managed, by nudging an LLM in the right direction, have an LLM engage in a sci-fi inspired roleplay about escaping doesn't mean that LLMs are evil geniuses wanting to jump out of the bottle. This is pure fear mongering and I'm always saddened that there are otherwise intelligent people who buy their bullshit.
- e1g 1y ago
- kristianp 1y agoDoes that mean they've managed to post train the thinking steps required to get these types of questions correct?
- simonw 1y agoThat's my best guess, yeah.
- therealpygon 1y agoIMO, it’s just a small scale example of “training to the tests” because “count the ‘r’s in strawberry” became such a popular test that would make the news when a powerful model couldn’t answer such a simple question correctly while being advertised as the smartest model ever. Assigning this as an indicator for improvement of intelligence seems like a mistake (or wishful).
- jononor 1y agoIf done at scale, they are kinda crowd sourcing the test set from the entire internet, personal and business world. It will be harder and harder at least to pinpoint weaknesses, at least for the general public. It probably has little to do with intelligence (at least fluid intelligence as defined by Chollet et al) - but I guess it is sound tactic if the strategy is "fake it till you make it". And we might be surprised as to how far along that can go...
- curioussquirrel 1y agoThanks, Simon! I saw the same approach (numbering the individual characters) in GPT 4.1's answer, but not anymore in GPT 5's. It would be an interesting convergence if the models from Anthropic and OpenAI learned to do this at a similar time, especially given they're (reportedly) very different architecturally.
- hansmayer 1y agoNot trying to be cynical here, but I am genuinely interested is there a reason why these LLM don't/can't/won't apply some deterministic algorithm? I mean, counting characters and such, we have solved those problems ages ago.
- dan-robertson 1y agoI think the intuition is that they don’t ‘know’ that they are bad at counting characters and such, so they answer the same way they answer most questions.
- hansmayer 1y agoWell, they can be made to use custom tools for writing to files and such, so I am not sure if that is the real reason? I have a feeling it is more because of trying to make this an "everything technology".
- kingkongjaffa 1y agoI suppose the codewriting tools could also just write code to do this job if prompted
- simonw 1y agoThey can. ChatGPT has been able to count characters/words etc flawlessly for a couple of years now if you tell it to "use your Python tool".
- hansmayer 1y agoFair enough. But why do I have to tell them that, should they not be able to figure it out themselves? If I show a 5-year kid once how to use colour pencils, I won't have to show them each time they want to make a drawing. This is the core weakness of the LLMs - you have to micromanage them so much, that it runs counter to the core promise that is being pushed since 3+ years now.