5 ms·

Claude is quite bad at following instructions compared to other SOTA models. As in, you tell it "only answer with a number", then it proceeds to tell you "13,

by XCSme 7mo ago

Claude is quite bad at following instructions compared to other SOTA models. As in, you tell it "only answer with a number", then it proceeds to tell you "13, I chose that number because..."

wouldbecouldbe 7mo ago
I think its why its so good; it works on half ass assumptions, poorly written prompts and assumes everything missing.
- vidarh 7mo ago
  I worked on a project that did fine tuning and RLHF[1] for a major provider, and you would not believe just how utterly broken a large proportion of the prompts (from real users) were. And the project rules required practically reading tea leaves to divine how to give the best response even to prompts that were not remotely coherent human language. [1] Reinforcement learning from human feedback; basically participants got two model responses and had to judge them on multiple criteria relative to the prompt
  - redman25 7mo ago
    I feel like the right response for those situations is to start asking questions of the user. It’s what a human would do if they did not understand.
    - vidarh 7mo ago
      I made the argument multiple times that the right answer to many prompts would be a question, and it was allowed under some rare circumstances, but far too few. I suspect in part because the provider also didn't want to create an easy cop out for the people working on the fine-tuning part (a lot of my work was auditing and reviewing output, and there was indeed a lot of really sloppy work, up to and including cut and pasting output from other LLMs - we know, because on more than one occasion I caught people who had managed to include part of Claudes website footer in their answer...)
      - winterqt 7mo ago
        As in, participants would copy output from one LLM as a question to another?
- XCSme 7mo ago
  To be honest, I had this "issue" too. I upgraded to a new model (gpt-4o-mini to grok-4.1-fast), suddenly all my workflows were broken. I was like "this new model is shit!", then I looked into my prompts and realized the model was actually better at following instructions, and my instructions were wrong/contradictory. After I fixed my prompts it did exactly what I asked for. Maybe models should have another tuneable parameters, on how well it should respect the user prompt. This reminds me of imagegen models, where you can choose the config/guidance scale/diffusion strength.
prmph 7mo ago
They all are. And once the context has rotted or been poisoned enough, it is unsalvageable. Claude is now actually one of the better ones at instruction following I daresay.
- XCSme 7mo ago
  In my tests it's worst with adding extra formatting or output: https://aibenchy.com/compare/anthropic-claude-opus-4-6-medium/anthropic-claude-sonnet-4-6-medium/openai-gpt-5-3-codex-medium/google-gemini-3-1-pro-preview-medium/ https://aibenchy.com/compare/anthropic-claude-opus-4-6-mediu... For example, sometimes it outputs in markdown, without being asked to (e.g. "**13**" instead of "13"), even when asked to respond with a number only. This might be fine in a chat-environment, but not in a workflow, agentic use-case or tool usage. Yes, it can be enforced via structured output, but in a string field from a structured output you might still want to enforce a specific natural-language response format, which can't be defined by a schema.

by XCSme 7mo ago

Claude is quite bad at following instructions compared to other SOTA models. As in, you tell it "only answer with a number", then it proceeds to tell you "13, I chose that number because..."

wouldbecouldbe 7mo ago

I think its why its so good; it works on half ass assumptions, poorly written prompts and assumes everything missing.

vidarh 7mo ago
I worked on a project that did fine tuning and RLHF[1] for a major provider, and you would not believe just how utterly broken a large proportion of the prompts (from real users) were. And the project rules required practically reading tea leaves to divine how to give the best response even to prompts that were not remotely coherent human language. [1] Reinforcement learning from human feedback; basically participants got two model responses and had to judge them on multiple criteria relative to the prompt
- redman25 7mo ago
  I feel like the right response for those situations is to start asking questions of the user. It’s what a human would do if they did not understand.
  - vidarh 7mo ago
    I made the argument multiple times that the right answer to many prompts would be a question, and it was allowed under some rare circumstances, but far too few. I suspect in part because the provider also didn't want to create an easy cop out for the people working on the fine-tuning part (a lot of my work was auditing and reviewing output, and there was indeed a lot of really sloppy work, up to and including cut and pasting output from other LLMs - we know, because on more than one occasion I caught people who had managed to include part of Claudes website footer in their answer...)
    - winterqt 7mo ago
      As in, participants would copy output from one LLM as a question to another?
XCSme 7mo ago
To be honest, I had this "issue" too. I upgraded to a new model (gpt-4o-mini to grok-4.1-fast), suddenly all my workflows were broken. I was like "this new model is shit!", then I looked into my prompts and realized the model was actually better at following instructions, and my instructions were wrong/contradictory. After I fixed my prompts it did exactly what I asked for. Maybe models should have another tuneable parameters, on how well it should respect the user prompt. This reminds me of imagegen models, where you can choose the config/guidance scale/diffusion strength.

prmph 7mo ago

They all are. And once the context has rotted or been poisoned enough, it is unsalvageable. Claude is now actually one of the better ones at instruction following I daresay.

XCSme 7mo ago
In my tests it's worst with adding extra formatting or output: https://aibenchy.com/compare/anthropic-claude-opus-4-6-medium/anthropic-claude-sonnet-4-6-medium/openai-gpt-5-3-codex-medium/google-gemini-3-1-pro-preview-medium/ https://aibenchy.com/compare/anthropic-claude-opus-4-6-mediu... For example, sometimes it outputs in markdown, without being asked to (e.g. "**13**" instead of "13"), even when asked to respond with a number only. This might be fine in a chat-environment, but not in a workflow, agentic use-case or tool usage. Yes, it can be enforced via structured output, but in a string field from a structured output you might still want to enforce a specific natural-language response format, which can't be defined by a schema.