3 ms·

> Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing t

> Use shorter prompts: In internal evaluations, replacing long, explicit system prompts with minimal prompts improved scores by roughly 10–15%, while reducing total tokens by 41–66% and cost by 33–67%. When has this ever not been the case? I don't think this is a GPT 5.6 specialty!

adam_arthur 3mo ago
Information density of the prompt is the most important factor in my experience. And interestingly, LLMs seem particularly bad at writing prompts for other LLMs for this reason (you can guide them to be more dense, just speaking by default). Conciseness is usually a byproduct of information density though.
- astrange 3mo ago
  > LLMs seem particularly bad at writing prompts for other LLMs for this reason Claude is terrible at this! Probably for the same reason that its writing style in prose is so annoying and full of claudisms.
  - irthomasthomas 3mo ago
    Claude use to be leader, too. Their metaprompt was great at the time with opus 3
- docjay 3mo ago
  Lexical-priming->semantic-space-constraint;specialized-lexis+=sharp distributional-signature;∴ tight concept-cluster; generic-lexis->diffuse-activation, broad candidate-set;Attention-heads key/query-match domain-tokens;"Hamiltonian"->{operator,eigenstate,quantum,energy}->register+domain locked;Net:constrained-decoding,vocab=soft-prior over output-distribution; register-matching;#taskdef=decompress->continue
  - adam_arthur 3mo ago
    Information density of the interpretability of the intent from the perspective of a human (or human-like). If the intent is not easy to understand, it's information sparse. Because it takes a lot of CPU (or brainpower) to interpret. You can run gzip on an English sentence to make it more textually dense, but clearly it is not more information dense in this context.
    - docjay 3mo ago
      I’d buy a ticket to ride the philosophical “human-like” comment with you, but I think you might have made an incorrect assumption. The model did not take longer to “decompress” the prompt than it would take for any other prompt of equal token length. If you run it with thinking enabled you might be mistaking that output as some kind of necessary gunzip step, but it’s not. Disable thinking and try again. The prompt was also “easier to understand”, purely in the sense that the response is more or less guarantee to be what I wanted it to say, which was the point behind the demonstration. I went into more detail on it in another comment around here.
      - adam_arthur 3mo ago
        I say "human-like" in the sense that LLMs are fed in text data largely in the exact form (mapped to tokens) that humans read them. Thus from first principles it's most likely that content which is more understandable to humans is also likely to be more understandable to LLMs. Of course they are still capable of interpreting very obscure structures too, but usually at the cost of cognitive performance. I'm open to being wrong about this, and I'm sure it's being researched. (Specifically for text representations) To your point, at some level of intelligence an LLM will be able to infer the intent of your prompt consistently without thinking enabled, in which case interpretability to a human matters less. But for complex tasks you aren't likely to get optimal performance with prompts that are difficult for humans to understand. And yes, you'd see that with thinking enabled as it churns over thousands of tokens trying to "mentally expand" a compressed prompt. Interesting discussion though!
daemonologist 3mo ago
There was a fad a while back of building insanely long prompts - tens of thousands of tokens - including having models write prompts for themselves. I always thought it was counterproductive, especially if you're going to use the prompt more than a couple of times. (That said, the e.g. Claude Code system prompt is insanely long, so if you genuinely have a lot of information to provide maybe it's beneficial. Like, shorter is better, but you don't want to be under-specified.)
CuriouslyC 3mo ago
For Gemini 2.5 and ~GPT5.0-5.1, longer prompts with lots of explicit instructions and examples produced better conformance. Seems like heavily second guessing the models started to get counter productive around the end of last year.

adam_arthur 3mo ago

Information density of the prompt is the most important factor in my experience. And interestingly, LLMs seem particularly bad at writing prompts for other LLMs for this reason (you can guide them to be more dense, just speaking by default). Conciseness is usually a byproduct of information density though.

astrange 3mo ago
> LLMs seem particularly bad at writing prompts for other LLMs for this reason Claude is terrible at this! Probably for the same reason that its writing style in prose is so annoying and full of claudisms.
- irthomasthomas 3mo ago
  Claude use to be leader, too. Their metaprompt was great at the time with opus 3
docjay 3mo ago
Lexical-priming->semantic-space-constraint;specialized-lexis+=sharp distributional-signature;∴ tight concept-cluster; generic-lexis->diffuse-activation, broad candidate-set;Attention-heads key/query-match domain-tokens;"Hamiltonian"->{operator,eigenstate,quantum,energy}->register+domain locked;Net:constrained-decoding,vocab=soft-prior over output-distribution; register-matching;#taskdef=decompress->continue
- adam_arthur 3mo ago
  Information density of the interpretability of the intent from the perspective of a human (or human-like). If the intent is not easy to understand, it's information sparse. Because it takes a lot of CPU (or brainpower) to interpret. You can run gzip on an English sentence to make it more textually dense, but clearly it is not more information dense in this context.
  - docjay 3mo ago
    I’d buy a ticket to ride the philosophical “human-like” comment with you, but I think you might have made an incorrect assumption. The model did not take longer to “decompress” the prompt than it would take for any other prompt of equal token length. If you run it with thinking enabled you might be mistaking that output as some kind of necessary gunzip step, but it’s not. Disable thinking and try again. The prompt was also “easier to understand”, purely in the sense that the response is more or less guarantee to be what I wanted it to say, which was the point behind the demonstration. I went into more detail on it in another comment around here.
    - adam_arthur 3mo ago
      I say "human-like" in the sense that LLMs are fed in text data largely in the exact form (mapped to tokens) that humans read them. Thus from first principles it's most likely that content which is more understandable to humans is also likely to be more understandable to LLMs. Of course they are still capable of interpreting very obscure structures too, but usually at the cost of cognitive performance. I'm open to being wrong about this, and I'm sure it's being researched. (Specifically for text representations) To your point, at some level of intelligence an LLM will be able to infer the intent of your prompt consistently without thinking enabled, in which case interpretability to a human matters less. But for complex tasks you aren't likely to get optimal performance with prompts that are difficult for humans to understand. And yes, you'd see that with thinking enabled as it churns over thousands of tokens trying to "mentally expand" a compressed prompt. Interesting discussion though!

daemonologist 3mo ago

There was a fad a while back of building insanely long prompts - tens of thousands of tokens - including having models write prompts for themselves. I always thought it was counterproductive, especially if you're going to use the prompt more than a couple of times. (That said, the e.g. Claude Code system prompt is insanely long, so if you genuinely have a lot of information to provide maybe it's beneficial. Like, shorter is better, but you don't want to be under-specified.)

CuriouslyC 3mo ago

For Gemini 2.5 and ~GPT5.0-5.1, longer prompts with lots of explicit instructions and examples produced better conformance. Seems like heavily second guessing the models started to get counter productive around the end of last year.