5 ms·

I've only used 5.4 for 1 prompt (edit: 3@high now) so far (reasoning: extra high, took really long), and it was to analyse my codebase and write an evaluation o

I've only used 5.4 for 1 prompt (edit: 3@high now) so far (reasoning: extra high, took really long), and it was to analyse my codebase and write an evaluation on a topic. But I found its writing and analysis thoughtful, precise, and surprisingly clearly written, unlike 5.3-Codex. It feels very lucid and uses human phrasing. It might be my AGENTS.md requiring clearer, simpler language, but at least 5.4's doing a good job of following the guidelines. 5.3-Codex wasn't so great at simple, clear writing.

irishcoffee 8mo ago
> It might be my AGENTS.md requiring clearer, simpler language If you gave the exact same markdown file to me and I posted ed the exact same prompts as you, would I get the same results?
- m3kw9 8mo ago
  you probably can't and asking agents.md to "make it clearer" will likely give you the illusion of clearer language without actual well structured tests. agents.md is to usually change what the llm should focus on doing more that suits you. Not to say stuff like "be better", "make no mistakes"
- creamyhorror 8mo ago
  I'm not sure if the model (under its temperature/other settings) produces deterministic responses. But I do think models' style and phrasing are fairly changeable via AGENTS.md-style guidelines. 5.4's choice of terms and phrasing is very precise and unambiguous to me, whereas 5.3-Codex often uses jargon and less precise phrases that I have to ask further about or demand fuller explanations for via AGENTS.md.
  - irishcoffee 8mo ago
    So sharing markdown files is functionally useless, or no?
    - Tarq0n 8mo ago
      No it's just stochastic like everything about LLMs. The md file will bias results towards a certain set of outcomes.
- Kostchei 7mo ago
  In case you missed it, json is less likely to be written over than markdown by agents- something to do with the structure being more rigid
sampton 8mo ago
That's been my experience as well switching from Opus to Codex. Reasoning takes longer but answers are precise. Claude is sloppy in comparison.
- throwaway911282 8mo ago
  codex has been really good so far and the fast mode is cherry on top! and the very generous limits is another cherry on top
  - slopinthebag 8mo ago
    It's well worth the $20 to not deal with any limits and have it handle all the boilerplate repetitive BS us programmers seem forced to deal with. I think 80% of the benefit comes from spending that $20 (20%? :P) and just having it do the lame shit that we probably shouldn't have to do but somehow need to.
- solenoid0937 8mo ago
  Weird, I have had the opposite experience. Codex is good at doing precisely what I tell it to do, Opus suggests well thought out plans even if it needs to push back to do it.
  - slopinthebag 8mo ago
    This is just the stochastic nature of LLM's at play. I think all of the SOTA models are roughly equivalent, but without enough samples people end up reading into it too much.
    - oorza 8mo ago
      There's a certain amount of variance in the way that people utilize these agents. Put five people in a room and ask them to compose the same prompt and you have five distinct prompts. Couple this with the fact that models respond better/worse to certain prompts depending on the stylistic composition of the prompt itself. And since people tend to write in the same style, you'd get people who have more luck with one model over another, where one model happens to align more readily with their prompt style. To wit, I have noticed that I tend to prefer Codex's output for planning and review, but Opus for implementation; this is inverted from others at work.
pembrook 8mo ago
The latest research these days is that including an AGENTS.md file only makes outcomes worse with frontier models.
- madeofpalk 8mo ago
  :( how can i get claude to always make sure it prettier-s and lints changes before pushing up the pr though?
  - mckirk 8mo ago
    I think what that research found is that _auto-generated_ agent instructions made results slightly worse, but human-written ones made them slightly better, presumably because anything the model could auto-generate, it could also find out in-context. But especially for conventions that would be difficult to pick up on in-context, these instruction files absolutely make sense. (Though it might be worth it to split them into multiple sub-files the model only reads when it needs that specific workflow.)
  - JofArnold 8mo ago
    Run prettier etc in a hook.
  - emsimot 8mo ago
    Git hooks
- solarkraft 8mo ago
  From what I remember, this was for describing the project’s structure over letting the model discover it itself, no? Because how else are you going to teach it your preferred style and behavior?
- FINDarkside 8mo ago
  I wouldn't draw such conclusions from one preprint paper. Especially since they measured only success rate, while quite often AGENTS.md exists to improve code quality, which wasn't measured. And even then, the paper concluded that human written AGENTS.md raised success rates.
torginus 8mo ago
Honestly, while I'd like to believe you, there's always a post about how $MODEL+1 delivered powerful insights about the very nature of the universe in precise Hegelian dialectic, while $MODEL's output was indistinguishable from a pack of screeching sexually frustrated bonobos
- Der_Einzige 8mo ago
  Hegel is a mind virus. Let his terrible thought rest in peace.
dana321 8mo ago
5.4 very high didn't notice in my codebase a glaring issue that drops all data being sent around the network.
joegibbs 8mo ago
The weird phrasing was my biggest gripe with 5.3 so I'm glad they've fixed that up. It couldn't say anything without a heap of impenetrable jargon and it was obsessed with the word "drive". Nothing could cause anything, it had to be "driven".
- joegibbs 8mo ago
  I tested 5.4 out and I don't think it's actually any better in this regard, it's still using odd words, and it has this very annoying habit of saying "the issue was caused by [correct solution], not [thing that I never suggested]" as if it's correcting me on it
startages 8mo ago
I thought I had something wrong within my setup, I could never use Codex 5.3 while everyone else was praising it. It uses some weird terms and complex jargon and doesn't really make it clear what it was doing or planning to do unlike Opus which makes things clear, this allows me to give accurate feedback and change plans and make proper decision.

irishcoffee 8mo ago

> It might be my AGENTS.md requiring clearer, simpler language If you gave the exact same markdown file to me and I posted ed the exact same prompts as you, would I get the same results?

m3kw9 8mo ago
you probably can't and asking agents.md to "make it clearer" will likely give you the illusion of clearer language without actual well structured tests. agents.md is to usually change what the llm should focus on doing more that suits you. Not to say stuff like "be better", "make no mistakes"
creamyhorror 8mo ago
I'm not sure if the model (under its temperature/other settings) produces deterministic responses. But I do think models' style and phrasing are fairly changeable via AGENTS.md-style guidelines. 5.4's choice of terms and phrasing is very precise and unambiguous to me, whereas 5.3-Codex often uses jargon and less precise phrases that I have to ask further about or demand fuller explanations for via AGENTS.md.
- irishcoffee 8mo ago
  So sharing markdown files is functionally useless, or no?
  - Tarq0n 8mo ago
    No it's just stochastic like everything about LLMs. The md file will bias results towards a certain set of outcomes.
Kostchei 7mo ago
In case you missed it, json is less likely to be written over than markdown by agents- something to do with the structure being more rigid

sampton 8mo ago

That's been my experience as well switching from Opus to Codex. Reasoning takes longer but answers are precise. Claude is sloppy in comparison.

throwaway911282 8mo ago
codex has been really good so far and the fast mode is cherry on top! and the very generous limits is another cherry on top
- slopinthebag 8mo ago
  It's well worth the $20 to not deal with any limits and have it handle all the boilerplate repetitive BS us programmers seem forced to deal with. I think 80% of the benefit comes from spending that $20 (20%? :P) and just having it do the lame shit that we probably shouldn't have to do but somehow need to.
solenoid0937 8mo ago
Weird, I have had the opposite experience. Codex is good at doing precisely what I tell it to do, Opus suggests well thought out plans even if it needs to push back to do it.
- slopinthebag 8mo ago
  This is just the stochastic nature of LLM's at play. I think all of the SOTA models are roughly equivalent, but without enough samples people end up reading into it too much.
  - oorza 8mo ago
    There's a certain amount of variance in the way that people utilize these agents. Put five people in a room and ask them to compose the same prompt and you have five distinct prompts. Couple this with the fact that models respond better/worse to certain prompts depending on the stylistic composition of the prompt itself. And since people tend to write in the same style, you'd get people who have more luck with one model over another, where one model happens to align more readily with their prompt style. To wit, I have noticed that I tend to prefer Codex's output for planning and review, but Opus for implementation; this is inverted from others at work.

pembrook 8mo ago

The latest research these days is that including an AGENTS.md file only makes outcomes worse with frontier models.

madeofpalk 8mo ago
:( how can i get claude to always make sure it prettier-s and lints changes before pushing up the pr though?
- mckirk 8mo ago
  I think what that research found is that _auto-generated_ agent instructions made results slightly worse, but human-written ones made them slightly better, presumably because anything the model could auto-generate, it could also find out in-context. But especially for conventions that would be difficult to pick up on in-context, these instruction files absolutely make sense. (Though it might be worth it to split them into multiple sub-files the model only reads when it needs that specific workflow.)
- JofArnold 8mo ago
  Run prettier etc in a hook.
- emsimot 8mo ago
  Git hooks
solarkraft 8mo ago
From what I remember, this was for describing the project’s structure over letting the model discover it itself, no? Because how else are you going to teach it your preferred style and behavior?
FINDarkside 8mo ago
I wouldn't draw such conclusions from one preprint paper. Especially since they measured only success rate, while quite often AGENTS.md exists to improve code quality, which wasn't measured. And even then, the paper concluded that human written AGENTS.md raised success rates.

torginus 8mo ago

Honestly, while I'd like to believe you, there's always a post about how $MODEL+1 delivered powerful insights about the very nature of the universe in precise Hegelian dialectic, while $MODEL's output was indistinguishable from a pack of screeching sexually frustrated bonobos

Der_Einzige 8mo ago
Hegel is a mind virus. Let his terrible thought rest in peace.

dana321 8mo ago

5.4 very high didn't notice in my codebase a glaring issue that drops all data being sent around the network.

joegibbs 8mo ago

The weird phrasing was my biggest gripe with 5.3 so I'm glad they've fixed that up. It couldn't say anything without a heap of impenetrable jargon and it was obsessed with the word "drive". Nothing could cause anything, it had to be "driven".

joegibbs 8mo ago
I tested 5.4 out and I don't think it's actually any better in this regard, it's still using odd words, and it has this very annoying habit of saying "the issue was caused by [correct solution], not [thing that I never suggested]" as if it's correcting me on it

startages 8mo ago

I thought I had something wrong within my setup, I could never use Codex 5.3 while everyone else was praising it. It uses some weird terms and complex jargon and doesn't really make it clear what it was doing or planning to do unlike Opus which makes things clear, this allows me to give accurate feedback and change plans and make proper decision.