5 ms·
Keep in mind all this kind of stuff can make the model less capable. If it has to think in "plain" English, it may well be squashing quality of code etc output.
by martinald 2mo ago
Keep in mind all this kind of stuff can make the model less capable. If it has to think in "plain" English, it may well be squashing quality of code etc output.
I'm not sure how true this is, but when using "forced" json output it def had a big drop off in quality - https://arxiv.org/html/2408.02442v3 https://arxiv.org/html/2408.02442v3.
I think you're better not fighting it with hacks like this and find a different model.
- silverwind 2mo agoChanging output style shouldn't affect thinking at all.
- semiquaver 2mo agoTell that to all the CLAUDE.md lines across dozens of repos I have to write to get them to understand that git commit standards and PR description standards are different.
- MostlyStable 2mo agoI would not overgeneralize from paper. Firstly: forcing JSON output is, in my opinion, a bigger change than asking it to match the above style guidelines, and secondly, as is always the case with these kinds of papers, what was true for the model tested in the paper may either be completely false, or greatly reduced, in later models. That paper is almost 2 years old and models today have been trained in very different ways (or more accurately post trained in very different ways) and are in general far more capable. Based on that paper, I would maybe try to check if it was true for a modern use case, I would very much not assume it was still true.
- firemelt 2mo agohow about stuff likes https://github.com/JuliusBrussee/caveman https://github.com/JuliusBrussee/caveman is it also make the model less capable?