4 ms·
For complex tasks like coding, my experience is that asking for a complex output format hurts performance on the underlying task. This showed up clearly in code
by anotherpaulg 3y ago
For complex tasks like coding, my experience is that asking for a complex output format hurts performance on the underlying task. This showed up clearly in code editing benchmarks of GPT-3.5 and GPT-4:
https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html
I’m curious if you have measured whether the “constrained generation” that you’re doing suffers from similar downsides?
- infecto 3y ago100% have observed the same over many tests. No loss in fidelity when responding in spoken language style of formatting but using json is disastrous.
- nouri 3y agoUsing OpenAI Function Calls or asking for JSON in the prompt?
- infecto 3y agoI have noticed it in both but have been working with json output before function calling was introduced so I have more evidence on that side. The times I have tried to implement it in a function call I was equally unimpressed with it.
- speedgoose 3y agoWhile not ideal, could a workaround be to ask in spoken language first, and then ask to format it in JSON?
- infecto 3y agoThat’s what we have been doing. Two passes. The task and then the format.
- darkteflon 3y agoWe’ve seen this too. We run them as two separate stages - “reason”, log the intermediate output, then parse.