3 ms·

Prior to this it was possible to get the same effect by defining a tool with the schema that you wanted and then telling the Anthropic API to always use that to

by simonw 1y ago

Prior to this it was possible to get the same effect by defining a tool with the schema that you wanted and then telling the Anthropic API to always use that tool. I implemented structured outputs for Claude that way here: https://github.com/simonw/llm-anthropic/blob/500d277e9b4bec67a853c536f03017c5b67862c9/llm_anthropic.py#L520-L527 https://github.com/simonw/llm-anthropic/blob/500d277e9b4bec6...

fnordsensei 1y ago
Same, but it’s a PITA when you also want to support tool calling at the same time. Had to do a double call: call and check if it will use tools. If not, call again and force the use of the (now injected) return schema tool.
mparis 1y ago
We've been running structured outputs via Claude on Bedrock in production for a year now and it works great. Give it a JSON schema, inject a '{', and sometimes do a bit of custom parsing on the response. GG Nice to see them support it officially; however, OpenAI has officially supported this for a while but, at least historically, I have been unable to use it because it adds deterministic validation that errors on certain standard JSON Schema elements that we used. The lack of "official" support is the feature that pushed us to use Claude in the first place. It's unclear to me that we will need "modes" for these features. Another example: I used to think that I couldn't live without Claude Code "plan mode". Then I used Codex and asked it to write a markdown file with a todo list. A bit more typing but it works well and it's nice to be able to edit the plan directly in editor. Agree or Disagree?
- Karrot_Kream 1y ago
  Before Claude Code shipped with plan mode, the workflow for using most coding agents was to have it create a `PLAN.md` and update/execute that plan. Planning mode was just a first class version of what users were already doing.
- kami23 1y ago
  Claude Code keeps coming out with a lot of really nice tools that others haven't started to emulate from what I've seen. My favorite one is going through the plan interactively. It turns it into a multiple choice / option TUI and the last choose is always reprompt that section of the plan. I had switch back to codex recently and not being able to do my planning solely in the CLI feels like the early 1900s. To trigger the interactive mode. Do something like: Plan a fix for: <Problem statement> Please walk me through any options or questions you might have interactively.
- tempaccount420 1y ago
  > Give it a JSON schema, inject a '{', and sometimes do a bit of custom parsing on the response I would hope that this is not what OpenAI/Anthropic do under the hood, because otherwise, what if one of the strings needs a lot of \escapes? Is it also supposed to newer write actual newlines in strings? It's awkward. The ideal solution would be to have some special tokens like [object_start] [object_end] and [string_start] [string_end].
koakuma-chan 1y ago
I don't think the tool input schema thing does that inference-time trick. I think it just dumps the JSON schema into the context, and tells the model to conform to that schema.
alex_duf 1y ago
It's not 100% success, I've had responses that didn't match my schema. I think the new feature goes on to limit which token can be output, which brings a guarantee, where the tools are a suggestion.

by simonw 1y ago

fnordsensei 1y ago

Same, but it’s a PITA when you also want to support tool calling at the same time. Had to do a double call: call and check if it will use tools. If not, call again and force the use of the (now injected) return schema tool.

mparis 1y ago

We've been running structured outputs via Claude on Bedrock in production for a year now and it works great. Give it a JSON schema, inject a '{', and sometimes do a bit of custom parsing on the response. GG Nice to see them support it officially; however, OpenAI has officially supported this for a while but, at least historically, I have been unable to use it because it adds deterministic validation that errors on certain standard JSON Schema elements that we used. The lack of "official" support is the feature that pushed us to use Claude in the first place. It's unclear to me that we will need "modes" for these features. Another example: I used to think that I couldn't live without Claude Code "plan mode". Then I used Codex and asked it to write a markdown file with a todo list. A bit more typing but it works well and it's nice to be able to edit the plan directly in editor. Agree or Disagree?

Karrot_Kream 1y ago
Before Claude Code shipped with plan mode, the workflow for using most coding agents was to have it create a `PLAN.md` and update/execute that plan. Planning mode was just a first class version of what users were already doing.
kami23 1y ago
Claude Code keeps coming out with a lot of really nice tools that others haven't started to emulate from what I've seen. My favorite one is going through the plan interactively. It turns it into a multiple choice / option TUI and the last choose is always reprompt that section of the plan. I had switch back to codex recently and not being able to do my planning solely in the CLI feels like the early 1900s. To trigger the interactive mode. Do something like: Plan a fix for: <Problem statement> Please walk me through any options or questions you might have interactively.
tempaccount420 1y ago
> Give it a JSON schema, inject a '{', and sometimes do a bit of custom parsing on the response I would hope that this is not what OpenAI/Anthropic do under the hood, because otherwise, what if one of the strings needs a lot of \escapes? Is it also supposed to newer write actual newlines in strings? It's awkward. The ideal solution would be to have some special tokens like [object_start] [object_end] and [string_start] [string_end].

koakuma-chan 1y ago

I don't think the tool input schema thing does that inference-time trick. I think it just dumps the JSON schema into the context, and tells the model to conform to that schema.

alex_duf 1y ago

It's not 100% success, I've had responses that didn't match my schema. I think the new feature goes on to limit which token can be output, which brings a guarantee, where the tools are a suggestion.