8 ms·

Show HN: Structured output from LLMs without reprompting

174 points by sandkoan 3y ago

Built a tool for transforming unstructured data into structured outputs using language models (with 100% adherence). If you're facing problems getting GPT to adhere to a schema (JSON, XML, etc.) or regex, need to bulk process some unstructured data, or generate synthetic data, check it out. We run our own tuned model (you can self-host if you want), so, we're able to have incredibly fine grained control over text generation. Repository: https://github.com/automorphic-ai/trex https://github.com/automorphic-ai/trex Playground: https://automorphic.ai/playground https://automorphic.ai/playground

dkjaudyeqooe 3y ago
Don't forget to put a license on your repository.
- sandkoan 3y ago
  Thanks for the reminder—done!
deleted 3y ago
[deleted]
beefnugs 3y ago
oh its 100% predictable alright, predictable to be garbage : the default example chooses height as the wrong "number" whatever that might assume to be, then if you try to change it to define height as perhaps "height in total inches" it still gets it wrong
- sandkoan 3y ago
  Ahh, that—due to compute limitations we're forced to run a very small model that isn't as capable of converting 5'8" to inches. The larger model is, though.
sunshadow 3y ago
What are the benefits over https://github.com/microsoft/guidance/ https://github.com/microsoft/guidance/ ?
- sandkoan 3y ago
  We enable conforming to arbitrary context free grammars in addition to regex patterns, and have a bunch of speed optimizations, as well. Though it may not seem too fast right now on account of the hundreds of simultaneous requests we're getting :)
  - swyx 3y ago
    100s of RPS? did you have a successful launch elsewhere or something? bc this repo currently only has 21 stars, which taking normal correlation into account does not imply that level of traffic
    - sandkoan 3y ago
      We have folks playing around with it mostly through the playground / raw HTTP endpoints as opposed to the Python API. And we've got some batch jobs running, which adds further traffic.
- behnamoh 3y ago
  guidance is a dead project. it worked well as a hobby side project by MS researchers, but it clearly isn't a long-term solution as new LLMs are introduced.
  - amkkma 3y ago
    What about LMQL?
    - darkteflon 3y ago
      I hadn't heard of LMQL before - have you tried building with it and, if so, would you be willing to share your experience?
darkteflon 3y ago
Could you contextualise this against OpenAI’s native functions?
- sandkoan 3y ago
  Problems with OpenAI: 1) You're wasting GPT tokens on outputting JSON instead of meaningful information. 2) GPT functions won't, with absolute, 100% certainty, return JSON in the schema you want. In 1% to 3% of cases it hallucinates fields, etc. 3) This also allows you to output data in arbitrary non-JSON formats. 4) You can't self-host OpenAI functions.
  - darkteflon 3y ago
    Thanks, all good points that would seem to make this library a good fit for certain use-cases. As with the other poster, I’d be interested to hear a bit more about point 1.
    - jacky2wong 3y ago
      Point 1 doesn't feel like a good enough reason. The number of tokens outputted as a JSON is so small if you tell GPT to output it properly.
      - sandkoan 3y ago
        Costs add up surprisingly quickly. A quote-colon-space-quote combo alone is four tokens wasted. Now scale that up....
        agroot12 3y ago
        Using tiktokenizer, these are only two tokens: quote-colon is token 498, space-quote is token 330 (as per https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/ ). But I agree to the general argument. I think what factors in even more when you use the API is that you do not have fine-grained control over the generation process. If you follow the MS guidance approach, you fill in structured text yourself, and then let the model generate only the value parts, e.g. up to the next quote. To do that more or less word by word, you have multiple API calls, and have to be very smart about providing the right stop tokens.
roseway4 3y ago
Looking at the playground, it appears the few shot examples in the prompt and CFG are duplicative. What is the relationship between the two? When you say in another comment that using OpenAI functions to output JSON is a waste of tokens, how are you generating the JSON output? And why do your prompts then include few shot examples of JSON objects?
- sandkoan 3y ago
  The prompt is given to our model as a guiding aid (a suggestion), and the cfg is used to constrain the model to generate only tokens that abide by the schema (an enforcement). That's how we ensure only valid outputs at text generation time. We also prefill some tokens depending on the set of allowed tokens at a given state, so the model doesn't waste resources trying to predict them.
  - roseway4 3y ago
    When you say “our model”, are you using a custom LLM for completions vs OpenAI or other LLM vendor?
    - sandkoan 3y ago
      Custom LLM—hence the self-hostability.
behnamoh 3y ago
the more it goes, the more I realize that the true power of LLMs is not in unstructured text that they can generate, but in structured output. but there are two approaches to achieve this: 1. LMQL/guidance/JSONformer/OP's post 2. finetuning the model to understand function calls and their (potentially) JSON schemas. there was a comment here about OpenAI's approach (finetuning a model to understand function call) which raised a good point: since finetuning is often forgetful (previous knowledge learnt by the model gets forgotten a little bit), it's not clear if OpenAI's approach has made GPT-4 less capable than it was before. Not to mention that you're still dealing with a statistical process (LLM), not a locked-in algorithm that generates the desired schema 100% the time. Which brings me to the other approach: steering the LLM's output __as it is generating tokens__, which is what LMQL does. This results in less token usage (you don't send function schema as part of your prompt/message to OpenAI) and 100% accuracy because token probabilities are modified (e.g., 0% chance of any character except ":" after a double quotation mark).
- brucethemoose2 3y ago
  > Which brings me to the other approach: steering the LLM's output __as it is generating tokens__ A relevant PR: https://github.com/ggerganov/llama.cpp/pull/1773 https://github.com/ggerganov/llama.cpp/pull/1773 The plan is to support arbitrary grammar files to constrain token generation, similar to the grammar files here: https://github.com/antlr/grammars-v4 https://github.com/antlr/grammars-v4
  - kelseyfrog 3y ago
    I mean even Jsonformer used LogitsWarper when generating numbers, but yes arbitrary grammars are infinitely more powerful.
- darkteflon 3y ago
  I'd never heard of LMQL before today, but it looks very nice. Do you have any experience building with it and, if so, would you be willing to comment on what it's like?
easygenes 3y ago
You mention self hosting the model. Do you have the model weights up on HuggingFace?
- sandkoan 3y ago
  This is model agnostic, actually—any model on HuggingFace is compatible. So if someone wanted to run this with their own model, they could.
  - easygenes 3y ago
    Ok, the way you wrote it implied the model itself is available to self host.
    - sandkoan 3y ago
      Ahh, no, the value of this isn't as much the model as it is the infrastructure enabling structure enforcement.
jensneuse 3y ago
We're using a similar approach with OpenAI. The user can define a schema using zod and call a prompt. We're then using OpenAI functions behind the scenes to parse the answer into the shape the user wants. Add JSON schema validation on top and we can be sure that the response conforms to our Schema. Some more details and examples can be found in this blog post: https://wundergraph.com/blog/beyond_functions_seamlessly_build_ai_enhanced_apis_with_openai https://wundergraph.com/blog/beyond_functions_seamlessly_bui...
marban 3y ago
Can easily be done with OAI's new function calling.
- sandkoan 3y ago
  https://news.ycombinator.com/item?id=36752991 https://news.ycombinator.com/item?id=36752991
minzi 3y ago
Can someone explain what they use structured output for? I’m just curious what kinds of use cases people have found for it.
- StackOverlord 3y ago
  I use it to create a narrative plan for multiple interleaved plots in an open-world interactive fiction. For instance: https://pastebin.com/QFZmEAJA https://pastebin.com/QFZmEAJA I use Clojure's EDN JSON-equivalent format, and what you can read in this paste is an attempt to make GPT write its own prompt in a conversation where I gradually built a format for this narrative structure using Clojure. It turns out GPT isn't able to produce EDN data using this prompt (it will produce something that looks like the "grammar" displayed in the paste from above that GPT came up with, not Clojure data as instructed). I can get it to output EDN but I need to provide an example, but then the story in the example will tend to leak into the generated story. And it still has problems, missing keys for instance, or it doesn't used nested subnarratives, or just fails at outputting strict EDN, for instance forgetting or adding surnumerary parenthesis. Here's what the EDN structure I want to get might look like: https://pastebin.com/iTxtn8gk https://pastebin.com/iTxtn8gk And here's what kind of text can be generated from it: https://pastebin.com/qJPjmrTd https://pastebin.com/qJPjmrTd For now I haven't even used the parseable EDN programmatically. I just feed it back to GPT as a string (realistically, I'd need to use a vector database to store these narrative blocks). However GPT will slowly erode the structure with every round-trip.
Janymos 3y ago
What's the difference between self hosting this and manually running https://github.com/r2d4/parserllm https://github.com/r2d4/parserllm ?
foundry27 3y ago
I think the references to this being a “tool” that you can “self-host if you want” are a little disingenuous, especially after seeing that the linked GitHub project doesn’t mention the fact that it’s just a thin wrapper client making requests to a remote server until you’re halfway through the README. The product might be great, but introducing it to the community in this way doesn’t foster much trust in your company from the perspective of a potential customer. The only reference I can find to this being a self-hosted model is a blurb in the GitHub README saying “If you'd like to self-host this in your own cloud, email us”. Sure, I can email my OpenAI/Microsoft rep and self-host GPT-4 in my own cloud for enough money too, but that doesn’t change the fact that the primary business model is SaaS. Just be up-front about this fact in community posts, rather than obfuscating it. Your website does a great job with that.
- ycmg 3y ago
  We're offering the API for free, and have the playground available for anyone to use as well. As for self-hosting, we haven't quite decided on the business model, and were trying to gauge interest. For now, we're happy to offer free self-hosting for the first few people, but may charge eventually. Our intention wasn't to obfuscate this, so thanks for the feedback. We'll try making that more apparent.

174 points by sandkoan 3y ago

dkjaudyeqooe 3y ago

Don't forget to put a license on your repository.

sandkoan 3y ago
Thanks for the reminder—done!

deleted 3y ago

[deleted]

beefnugs 3y ago

oh its 100% predictable alright, predictable to be garbage : the default example chooses height as the wrong "number" whatever that might assume to be, then if you try to change it to define height as perhaps "height in total inches" it still gets it wrong

sandkoan 3y ago
Ahh, that—due to compute limitations we're forced to run a very small model that isn't as capable of converting 5'8" to inches. The larger model is, though.

sunshadow 3y ago

What are the benefits over https://github.com/microsoft/guidance/ https://github.com/microsoft/guidance/ ?

sandkoan 3y ago
We enable conforming to arbitrary context free grammars in addition to regex patterns, and have a bunch of speed optimizations, as well. Though it may not seem too fast right now on account of the hundreds of simultaneous requests we're getting :)
- swyx 3y ago
  100s of RPS? did you have a successful launch elsewhere or something? bc this repo currently only has 21 stars, which taking normal correlation into account does not imply that level of traffic
  - sandkoan 3y ago
    We have folks playing around with it mostly through the playground / raw HTTP endpoints as opposed to the Python API. And we've got some batch jobs running, which adds further traffic.
behnamoh 3y ago
guidance is a dead project. it worked well as a hobby side project by MS researchers, but it clearly isn't a long-term solution as new LLMs are introduced.
- amkkma 3y ago
  What about LMQL?
  - darkteflon 3y ago
    I hadn't heard of LMQL before - have you tried building with it and, if so, would you be willing to share your experience?

darkteflon 3y ago

Could you contextualise this against OpenAI’s native functions?

sandkoan 3y ago
Problems with OpenAI: 1) You're wasting GPT tokens on outputting JSON instead of meaningful information. 2) GPT functions won't, with absolute, 100% certainty, return JSON in the schema you want. In 1% to 3% of cases it hallucinates fields, etc. 3) This also allows you to output data in arbitrary non-JSON formats. 4) You can't self-host OpenAI functions.
- darkteflon 3y ago
  Thanks, all good points that would seem to make this library a good fit for certain use-cases. As with the other poster, I’d be interested to hear a bit more about point 1.
  - jacky2wong 3y ago
    Point 1 doesn't feel like a good enough reason. The number of tokens outputted as a JSON is so small if you tell GPT to output it properly.
    - sandkoan 3y ago
      Costs add up surprisingly quickly. A quote-colon-space-quote combo alone is four tokens wasted. Now scale that up....
      - agroot12 3y ago
        Using tiktokenizer, these are only two tokens: quote-colon is token 498, space-quote is token 330 (as per https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/ ). But I agree to the general argument. I think what factors in even more when you use the API is that you do not have fine-grained control over the generation process. If you follow the MS guidance approach, you fill in structured text yourself, and then let the model generate only the value parts, e.g. up to the next quote. To do that more or less word by word, you have multiple API calls, and have to be very smart about providing the right stop tokens.

roseway4 3y ago

Looking at the playground, it appears the few shot examples in the prompt and CFG are duplicative. What is the relationship between the two? When you say in another comment that using OpenAI functions to output JSON is a waste of tokens, how are you generating the JSON output? And why do your prompts then include few shot examples of JSON objects?

sandkoan 3y ago
The prompt is given to our model as a guiding aid (a suggestion), and the cfg is used to constrain the model to generate only tokens that abide by the schema (an enforcement). That's how we ensure only valid outputs at text generation time. We also prefill some tokens depending on the set of allowed tokens at a given state, so the model doesn't waste resources trying to predict them.
- roseway4 3y ago
  When you say “our model”, are you using a custom LLM for completions vs OpenAI or other LLM vendor?
  - sandkoan 3y ago
    Custom LLM—hence the self-hostability.

behnamoh 3y ago

the more it goes, the more I realize that the true power of LLMs is not in unstructured text that they can generate, but in structured output. but there are two approaches to achieve this: 1. LMQL/guidance/JSONformer/OP's post 2. finetuning the model to understand function calls and their (potentially) JSON schemas. there was a comment here about OpenAI's approach (finetuning a model to understand function call) which raised a good point: since finetuning is often forgetful (previous knowledge learnt by the model gets forgotten a little bit), it's not clear if OpenAI's approach has made GPT-4 less capable than it was before. Not to mention that you're still dealing with a statistical process (LLM), not a locked-in algorithm that generates the desired schema 100% the time. Which brings me to the other approach: steering the LLM's output __as it is generating tokens__, which is what LMQL does. This results in less token usage (you don't send function schema as part of your prompt/message to OpenAI) and 100% accuracy because token probabilities are modified (e.g., 0% chance of any character except ":" after a double quotation mark).

brucethemoose2 3y ago
> Which brings me to the other approach: steering the LLM's output __as it is generating tokens__ A relevant PR: https://github.com/ggerganov/llama.cpp/pull/1773 https://github.com/ggerganov/llama.cpp/pull/1773 The plan is to support arbitrary grammar files to constrain token generation, similar to the grammar files here: https://github.com/antlr/grammars-v4 https://github.com/antlr/grammars-v4
- kelseyfrog 3y ago
  I mean even Jsonformer used LogitsWarper when generating numbers, but yes arbitrary grammars are infinitely more powerful.
darkteflon 3y ago
I'd never heard of LMQL before today, but it looks very nice. Do you have any experience building with it and, if so, would you be willing to comment on what it's like?

easygenes 3y ago

You mention self hosting the model. Do you have the model weights up on HuggingFace?

sandkoan 3y ago
This is model agnostic, actually—any model on HuggingFace is compatible. So if someone wanted to run this with their own model, they could.
- easygenes 3y ago
  Ok, the way you wrote it implied the model itself is available to self host.
  - sandkoan 3y ago
    Ahh, no, the value of this isn't as much the model as it is the infrastructure enabling structure enforcement.

jensneuse 3y ago

We're using a similar approach with OpenAI. The user can define a schema using zod and call a prompt. We're then using OpenAI functions behind the scenes to parse the answer into the shape the user wants. Add JSON schema validation on top and we can be sure that the response conforms to our Schema. Some more details and examples can be found in this blog post: https://wundergraph.com/blog/beyond_functions_seamlessly_build_ai_enhanced_apis_with_openai https://wundergraph.com/blog/beyond_functions_seamlessly_bui...

marban 3y ago

Can easily be done with OAI's new function calling.

sandkoan 3y ago
https://news.ycombinator.com/item?id=36752991 https://news.ycombinator.com/item?id=36752991

minzi 3y ago

Can someone explain what they use structured output for? I’m just curious what kinds of use cases people have found for it.

StackOverlord 3y ago
I use it to create a narrative plan for multiple interleaved plots in an open-world interactive fiction. For instance: https://pastebin.com/QFZmEAJA https://pastebin.com/QFZmEAJA I use Clojure's EDN JSON-equivalent format, and what you can read in this paste is an attempt to make GPT write its own prompt in a conversation where I gradually built a format for this narrative structure using Clojure. It turns out GPT isn't able to produce EDN data using this prompt (it will produce something that looks like the "grammar" displayed in the paste from above that GPT came up with, not Clojure data as instructed). I can get it to output EDN but I need to provide an example, but then the story in the example will tend to leak into the generated story. And it still has problems, missing keys for instance, or it doesn't used nested subnarratives, or just fails at outputting strict EDN, for instance forgetting or adding surnumerary parenthesis. Here's what the EDN structure I want to get might look like: https://pastebin.com/iTxtn8gk https://pastebin.com/iTxtn8gk And here's what kind of text can be generated from it: https://pastebin.com/qJPjmrTd https://pastebin.com/qJPjmrTd For now I haven't even used the parseable EDN programmatically. I just feed it back to GPT as a string (realistically, I'd need to use a vector database to store these narrative blocks). However GPT will slowly erode the structure with every round-trip.

Janymos 3y ago

What's the difference between self hosting this and manually running https://github.com/r2d4/parserllm https://github.com/r2d4/parserllm ?

foundry27 3y ago

I think the references to this being a “tool” that you can “self-host if you want” are a little disingenuous, especially after seeing that the linked GitHub project doesn’t mention the fact that it’s just a thin wrapper client making requests to a remote server until you’re halfway through the README. The product might be great, but introducing it to the community in this way doesn’t foster much trust in your company from the perspective of a potential customer. The only reference I can find to this being a self-hosted model is a blurb in the GitHub README saying “If you'd like to self-host this in your own cloud, email us”. Sure, I can email my OpenAI/Microsoft rep and self-host GPT-4 in my own cloud for enough money too, but that doesn’t change the fact that the primary business model is SaaS. Just be up-front about this fact in community posts, rather than obfuscating it. Your website does a great job with that.

ycmg 3y ago
We're offering the API for free, and have the playground available for anyone to use as well. As for self-hosting, we haven't quite decided on the business model, and were trying to gauge interest. For now, we're happy to offer free self-hosting for the first few people, but may charge eventually. Our intention wasn't to obfuscate this, so thanks for the feedback. We'll try making that more apparent.