9 ms·
Jsonformer: Generate structured output from LLMs
- wy35 3y agoVery interesting. I've only been using OpenAI APIs so this logit stuff is new to me.
- esafak 3y agoIt's a testament to the democratization of ML that practitioners today can get by without knowing what a logit is.
- tysam_and 3y agoAnd I am personally glad for that, for one! This means it's accessible to more people without requiring specialized knowledge, and while, yes, I think that always triggers an internal reaction from most of us when it comes to thinking about field dilution, it's almost a necessary tradeoff (like, say, the uncertainty principle) when expanding the field out to more people. So, hurray! We've made it more accessible. And hopefully in years to come, even very much more so! <3 :)
- koboll 3y agoI'm flabbergasted that OpenAI does not yet offer an API that reliably returns JSON based on some schema you feed it. It's sort of possible to force it to do this but not really to a production-ready degree.
- Der_Einzige 3y agoI've complained bitterly and openly about how annoying it is that OpenAI locks down access to the full probability distribution. Glad to see that others are running into this stupid limitation and are doing work related to it.
- benob 3y agoHow about going one step further and constrain transformer output with a context-free grammar? That way you can generate more conformant code such as Python or C.
- Der_Einzige 3y agoThis may be possible as constraints using constrained beam search, which huggingface has quietly supported for a long time.
- gamegoblin 3y agoWouldn't even need to beam search if you restrict it to deterministic context free grammars, which would satisfy > 95% of these "generate some JSON schema" use-cases. For DCFGs you can just zero-out the probability for any token that is invalid in the context, no lookahead or search needed. Wouldn't work for truly context free things like most programming languages, though.
- sundarurfriend 3y ago> Bulletproof JSON generation: Jsonformer ensures that the generated JSON is always syntactically correct and conforms to the specified schema. This is an important definition to take note of: "bulletproof" doesn't mean that you'll get good or correct data. It only means that it'll be valid JSON and in a particular schema that you specify (because the LLM isn't building the JSON in the first place, the library is). It's an interesting idea. But it's not clear if they've validated the heuristics they use, to see how well it performs in terms of accuracy against, say, some kind of BeautifulSoup-like attempt to make sense of the JSON-ish that the LLM produces and correct that to be valid JSON, or any other approach to the problem.
- dragonwriter 3y agoI wonder if LLMs are at the point where reprompting the LLM with a very similar error message to what you would output to a human from a user-friendly JSON processing tool for the error would usually be a good way to fix errors.
- execveat 3y agoYeah, but that could require multiple queries, which isn't very efficient. Training model just to fix JSON would be better.
- newhouseb 3y agoSometimes, but it very much depends on the context (no pun intended). If it's a pure syntax issue, OpenAI models will almost certainly make the right correction. If it's more abstract, like the LLM has hallucinated a property that is invalid as part of some larger schema you can quickly descend into the LLM gaslighting you into saying that it has fixed things when it hasn't.
- Jayakumark 3y agoThis is great that it does not use OpenAI and runs locally
- Der_Einzige 3y agoLove to see further work on constrained decoding like this and other systems introduced in the comments! See my work and the paper about it. I've got a lot of y'all beat on this (constrained decoding, not the templating and structuring) by about a year: https://github.com/hellisotherpeople/constrained-text-generation-studio https://github.com/hellisotherpeople/constrained-text-genera...
- newhouseb 3y agoOh nice! I built a similar system a few weeks ago: https://github.com/newhouseb/clownfish https://github.com/newhouseb/clownfish I think the main differentiating factor here is that this is better if you have a simpler JSON schema without enums or oneOf constraints. If you do have these constraints, i.e. let's say you wanted an array of different types that represented a items on a menu { kind: pizza, toppings: [pepperoni] } or { kind: ice_cream, flavor: vanilla | strawberry } then you would need something more sophisticated like clownfish that can ask the LLM to pick specific properties (and an ability to do some backtracking so you can do proper beam search). For completeness, another common approach can be found here: https://github.com/ShreyaR/guardrails https://github.com/ShreyaR/guardrails which essentially boils down to "provide the schema in the prompt and ask the LLM to correct things if it fails to get the schema right the first time."
- gamegoblin 3y agoI hate that gpt-3.5-turbo is so cheap that using systems like guardrails is a sane thing to do. I can almost always prompt davinci-003 without guardrails in a way to get my exact schema 1-shot, whereas guardrails + 3.5-turbo will often consume 2-4x more tokens, but that still makes it significantly cheaper.
- brigadier132 3y agoThe problem people are having is hitting the rate limits for chat gpt.
- selcuka 3y agoIs there a rate limit for the paid API [1] too? [1] https://platform.openai.com/docs/models/gpt-3-5 https://platform.openai.com/docs/models/gpt-3-5
- gamegoblin 3y agoYou will get rate limit errors on all the models if you push hard enough. I always wrap the API with some exponential backoff retries to work around this.
- kcorbitt 3y agoI've thought about building this for a while, glad it's out there! Not only does this guarantee your output is JSON, it lowers your generation cost and latency by filling in many of the repetitive schema tokens without passing them through the LLM. For the very common case of "extracting multiple structured fields from a piece of unstructured text," I believe there's an even stronger optimization possible that would further decrease costs, latency and potentially even improve accuracy. Assuming the fields you want to extract are independent (and they often are), you don't need to generate them all in one go autoregressively. Eg. instead of running the following pseudo-prompt: "Input: 'It's sunny and cold today' Output schema: {"sunny": boolean, "temperature": string}" You could instead run the following two: "Input: 'It's sunny and cold today' Output schema: {"sunny": boolean}" "Input: 'It's sunny and cold today' Output schema: {"temperature": string}" We don't do that today because when done naively it's very inefficient -- you'd be tokenizing, passing to the GPU, and computing the KV cache of the shared part of the prompt twice. But a library with the right abstraction could run the second two queries in a batch in parallel and reuse the same tokenization and KV cache for both of them. It would actually be more efficient than generating both fields in one go, since when you factor out the shared prefixes both the generated text and its context are shorter! I mentioned above that this could also improve accuracy. Of course it doesn't do that by default (except that by excluding all the irrelevant fields it makes self-attention's job easier). But what it does do is give you an independent prompt for each field you're interested in. And so for particularly tricky fields you're trying to extract, you have the flexibility to eg. add several examples to make the generation N-shot.
- bckr 3y agoCan you briefly describe how you got to the point of having this kind of intuition about language models?
- kolinko 3y agoNot op, but I can share my approach - I went line by line by Recmo's Cria: https://github.com/recmo/cria https://github.com/recmo/cria - which is an implementation of Llama in Numpy - so very low level. Took me I think 3-4 days x 10 hours + 1-2 days of reading about Transformers to understand what's going on - but from that you can see how models generate text and have a deep understanding of what's going on.
- tough 3y agoI knew a similar one called GPTyped, just posted it on HN https://news.ycombinator.com/item?id=35793056#35793057 https://news.ycombinator.com/item?id=35793056#35793057
- apalmer 3y agoTrying to understand why this is necessary? LLMs cannot reliably generate valid Jason?
- dwallin 3y agoTwo ways in which it is useful over existing techniques: - It is guaranteed to match your schema - It is much lighter weight
- Der_Einzige 3y agoAlso, it costs tokens to ask a model to do something, and it may choose not to do it re: constraints You can force it by banning the vocabulary which violates a constraint for free.
- tysam_and 3y agoYes, but Mike on the other hand....
- SkyPuncher 3y agoThere are typically two issues: * Occasionally, random junk will get tossed in alongside valid JSON. If you're a human, it's easy to identify and fix. Programmatically, it can be much harder to fix. * Context lengths can create issues.
- rickcarlino 3y agoHas anyone seen a tool like this that uses Node rather than Python? I have this exact problem in a GPT-based web application I am building and have had to resort to some “creative” solutions. At the very least I am glad to see people are tackling this problem.
- msikora 3y agoSame here. Considering switching my project (or at least part of it) to Python. For anything to do with LLMs or ML in general Python has by far the best libraries. JS is probably second, at least for LLM stuff, but it is a distant second place...
- SparkyMcUnicorn 3y agoI've had good luck with Langchain's output parsers[0], but in addition to the format instructions I also append something like "Do not provide any explanations, just the JSON output.", which helps eliminate content being generated outside of the JSON block. [0] https://js.langchain.com/docs/modules/prompts/output_parsers/#structured-output-parser-with-zod-schema https://js.langchain.com/docs/modules/prompts/output_parsers...
- jumploops 3y agoI created a toy[0] in Typescript that maps LLM responses to type-safe output. It uses JSONSchema internally, but I’m thinking of revising it to just use Typescript directly after learning more about the ChatGPT plugin implementation (via their hackathon). [0]https://github.com/jumploops/magic https://github.com/jumploops/magic
- andrewcamel 3y agoSeen a lot of things trying to do this by pressure testing the outputs, but all feel like anti-patterns. This is the first that seems like the "right" way to do it. Better to manage how the model is generating vs creating one more potentially faulty "glue" layer.
- lt 3y agoCan you elaborate about what you mean by pressure testing? Haven't heard this term yet.
- andrewcamel 3y agoMaybe not the right term... Just that a lot of other libs act like guardrails, i.e. let the model generate what it does (in full form text / GPT output), and try to then parse out what you want, error if output doesn't conform to standard format. As opposed to basically only allowing the model to generate into the already-built JSON form fields. Understandable why this guardrails/parsing approach is so popular though... can't do what this library is doing with OpenAI API. Need to be able to manipulate the token generation; otherwise you're forced to take full text output and try to parse it.
- tysam_and 3y agoMathematically it requires less information to impose a certain prior on data in the process of generation than it does to read the data, do error detection and correction according to a prior, and then return it, if I understand correctly. Something always felt incredibly icky to me about any kind of ad-hoc 'fixer' scripts that were part of a pipeline that was fully controlled by a user.
- phh 3y agoI hope that this is new to no-one generating JSON using LLM, because it felt like the first thing you'd do when I implemented that kind of stuff. That being said, it's nice to have that as a library ready-to-go.
- ianbutler 3y agoNice this codifies something similar I've been doing in my prompts! Will be using this instead. What I currently have been doing: The JSON template for your response is provided below. The parts to fill out are capitalized. Please do not modify the template. Please fill in the template with one of the above options for your response. <result> { "rating": "N. RATING", "reason": "REASON" } </result>
- layoric 3y agoI might be reading the code wrong but it looks like it crawls the schema making a generation per primitive type. While that’s a clever way to ensure valid JSON, I don’t know if I’d go as far as to describe it as efficient. Saying that if the model is unable to generate JSON due to its training/fine tuning, this is indeed a clever solution!
- qeternity 3y agoIt’s clever if you’re running your own models, or only paying for tokens generated. Since you’re generating the variable fields anyway, it will actually require fewer forward passes even if broken up in multiple prompts than if you generated the static fields as well. Of course this doesn’t work for OpenAI apis which charge for input context on a per invocation basis.
- visarga 3y agoI wanted to see the opposite - parsing JSON and YAML generated from LLMs. It doesn't happen much with GPT-4 but lesser models might mess up the format and then you can't simply parse it.
- ImaCake 3y agoIt sorta feels like LLMs or some kind of NN should be useful (with training) for parsing malformed jsons. I suspect its a hard problem but honestly it would be such a massive help for those of us dealing with data at work!
- pklee 3y agoThis is pretty cool. I tried with dolly and then I tried with T5-base, both of it did not give me result. It broke for me. Has anyone tried it ?
- motoboi 3y agoI found it rather strange that the new AndrewNG course about prompting, that features an OpenAI employee, says nothing about templated output. To me this is a killer feature of GPT, being able to turn a document into a json or any other template. The kind of prompt is just amazing for GPT (try it with a blog post, document or any other thing): "Analyze this document and transform it into the following format: <title> <summary (text conciseness: 5/10)> <content bullet points (text conciseness 3/10)> <content_item 1> <content_item 2> <content_item N>" Also you can ask the same prompt in a json and GPT will gladly transform a PDF into a JSON.
- ryan-allen 3y agoTemplated output means you could build flexible user interfaces to interact in novel ways beyond a mere text input. What I find absolutely incredible right now is that any novel idea I have about "it would be nice if you could do X" is only taking a few days to reach any mainstream tech news source. I used to think the same thing about RubyGem ideas in the late 2000s, and within 6 months a useful package would come out. I put it down to many people consuming the same information at the same time coming up with the same ideas. It's happening much faster this time. 12 months from now who knows what's going to happen!
- zaptheimpaler 3y agoHow does this work? I guess its different from something like fine-tuning because that wouldn't 100% guarantee the right schema?
- xephoid42 3y agoI actually did this with an silly little app I made that generates fake social media profiles (https://lookface.app https://lookface.app). I gave it a prompt telling it what to generate and an example JSON. As long as you say it must be in JSON I haven't had any problems with it generating bad JSON.
- syntaxing 3y agoIs there a way to do something like this but with fine tuning? For example, I want to train a Lora to become a email spam classifier. I have the training data for the prompt as the email and the response as {Boolean:True/False}?
- galleywest200 3y agoWhen I ask GPT-4 to give me a response in JSON format it just does it, so I assume that yes you can fine-tune an LLM to handle this.
- phate334 3y agoIt may be easier to use with Pydantic.
- drbojingle 3y agoI like the idea of getting ChatGPT to return something easily parse-able by a program. I've been using an XML derivative for that. https://github.com/ColinRyan/Chat-Markup-Language https://github.com/ColinRyan/Chat-Markup-Language Never thought to use json schema. I'll check this out!
- deleted 3y ago[deleted]
- EGreg 3y agoFantastic! This makes it easy to let humans write prompts and generate requests an API can consume.
- Aerbil313 3y agoIs it possible to use this with OpenAI’s models? i.e do they support such in-line token generation?
- yawnxyz 3y agoEfficiency: By generating only the content tokens and filling in the fixed tokens, Jsonformer is more efficient than generating a full JSON string and parsing it. I was excited to try this in Replit... and realized it required pytorch. Ouch. Replit was not happy about that!
- rain1 3y agoThere is no point in constructing a fixed template JSON object like that just to parse it again.
- tanepiper 3y agoNice job - I've tried to massage the outputs to be structured and sometimes it works, but sometimes it fails badly. Having a more specific set of constraints around it will definitely make it more effective.
- ryan-allen 3y agoI've wondered if despite instruction it forgets due to a context limit. There's a lot I still don't understand. I found if it forgot to format in a certain way you could 'remind it', and it would get back on track, but it's an odd way to think of writing software. Kind of like 'turning it off and on again'.
- 91Jacob 3y agoI'm trying to imagine what a possible use case would be for this. Any simple examples?
- kaba0 3y agoAlmost any integration with computer systems would need some structured format, let’s say you want to write an intelligent file manager where you could prompt to “find a file that ends in Final”, and with enough instructions the LLM might give you back something like: { “command”: “find”, “param”: { “type”: “regex”, “value”: “[^.]*Final\.[^.]*” } } which you can actually interpret and execute for the user. But I’m absolutely a novice at anything ML, so take my comments with a grain of salts.
- aligajani 3y agoNice tool, will check it out. I had to go through a painstaking trial and error process to generate valid and deterministic JSON for my AI presentation tool called Slide Genie (https://slidegenie.vercel.app/ https://slidegenie.vercel.app/). The hard part was making it work when temperature > 0.
- pankajdoharey 3y agoIts not very hard through prompting. You can just ask the LLM to generate on these parameters. I did this exact same thing and never wrote any code for it.
- diginova 3y agoSomething like this should be integrated with library like https://fakerjs.dev/ https://fakerjs.dev/ With LLM or in general AI based generation of the fake data it can be more diverse and generalized for lot's more applications and help developers My bad if I am unaware of faker having AI based generation already, but afaik it does not have right now