9 ms·
Show HN: GPT-JSON – Structured and typehinted GPT responses in Python
Hey HN, I've been using GPT a lot lately in some side projects around data generation and benchmarking. During the course of prompt tuning I ended up with a pretty complicated request: the value that I was looking for, an explanation, a criticism, etc. JSON was the most natural output format for this but results would often be broken, have wrong types, or contain missing fields.
There's been some positive movement in this space, like with jsonformer (https://github.com/1rgs/jsonformer https://github.com/1rgs/jsonformer) the other day. But nothing that was plug and play with GPT.
This library consolidates the separate logic that I built across 5 different projects. It lets you prompt the model for how it should return fields, inject variable prompts, handle common formatting errors, then cast to pydantic when you're done for typehinting and validation in your IDE. If you're able to play around with it, let me know what you think.
- knowsuchagency 3y agoNice project! I took some inspiration from this as well as https://github.com/jiggy-ai/pydantic-chatcompletion/blob/master/pydantic_chatcompletion/__init__.py https://github.com/jiggy-ai/pydantic-chatcompletion/blob/mas... to create the following: https://github.com/knowsuchagency/struct-gpt https://github.com/knowsuchagency/struct-gpt I tried to make the API as intuitive as possible and added the ability to provide examples to improve the reliability and quality of the LLM's output.
- deleted 3y ago[deleted]
- m3kw9 3y agoWasn’t there a yesterday proj that asks gpt to output non json but then it turns it into json after? Thus making it less prone to json errors
- m3kw9 3y agoAh, this one: https://github.com/1rgs/jsonformer https://github.com/1rgs/jsonformer
- yawnxyz 3y agoThat one is robust but requires pytorch and huggingface and all kinds of things. I think most of us want something that's reasonable few lines of code and can run in places like serverless and replit.
- crazyedgar 3y agoI like the idea, but I think a library that focuses on producing requests and parsing responses according to schema is better. Sending requests to the server is orthogonal to the purpose. What we've found useful in practice in dealing with similar problems: - Use json5 instead of json when parsing. It allows trailing commas. - Don't let it respond in true/false. Instead, ask it for a short sentence explaining whether it is true or false. Afterwards, use a small embedding model such as sbert to extract true/false from the sentence. We've found that GPT is able to reason better in this case, and it is much more robust. - For numerical scores, do a similar thing by asking GPT for a description, then with the small embedding model write a few examples matching your score scale, and for each response use the score of the best matched example. If you let GPT give you scores directly without explanation, 20% of the time it will give you nonsense.
- icyfox 3y agoThanks for the thoughts! I've deployed a few meta models that act like you're describing for second stage predictions, but for fuzzy task definitions have actually seen similar luck with having GPT explicitly explain its rational and then force it to choose a true/false rating. My payloads often end up looking like: class Payload: reasoning: str = Field(description="Why this value might be true or false), answer: bool Since it's autoregressive I imagine the schema helps to define the universe of what it's supposed to do, then the decoder attention when it's filling the `answer` can look back on the reasoning and weigh the sentiment internally. I imagine the accuracy specifics depend a lot on the end deployment here.
- iamflimflam1 3y agoThere’s some really good info along thff we same lines in this course https://learn.deeplearning.ai/chatgpt-prompt-eng https://learn.deeplearning.ai/chatgpt-prompt-eng
- syntaxing 3y agoIs it possible to give an example what a small embedded model would look like? Curious how to make something like this!
- wskish 3y agoHere's 40 lines of python code that I've found to be unreasonably effective at accomplishing something similar: https://github.com/jiggy-ai/pydantic-chatcompletion/blob/master/pydantic_chatcompletion/__init__.py https://github.com/jiggy-ai/pydantic-chatcompletion/blob/mas...
- johntash 3y agoThanks! I have something similar that I've been using and struggling to keep consistent. I like that this is a relatively small package, I'll probably end up using it to play around with.
- SeanA208 3y agoVery cool. Out of curiosity, for the retries why are the errors appended as system messages as opposed to appending to the user message? And in either case, would it help to prepend the error with something like “Be sure to avoid outputting something that would cause this error:”?
- wskish 3y agobased on my experience with GPT-4 in coding tasks it responds extremely well to just appended raw error output text without additional explanation.
- TeeWEE 3y agoI dont see how these infinite loops are a good idea... You never sure if you actually getting a good result ? What is the failure rate?
- ZephyrBlu 3y agoUm, what? This is a standard retry loop. It's just generally good practice.
- slig 3y agoThanks, I'm also doing something similar and figured out that asking a question with examples as `user`, adding the perfect response as `assistant`, replying with `Perfect. Now do this {}` works really well and cuts of a lot of trial/error.
- armcat 3y agoHave you considered Guardrails, https://shreyar.github.io/guardrails/ https://shreyar.github.io/guardrails/, it’s like Pydantic but for LLMs?
- CupofChineseTea 3y agoI happen to have a very similar idea recently and created this GPT-logic package for node. It basically transforms GPT generated results into JS data types. Check it out if you are interested. https://github.com/wayneshn/gpt-logic https://github.com/wayneshn/gpt-logic
- andrewstuart 3y agoTyped ChatGPT. Someone had to come along and spoil the fun. Next it'll be ChatGPT/XML.
- btbuildem 3y agoSome folks are more comfortable swaddled in well-defined boxes.
- KingOfCoders 3y agoIdiot question, I'd like to register and pay for the ChatGPT API but can't find the link to get an API token :-)
- vmarkham 3y agoIf you already have an account with them: https://platform.openai.com/account/api-keys https://platform.openai.com/account/api-keys.
- KingOfCoders 3y agoThanks!
- toteno 3y agoYou can get API keys from the page https://platform.openai.com/account/api-keys https://platform.openai.com/account/api-keys (which you can find open by clicking on your avatar and selecting "View API keys").
- KingOfCoders 3y agoThanks!
- lharries 3y agoLangchain also has a built in Pydantic output parser https://python.langchain.com/en/latest/modules/prompts/output_parsers/examples/pydantic.html https://python.langchain.com/en/latest/modules/prompts/outpu... However I've found it to be generally unreliable and adds a lot of text to each call. I suspect this can be improved by: - Only writing the parts of the spec which are needed, rather than the full JSON schema spec - Including relevant examples rather than arbitrary JSON schema examples
- olup 3y agoI built zod-chatgpt https://github.com/olup/zod-chatgpt https://github.com/olup/zod-chatgpt a couple of weeks ago to serve same purpose in typescript
- kleneway1 3y agoYes, this is a very similar technique that I have been using and it works great. One suggestion of something that worked well for me was to use safeParse instead of parse. Then if it doesn’t pass validation, you can retry by passing in the JSON object and the validation error messages. You could also use tricks like starting with a smaller model, then try larger models if you hit a validation failure. Not a great approach for real-time chat but very useful for when you need high-quality results.
- jumploops 3y agoI built a toy[0] for Typescript that works similarly to this. It takes the expected return type of a function, translates it into a JSON Schema, queries OpenAI, validates the response, and then magically returns the response as typed output. [0]https://github.com/jumploops/magic https://github.com/jumploops/magic
- teleforce 3y agoI asked ChatGPT (GPT-4) about the idea of integrating GPT-JSON with CUE language (https://cuelang.org/ https://cuelang.org/) for potential improvements and the answers are quite impressive: "Yes, integrating the CUE language with GPT-JSON can potentially lead to improvements in its capability. CUE, short for Configuration, Unification, and Environment, is an open-source data constraint language designed to simplify the definition, generation, and validation of data in structured formats, like JSON, YAML, and others. Integrating CUE with GPT-JSON can bring several benefits: 1) Validation: By using CUE, you can define schemas for the input and output JSON, which helps in validating the data structure and ensuring it adheres to the expected format. This can help reduce errors and improve the quality of the interaction with the GPT model. 2) Simplification: CUE can help simplify the process of defining and generating JSON inputs and outputs for GPT-JSON. By providing a more concise and expressive syntax, CUE can make it easier to craft structured inputs and outputs, which can be especially helpful when working with complex data structures. 3) Interoperability: Integrating CUE can improve the interoperability between GPT-JSON and other systems or applications. With CUE's ability to work with multiple data formats (such as JSON, YAML, or HCL), it becomes easier to integrate GPT-JSON with various tools and platforms. To integrate CUE with GPT-JSON, you would need to follow these general steps: 1) Define CUE schemas for the input and output JSON structures used by GPT-JSON. 2) Use the CUE language to validate and generate input JSON for GPT-JSON. 3) Process the output JSON generated by GPT-JSON using CUE to ensure it adheres to the defined schema and transform it as needed. Integrating CUE with GPT-JSON can improve the overall robustness and ease of use of the library, making it more accessible and useful for a wider range of applications."
- TeMPOraL 3y agoMaybe not the best thread to ask, but - is there an ELI5 explanation of what exactly CUE is and what is it for? I've landed on that website several times in the last two years, and I could never make heads or tails of it. That the name is just a mix of random, unrelated verbs doesn't help.
- deleted 3y ago[deleted]
- BiteCode_dev 3y agoYou can use simpler types. Instead of : messages=[ GPTMessage( role=GPTMessageRole.SYSTEM, content=SYSTEM_PROMPT, ), GPTMessage( role=GPTMessageRole.USER, content="Text: I love this product. It's the best thing ever!", ) ] Try: messages=( ("system", SYSTEM_PROMPT), ("user", "Text: I love this product. It's the best thing ever!") ) Or: messages=( SystemMsg(SYSTEM_PROMPT), UserMsg("Text: I love this product. It's the best thing ever!") ) This is still Python, not Java.
- rain1 3y agopython newbie here, why did messages change from [ to (
- BiteCode_dev 3y agoDoesn't matter much in this particular case, one is a tuple, the other is a list, both can be read with a for loop. The important part is that the way messages types are defined is very heavy, and you can get them to be lighter without sacrificing type safety.
- verdverm 3y agoWe've been using a variation like this to great effect: TYPES: [string, int, bool, float, uuid, datetime, email, url] RELATIONS: [belongs-to, has-one, has-many, many-to-many] SCHEMA: """ { "Datamodel": { "Name": "<application-name>", "Models": { "<model-name>": { "<field-name>": "<field-type>", "<field-name>": "<field-type>", "$relations": { "<relation-name>": { "name": "<relation-name>", "type": "<relation-type>", "model": "<model-name>" }, "<relation-name>": { ... } } }, "<model-name>": { ... } } } } """ To let users write things like this: https://twitter.com/verdverm/status/1652504163635347456 https://twitter.com/verdverm/status/1652504163635347456
- AmazingTurtle 3y agoWhat about few shot learning, e.g. injecting 1 or 2 examples of JSON in the prompt? That should be fine as well
- alalalal 3y agoI literally tried that yesterday. You will be introducing bias. It works well until you infer with something very similar to one of the examples you used in the prompt but not exactly the same then it will always return the json you used in your example.
- senko 3y agoI wait for the day someone fine-tunes an open model to always respond in JSON. I suspect it won't be long now.
- fswd 3y agoHa, I just spent two hours hacking something up similar and I eventually just sent a system role with zod schema in the content
- reissbaker 3y agoHey, this is really neat! I've taken a very similar approach in TypeScript. +1 to the sibling comment that recommended parsing with json5 (but don't tell the AI you're doing that, it's a waste of context space and it might get more confused anyway). I've had luck doing chain-of-thought prompting in the JSON payload as you've described, too. Cheers, really validating to see someone taking a similar approach.
- robflaherty 3y agoIs there a reason we won’t have LLMs that only speak in JSON? These JSON hacks are clever and cool but feel like they’ll be obsolete in 6 weeks.
- verdverm 3y agoA lot of their value is turning unstructured text or user intent into the structured data. Even if the primary input is JSON, you want to explain what to do with it in words, and then get JSON back. Why do you think they will be obsolete in 6 weeks?
- icyfox 3y agoFingers crossed OpenAI / Anthropic / etc do this! Would make working with these APIs for prediction projects that much easier. Technically speaking it's pretty to force the model into an valid JSON-schema if you have access to the inference autoregressive loop and the logit activations. You can either force known areas of the template to a prefixed template, or fill in the basic JSON wrapper and force the model to choose arbitrary keys and values. Imagine it might come down to how much of their usage is on generating text vs generating structured prediction payloads.
- valyagolev 3y agothe dirtyjson library has been doing this work for me quite well
- Loeffelmann 3y agoI Found that LLMs are pretty good with TOML. The multiline strings are also a real bonus. One thing I thought that was interesting is that sometimes the LLM will mistake the triple quoutes for the backticks so it will output something like this ``` [TOML] key=""" value """ """
- verdverm 3y agoYou can instruct the LLM on what the outer markers are. > Generate some TOML for me, surround the output with a pair of triple equal signs (===).
- mmaia 3y agoI'm having success with simple YAML schema. One thing that's very helpful for the prompt is to include "description" and "example": - column name: salary_max format: number example: 150,000 description: Salary Maximum - column name: keywords format: string example: engineer, python, docker, remote description: Relevant Keywords (Comma separated keywords used for filtering and matching jobs to candidates)
- vlaaad 3y agoNice! Another reason to prefer YAML is token count — YAML is 3x cheaper than JSON: https://twitter.com/v1aaad/status/1643889605538635782 https://twitter.com/v1aaad/status/1643889605538635782
- icyfox 3y agoGood suggestion mmaia - I'm opening a new issue to keep track of the different output schemas that are being suggested. One thing that originally worried me a bit with yaml was its relative reliance on space-based formatting to drive meaning. GPT generally tokenizes newlines/spaces but a lot of preprepared datasets strip these out, so I preferred the explicitness of a json that's idempotent to spaces. Have you tried using this approach for non-tabular or nested data like lists or dictionaries?
- deleted 3y ago[deleted]
- btbuildem 3y agoI've been interfacing with GPT programmatically for a little while now, leveraging it's "soft and fuzzy" interface to produce hard / machine-readable results. JSON was the format that felt best-suited for the job. I see a ton of code in this project, and I don't know what most of it does. As far as GPT troubles with JSON, I'll add a couple: sometimes it likes to throw comments in there as if it was JS. And sometimes it'll triple-quote the JSON string as if it was Python. My approach to solve these problems was via prompt engineering - using the system message part of the API call. Asking it to "return valid json, do not wrap it in text, do not preface it with text, do not include follow-up explanations, make sure it's valid json, do not include comments" - seems to work 99% of the time. For the remainder, a try-and-catch block with some fallback code that "extracts" json (via dumb REs) from whatever text was returned. Hasn't failed yet. It's fascinating to watch the new paradigm arrive, and people using old habits to deal with it. This entire project is kind of pointless, you can just ask GPT to return the right kind of thing.
- knowsuchagency 3y agoI'm curious what you might think of https://github.com/knowsuchagency/struct-gpt https://github.com/knowsuchagency/struct-gpt
- olabyne 3y agoBut the problem is the 99%, no ?
- verdverm 3y agoIt works fine 99% of the time by just using a small amount of extra instruction in the actual prompt. The method GP describes works in any language with just the basic building blocks of http requests, regexp, and a json decoder. Why do we need a library for this?
- icyfox 3y agoYou might not! Depends on what you're looking for. I've been finding this library most helpful in places where I have a lot of GPT calls in pipelines, so having typehinted schema return values / some built in error correction / variable injection / establishing standards for the IO of prompt schema is the most useful. So IMO I see its main use as a good standard set of operations that work pretty well out of the box and that allow you to hack around them with decent flexibility.
- killthebuddha 3y agoIn the last 24 hours I've seen a bunch of projects doing LLM -> JSON. I think we want to be focusing on markdown instead. An intuition I have developed is that an ideal prompt has a very clear narrative structure and very tight "semantic locality" (the instruction is at the end, the most salient data is close to the instruction, etc). JSON is admittedly way easier to work with up front, but markdown seems to be a more scaleable choice. Of course, this is all very much an opinion and highly anecdotal at the moment.
- pigcat 3y agoSince others are sharing their prompt-only solutions to get JSON, I'll share what I've been using. Has been working reliably: "Do not include any explanations, only provide a RFC8259 compliant JSON response following this format without deviation. { "author": "string describing the author full name", "year": "number describing the year the book was written", "isFiction": "boolean describing if the book is a work of fiction" ... } "