5 ms·
I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten. But it's still probabilistic, and nine
by panarky 3y ago
I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten.
But it's still probabilistic, and nine times out of ten isn't good enough.
Occasionally it will hallucinate responses like this:
{"key1": "value1", "key2": "value2" for i in range(n)}
Re-prompting with the parsing error message is usually enough to get it on the second try.
But escaping double-quotes and newline characters is less reliable. Even after giving it multiple examples, it correctly escapes only about half the time.
Re-prompting for escaping errors still yields a ~50% success rate.
- keiferwiseman 3y agoIt took some iterations but I've managed to get the OpenAI API to give me valid JSON 100% of the time now(based on my testing). I think I put in the prompt to never use newlines because it was causing issues lol.
- padolsey 3y agoI've had more luck with getting it to output XML as (1) You can imbue XML with actual language/meaning (which LLMs adore) and (2) parsers can be made to be more forgiving. I get why people want to make JSON, but to me it's a bit like trying to get a cat to swim - you might eventually succeed, but it's not their natural inclination.
- gowld 3y agoHow do you imbue XML with meaning?
- padolsey 3y agoXML Elements themselves: their naming, their attributes, comments, indentation. There's more opportunity at every level of the hierarchy to demarkate and establish meaning. Having closing-tags as well, I've found, is a massive boon; LLMs can better understand what "finishing" looks like if its delimited in a semantic way - with a name.
- BoorishBears 3y agoSame works for JSON. Naming JSON keys works for adjusting what the output is nicely, and you can comment in your definitions (by defining them in a JSON Schema, or inserting placeholder text like `"someKeyWithClarifyingDetails": <some detailed instruction>`) I'm actually partial to CSV these days though, it can really cut down on response times just not needing to return all the extra tokens for JSON/XML delimiters
- padolsey 3y agoOstenibly yeh JSON should be able to encapsulate mose of that semantic stuff but having replaced an XML schema in the system prompt with gpt's function-calling API I've been very umimpressed. It feels much less capable. I would have to provide a lot more clarifying prompts to make it more capable. I think I will, for now, bias to using schemas that are closest to prose.
- DonHopkins 3y agoYikes. This makes me think that JSON's stubborn mistake of not allowing comments is yet another "Billion-Dollar Mistake", since it's way too late to just change the standard to allow comments, update all the JSON content on the internet to use comments, and retrain all the LLMs to understand comments. Great point about CSVs! But using placeholder keys for JSON comments in untenable, and using schema instead of inline comments is clumsy and indirect. Of course JSON schema are quite useful in certain situations, but LLMs would get a lot more meaning out of casual common JSON if it just allowed comments, and it would also greatly benefit humans. Between JavaScript's and JSON's mistakes, that's at least <DoctorEvilVoice>THREE BILLION DOLLARS!!!</DoctorEvilVoice> ;) https://en.wikipedia.org/wiki/Tony_Hoare#Research_and_career https://en.wikipedia.org/wiki/Tony_Hoare#Research_and_career >Speaking at a software conference in 2009, Tony Hoare apologized for inventing the null reference: >"I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years." -Tony Hoare https://news.ycombinator.com/item?id=19568378 https://news.ycombinator.com/item?id=19568378 >"My favorite is always the Billion-Dollar Mistake of having null in the language. And since JavaScript has both null and undefined, it's the Two-Billion-Dollar Mistake." -Anders Hejlsberg >"It is by far the most problematic part of language design. And it's a single value that -- ha ha ha ha -- that if only that wasn't there, imagine all the problems we wouldn't have, right? If type systems were designed that way. And some type systems are, and some type systems are getting there, but boy, trying to retrofit that on top of a type system that has null in the first place is quite an undertaking." -Anders Hejlsberg
- prempv 3y agoI've had the same experience as well. I suspect if it's due to large presence of HTML in the training data as part of codebases and online content
- MuffinFlavored 3y agoI wonder if the next iteration of OpenAI features is something like: right now you can inject prompts that the LLM takes into consideration before the output I wonder if you can make it have a "post" generation function that says like "keep re-trying in a loop (aka hallucinating with randomness) until the output message passes XYZ format/checks/scoring"
- padjo 3y agoIt’s starting to feel like LLMs are to “classical” software engineering what quantum physics was to classical physics
- catlifeonmars 3y agoHow so? I’m not quite following the analogy.
- antonvs 3y agoJust guessing what was meant, but quantum physics in some sense tries all possible paths before an outcome is selected. The problem with that is that without a quantum computer, or without some sort of filtering, that process can take up to infinite time.
- padjo 3y agoOh it was just a glib way of moaning about non-determinism making its way into software engineering. Much like how physicists had to make peace with the probabilistic nature of quantum physics.
- kristjansson 3y agoWhy wait for OpenAI?
- thumbsup-_- 3y agoYeah same thing. I have done the same with GPT-3.5. Simply ask it to output using provided schema only and give a few examples. Always outputs in provided json format
- orasis 3y agoWhat about using ChatGPT’s new function calling mechanism?
- superasn 3y agoThat returns broken JSON a lot of the times too
- phillipcarter 3y agoThis is what we do, but for GPT-3.5. And it doesn't need to be system messages either. We even have it emitting only JSON in a specific structure (except for when it fails to produce an output altogether). This is without the function calling model.
- nextaccountic 3y agoWhat about reprompting with a different temperature value? If this works, how to select the optimal value? Maybe you can train a model that can excel at the task of querying gpt4 for valid jsons
- simonw 3y agoThat re-prompting on error trick is what this new Microsoft library does, too: https://github.com/microsoft/TypeChat https://github.com/microsoft/TypeChat Here's their prompt for that: https://github.com/microsoft/TypeChat/blob/c45460f4030938da39629d882c3e0e2c718f8680/src/program.ts#L212-L216 https://github.com/microsoft/TypeChat/blob/c45460f4030938da3... I think the approach using grammars (seen here, but also in things like https://github.com/ggerganov/llama.cpp/pull/1773 https://github.com/ggerganov/llama.cpp/pull/1773 ) is a much more elegant solution.
- creatonez 3y agoA "repair prompt" instead of rewinding and starting back from the error seems like the wrong choice, and might only make sense with how payment for OpenAI API usage currently works.
- caesil 3y agoWith ChatGPT function calling I get valid JSON 100% of the time from GPT-4 unless I have made some error in prompting. The chief error is not providing escape hatches. LLMs look for a right answer. If you are feeding it some texts and asking it to return structured data about the texts, but then one of the texts is blank, it will be difficult to determine a right answer, so you get hallucinations. The solution is an escape hatch where one of the arguments is a `textIsMissing` boolean or something. As long as you've accounted for these failure modes, it works flawlessly.
- reissbaker 3y agoGPT-4 is amazing, but the upside of smaller models is much lower cost. I get basically 100% accuracy on JSON modeling with GPT-4 with function calling too, but I will say that gpt-3.5-turbo with function calling is somewhat less accurate — it usually generates valid JSON in terms of JSON.parse not exploding, but not necessarily JSON following the schema I passed in (although it's surprisingly good, maybe ~90% accurate?). I use 3.5-turbo a decent amount in API calls because it's just a lot cheaper, and performs well enough even if it's not gpt-4 level. I haven't gotten a chance to earnestly use the smaller Llama models yet in more than small prototypes (although I'm building a 4090-based system to learn more about finetuning them), but the little amount of experimenting I've done with them makes me think they need a decent amount of help with generating consistently-valid JSON matching some schema out of the box. This is a pretty neat tool to use for them, since it doesn't require finetuning runs, it just masks logits.
- BoorishBears 3y agoclaude-1.2-instant came out last week and is doing extremely well at following schemas. I'd say it's reached 3.5 turbo with the format following skills of GPT-4, which is powerful once you give it chain-of-thought
- selcuka 3y agoThe premise of function calling is great, but in my experience (at least on GPT-3.5, haven't tried it with GPT-4 yet) it seems to generate wildly different, and less useful results, for the same prompt.
- andreygrehov 3y agoMeh... I asked GPT4 to return a sample PHP code inside of a random JSON. It failed the JSON linter from the very first try. I actually couldn't pass the validation despite many retries, eg follow up corrections. Not a single time it generated a 100% valid JSON, I eventually gave up.
- ipaddr 3y agoThis worked with chatGPT: create a sample hello world in php store that code in a json[object code: { "php_code": "<?php echo 'Hello, World!'; ?>" }
- adamrezich 3y agoif you think that's bad, try to get it to generate Inform 7 games—Inform's natural-English-ish syntax completely throws all LLMs for a loop, consistently. it generates code that looks possibly correct (to an Inform newbie at least), but fails to compile far more often than not. I find this super interesting.
- karmasimida 3y agoI see grammar constrained generation for 2 major advantages: 1. It consumes fewer tokens, no need to add too many examples into the prompt. 2. It suffers less from the forgetting issue. Another minor advantage is you can control precisely where your desired output to begin with. But overall, those are nice perks not too substantial IMO.
- msp26 3y ago>I can make GPT4 return valid JSON simply by providing examples in the system message. This works nine times out of ten But you can do both. For my current use case of extracting information from articles, I have a json schema + one/two example articles along with their correct answers. This increases token costs but 3.5 is so cheap that it doesn't matter and for 4 you can use batching to decrease token cost per article.
- vsrinivasan 3y agoCan you please explain what is batching ? any pointers?