27 ms·

Structured Outputs in the API

764 points by davidbarker 2y ago

lmshn 2y ago
[flagged]
- deleted 2y ago
  [deleted]
gamegoblin 2y ago
I'm glad they gave up on their "fine-tuning is all you need" approach to structured output. It's possible fine-tuning will work in the long term, but in the short-term, people are trying to build things, and fine-tuning wasn't cutting it. Surprised it took them so long — llama.cpp got this feature 1.5 years ago (actually an even more general version of it that allows the user to provide any context free grammar, not just JSON schema)
- chhabraamit 2y ago
  How does llama.cpp’s grammar adherence work? Does it keep validating the predicted tokens and backtrack when it’s not valid?
  - gamegoblin 2y ago
    It's essentially an Earley Parser[0]. It maintains a set of all possible currently valid parses, and zeroes out the probability of any token that isn't valid in at least 1 of the current potential parse trees. There are contrived grammars you can give it that will make it use exponential memory, but in practice most real-world grammars aren't like this. [0] https://en.wikipedia.org/wiki/Earley_parser https://en.wikipedia.org/wiki/Earley_parser
    - jules 2y ago
      Earley parsers should not need more than O(n^2) memory.
    - orlp 2y ago
      You don't even need that for JSON. JSON can be expressed using a LR(1) grammar, so you can do it in linear time and space.
      - gamegoblin 2y ago
        Yes, the llama.cpp work supports arbitrary CFGs, not just JSON
- tcdent 2y ago
  GPT is still a language model, so at some point it's still just tokens. Is this just a schema validation layer on their end to avoid the round trip (and cost) of repeating the call?
  - gamegoblin 2y ago
    Language models like GPT output a large vector of probabilities for the next token. Then a sampler decides which of those tokens to pick. The simplest algorithm for getting good quality output is to just always pick the highest probability token. If you want more creativity, maybe you pick randomly among the top 5 highest probability tokens or something. There are a lot of methods. All that grammar-constrained decoding does is zero out the probability of any token that would violate the grammar.
    - nickreese 2y ago
      Thank you for this explanation. A few things just clicked for me.
- BoorishBears 2y ago
  I was surprised it took so long until I reached this line: > The model can fail to follow the schema if the model chooses to refuse an unsafe request. If it chooses to refuse, the return message will have the refusal boolean set to true to indicate this. I'm not sure how they implemented that, maybe they've figured out a way to give the grammar a token or set of tokens that are always valid mid generation and indicate the model would rather not continue generating. Right now JSON generation is one of the most reliable ways to get around refusals, and they managed not to introduce that weakness into their model
- Der_Einzige 2y ago
  For many things, fine-tuning as we know of it will NEVER fully solve it, there's no hope. Even fine-tuning a model to not use the letter "e" to an overwhelming degree doesn't entirely prevent it, only reduces its chances to increasingly small amounts. Shamesless self plug, and from before the ChatGPT era too! https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai-1 https://paperswithcode.com/paper/most-language-models-can-be...
leetharris 2y ago
At the bottom: >Acknowledgements Structured Outputs takes inspiration from excellent work from the open source community: namely, the outlines, jsonformer, instructor, guidance, and lark libraries. It is cool to see them acknowledge this, but it's also lame for a company named "OpenAI" to acknowledge getting their ideas from open source, then contributing absolutely NOTHING back to open source with their own implementation.
- warkdarrior 2y ago
  > it's also lame for a company named "OpenAI" to acknowledge getting their ideas from open source, then contributing absolutely NOTHING back to open source with their own implementation Maybe those projects were used as-is by OpenAI, so there was nothing new to contribute.
  - reustle 2y ago
    I think they may be alluding to sponsorships as well as code contributions. i.e. https://github.com/sponsors/jxnl https://github.com/sponsors/jxnl
- spencerchubb 2y ago
  Is offering gpt4o for free through chatgpt not enough of a contribution? They didn't release source code, but they made a product free to use
  - mplewis 2y ago
    No. If it were free you'd be able to use it as a programming API. It's not free and it's not unlimited - it's a time-limited marketing tool.
    - spencerchubb 2y ago
      How are you defining the word free?
  - talldayo 2y ago
deleted 2y ago
[deleted]
jjcm 2y ago
Interesting tidbit at the very end that's worth noting for anyone using the API today: > By switching to the new gpt-4o-2024-08-06, developers save 50% on inputs ($2.50/1M input tokens) and 33% on outputs ($10.00/1M output tokens) compared to gpt-4o-2024-05-13.
- scrollop 2y ago
  From what I've learned from OpenAI, the "latest" "cheaper" model will perform worse than the previous model on various tasks (esp reasoning).
  - ralusek 2y ago
    I don't think it's been well enough acknowledged that all of the shortcuts LLMs have been taking with ways of attempting to compress/refine/index the attention mechanism seem to result in dumber models. GPT 4 Turbo was more like GPT 3.9, and GPT 4o is more like GPT 3.7.
    - scrollop 2y ago
      Some commenters acknowledge it - and quantify it: https://www.youtube.com/watch?v=Tf1nooXtUHE&t=689s https://www.youtube.com/watch?v=Tf1nooXtUHE&t=689s
    - Der_Einzige 2y ago
      They try to gaslight us and tell us this isn't true because of benchmarks, as though anyone has done anything but do the latent space exploration equivalent of throwing darts at the ocean from space. It's taken years to get even preliminary reliable decision boundary examples from LLMs because doing so is expensive.
    - alach11 2y ago
      Do you have benchmarks demonstrating this? In my own personal/team benchmarks, I've seen 4o consistently outperform the original gpt-4.
deleted 2y ago
[deleted]
nerdjon 2y ago
I have a bad feeling that this is just going to introduce more shovelware apps that try to shove AI use in without really understanding what they are going to get back. Yay I can now ensure the json object will look how I want, but lets completely disregard any concern of wether or not the data returned is valuable. I don't understand why we are already treating these systems as general purpose AI when they are not. (Ok I do understand it, but it is frustrating). The example given of "look up all my orders in may of last year that were fulfilled but not delivered on time". First I have found these models incredibly dumb when it comes to handling time. But even beyond that, if you really are going to do this. I really hope you double check the data before presenting the data you get back as true. Worse that is just double checking what it gives back to you is accurate, not checking that it isn't telling you about something. Every time I try to experiment with supplying data and asking for data back, they fall flat on their face before we even get to the json being formatted properly. That was not the issue that needed to be solved yet when it still fundamentally messes up the data. Often just returning wrong information. Sometimes it will be right though, but that is the problem. It may luck out and be right enough times that you gain confidence in it and stop double checking what it is giving back to you. I guarantee you someone is going to have a discussion about using this, feeding it data, and then storing the response in a database.
titzer 2y ago
It's so wild that the bar for AI performance is both absurdly high and absurdly low at the same time. To specify an output format (language or grammar) for solving a computational problem is one of the oldest exercises around. On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification. It's weird reading the kind of self-congratulating blogpost about this, like OpenAI has just discovered flint knives. On the other hand, a computer system can process natural language with extremely ambiguous, open-ended problems, compute solutions to said problems, even correct its own mistakes--and then it can format the output correctly. And then on yet another hand, it only took about 10^25 floating point operations (yeah, just ten million trillion trillion, right!?) to get this outcome.
- codingwagie 2y ago
  I think it will take a long time for the world at large to realize and then operationalize the potential of this "mundane" technology. It is revolutionary, and also sitting in plain sight. Such a huge technological shift that was considered decades out only a few years ago
  - ben_w 2y ago
    Although I am an optimist* about what this can do, I am very much aware — from personal experience — how easy it is to see more than is really there. The realisation of the tech might be fantastic new things… or it might be that people like me are Clever Hans-ing the models. * that may be the wrong word; "strong capabilities" is what I think is present, those can be used for ill effects which is pessimistic.
- srcreigh 2y ago
  If I wanted to be a silly pedant, I’d say that Turing machines are language specifications and thus it’s theoretically impossible for an LLM or any program to validate output formats in general.
  - deleted 2y ago
    [deleted]
wewtyflakes 2y ago
Why would someone want `strict` to be anything other than `true`?
- davidkunz 2y ago
  Maybe if you can't precisely model your structure with (OpenAI's subset of) JSON schema.
- ComputerGuru 2y ago
  There are many reasons, though I am not sure which they had in mind. One thing is that LLMs in general tend to do better when they can be more verbose in their output and sort of “think aloud” to reach an answer. Insisting on strict output format would rob it of the benefits (because it doesn’t just not emit but completely skips those stages, or else you’d be paying for those elided output tokens).
  - wewtyflakes 2y ago
    But then why would someone specify that the response has to be in a given JSON schema (by presence of the schema itself), but then also not care if it is actually using that schema (by specifying `strict` as `false`)? That is the use-case I can't wrap my head around.
    - ComputerGuru 2y ago
      Ah, good point. It honestly feels/looks like a hack to flip a switch on their end and they’re just bubbling it back up to the user.
- tedsanders 2y ago
  We didn't cover this in the announcement post, but there are a few reasons: - The first request with each JSON schema will be slow, as we need to preprocess the JSON schema into a context-free grammar. If you don't want that latency hit (e.g., you're prototyping, or have a use case that uses variable one-off schemas), then you might prefer "strict": false - You might have a schema that isn't covered by our subset of JSON schema. (To keep performance fast, we don't support some more complex/long-tail features.) - In JSON mode and Structured Outputs, failures are rarer but more catastrophic. If the model gets too confused, it can get stuck in loops where it just prints technically valid output forever without ever closing the object. In these cases, you can end up waiting a minute for the request to hit the max_token limit, and you also have to pay for all those useless tokens. So if you have a really tricky schema, and you'd rather get frequent failures back quickly instead of infrequent failures back slowly, you might also want "strict": false But in 99% of cases, you'll want "strict": true.
_vaporwave_ 2y ago
Anyone else catch this reference in one of the examples? > 9.11 and 9.9 -- which is bigger https://community.openai.com/t/why-9-11-is-larger-than-9-9-incredible/869824 https://community.openai.com/t/why-9-11-is-larger-than-9-9-i...
- jodacola 2y ago
  Amusingly, I immediately thought 9.11 - but in the context of a newer version of software. Ever have those moments where you're so deep in context of some ecosystem that you skip right past the basics, like 9.9 being a larger number than 9.11?
elpocko 2y ago
Doesn't the BNF grammar approach in llama.cpp solve this issue in a generic way that should work with any model? Why wouldn't they use that?
- ejones 2y ago
  Similar approach to llama.cpp under the hood - they convert the schema to a grammar. Llama.cpp's implementation was specific to the ggml stack, but what they've built sounds similar to Outlines, which they acknowledged.
- HanClinto 2y ago
  llama.cpp's GBNF grammar is generic, and indeed works with any model. I can't speak for other approaches, but -- while llama.cpp's implementation is nice in that it always generates valid grammars token-by-token (and doesn't require any backtracking), it is tough in that -- in the case of ambiguous grammars (where we're not always sure where we're at in the grammar until it finishes generating), then it keeps all valid parsing option stacks in memory at the same time. This is good for the no-backtracking case, but it adds a (sometimes significant) cost in terms of being rather "explosive" in the memory usage (especially if one uses a particularly large or poorly-formed grammar). Creating a grammar that is openly hostile and crashes the inference server is not difficult. People have done a lot of work to try and address some of the more egregious cases, but the memory load can be significant. One example of memory optimization: https://github.com/ggerganov/llama.cpp/pull/6616 https://github.com/ggerganov/llama.cpp/pull/6616 I'm not entirely sure what other options there are for approaches to take, but I'd be curious to learn how other libraries (Outlines, jsonformer) handle syntax validation.
behnamoh 2y ago
Well, there goes one of the big advantages of open-source models... For a long time, I was relying on such guaranteed structured outputs as a "secret sauce" that only works using llama.cpp's GBNF grammars. Now OpenAI literally introduced the same concept but a bit more accessible (since you create a JSON and they convert it to a grammar). Those of you who have used GBNF, do you think it still has any advantage over what OpenAI just announced?
- ejones 2y ago
  FWIW, llama.cpp has always had a JSON schema -> GBNF converter, although it launched as a companion script. Now I think it's more integrated in the CLI and server. But yeah I mean, GBNF or other structured output solutions would of course allow you to supply formats other than JSON schema. It sounds conceivable though that OpenAI could expose the grammars directly in the future, though.
  - behnamoh 2y ago
    I think for certain tasks it's still easier to write the grammar directly. Does converting from JSON to a CFG limit the capabilities of the grammar? i.e., are there things JSON can't represent that a context free grammar can?
    - ejones 2y ago
      You might be right that they're similarly powerful. In some cases, an arbitrary output format might in and of itself be desirable. Like it might result in token savings or be more natural for the LLM. For instance, generating code snippets to an API or plain text with constraints. And this is more esoteric, but technically in the case of JSON I suppose you could embed a grammar inside a JSON string, which I'm not sure JSON schema can express.
- J_Shelby_J 2y ago
  JSON is a sub-set of what GBNF can do, so there are still advantages to that approach. But even GBNF doesn’t go far enough. Ever try to restrict a model to a single sentence? root ::= \" \" item{{{min_count},{max_count}}} item ::= [A-Z] [^\\r\\n\\x0b\\x0c\\x85\\u2028\\u2029.?!]+ [a-z] (\". \" | \"? \" | \"! \") This kinda works if you don't mind no abbreviations, but you can't do something like this with JSON grammars afaik.
enobrev 2y ago
In a startup I was working on last year, I had a surprisingly good experience with using a json-schema in my prompt. I had to tweak the json response a bit because it was always invalid, but the issue was generally a missing colon or misplaced bracket. Data-wise it stuck to the schema very well, and cleaning up the json was simple enough that we got to zero parsing errors. I believe this was with 3.5. Sadly, that project was a final (relatively successful) attempt at getting traction before the startup was sold and is no longer live. Edit: Ouch, are the down-votes disbelief? Annoyance? Not sure what the problem is.
nichochar 2y ago
I'm a little confused why you have to specify "strict: true" to get this behavior. It is obviously always desired, I would be surprised for people to ever specify "strict: false". That API design leaves to be desired. I also learned about constrainted decoding[1], that they give a brief explanation about. This is a really clever technique! It will increase reliability as well as reduce latency (less tokens to pick from) once the initial artifacts are loaded. [1] https://www.aidancooper.co.uk/constrained-decoding/ https://www.aidancooper.co.uk/constrained-decoding/
- dgellow 2y ago
  Could you develop a bit re: the API? What do you dislike other than the “strict: true”?
- athyuttamre 2y ago
  Hi, I work on the OpenAI API — structured outputs schemas have limitations (e.g. all fields must be required, no additional properties allowed): https://platform.openai.com/docs/guides/structured-outputs/supported-schemas https://platform.openai.com/docs/guides/structured-outputs/s.... If your schema is not supported, but you still want to use the model to generate output, you would use `strict: false`. Unfortunately we cannot make `strict: true` the default because it would break existing users. We hope to make it the default in a future API version.
  - Der_Einzige 2y ago
    You should also mention that before you had done custom alignment accounting for this feature, that it was an excellent alignment breaker (therefor a big no-no to release too early) For example, if I ask an LLM to generate social security numbers, it will give the whole "I'm sorry Hal, I can't do that". If I ban all tokens except numbers and hyphens, prior to your "refusal = True" approach, it was guaranteed that even "aligned" models would generate what appeared to be social security numbers.
    - ethical_source 2y ago
srcreigh 2y ago
> The tokens that are valid at the beginning of the output include things like {, {“, {\n, etc. However, once the model has already sampled {“val, then { is no longer a valid token Oops, this is incorrect. {“val{“:2} is valid json. (modulo iOS quotes lol)
- jhgg 2y ago
  Valid JSON, sure, but that key does not conform to the schema provided in the example. The LLM must generate valid JSON that also conforms to the provided schema.
- jeffhuys 2y ago
  FYI you can just hold down the “ button and get a ".
simonw 2y ago
The price decrease is particularly notable because it represents a 50% cut in the price to handle image inputs, across any OpenAI model. Previously image inputs on GPT-4o-mini were priced the SAME as GPT-4o, so using mini wouldn't actually save you any money on image analysis. This new gpt-4o-2024-08-06 model is 50% cheaper than both GPT-4o AND GPT-4o-mini for image inputs, as far as I can tell. UPDATE: I may be wrong about this. The pricing calculator for image inputs on https://openai.com/api/pricing/ https://openai.com/api/pricing/ doesn't indicate any change in price for the new model.
- minimaxir 2y ago
  The calculator doesn't account for the fact that there are now two different prices in a given price matrix.
- jeffharris 2y ago
  yep image input on the new model is also 50% cheaper and apologies for the outdated pricing calculator ... we'll be updating it later today
  - brianjking 2y ago
    So we can send an image + text to the new structured output model and use the chain of thought schema? I'm getting an error. openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid content type. image_url is only supported by certain models.', 'type': 'invalid_request_error', 'param': 'messages.[1].content.[1].type', 'code': None}}
cvhc 2y ago
I wonder why the top level has to be an object instead of an array... I have some pretty normal use cases where I expect the model to extract a list of objects from the text. ``` openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid schema for response_format \'PolicyStatements\': schema must be a JSON Schema of \'type: "object"\', got \'type: "array"\'.', 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}} ``` I know I can always put the array into a single-key object but it's just so annoying I also have to modify the prompts accordingly to accomodate this.
- manquer 2y ago
  I can't say for OpenAI, but in general I have seen and used this design pattern to keep consistency of root object output and remove a lot of unnecessary validations and branching flows Otherwise you will to handle the scenarios in code everywhere if you don't know if the root is object or array. If the root has a key that confirms to a known schema then validation becomes easier to write for that scenario, Similar reasons to why so many APIs wrap with a key like 'data', 'value' or 'error' all responses or in RESTful HTTP endpoints collection say GET /v1/my-object endpoints do no mix with resource URIs GET /v1/my-object/1 the former is always an array the latter is always an object.
- tomComb 2y ago
  Well, this wouldn’t be a very satisfying explanation, but these JSON objects are often represented as Python dictionaries and those can’t have top level arrays.
  - Too 2y ago
    Try json.loads("[1,2,3]"), you'll get a list back. The reasons others already posted about extensibility are more correct.
- heliophobicdude 2y ago
  Back in the old days, top level arrays were a security risk because the array constructor in JS could be redefined and do bad-guy stuff. I cannot think of any json parsing clients that are vulnerable to this.
__jl__ 2y ago
There is another big change in gpt-4o-2024-08-06: It supports 16k output tokens compared to 4k before. I think it was only available in beta before. So gpt-4o-2024-08-06 actually brings three changes. Pretty significant for API users 1. Reliable structured outputs 2. Reduced costs by 50% for input, 33% for output 3. Up to 16k output tokens compared to 4k https://platform.openai.com/docs/models/gpt-4o https://platform.openai.com/docs/models/gpt-4o
- Culonavirus 2y ago
  That's actually pretty impressive... if they didn't dumb it down that is, which only time will tell.
- santiagobasulto 2y ago
  I’ve noticed that lately GPT has gotten more and more verbose. I’m wondering if it’s a subtle way to “raise prices”, as the average response is going to incur I more tokens, which makes any API conversation to keep growing in tokens of course (each IN message concatenates the previous OUT messages).
  - sashank_1509 2y ago
    they also spend more to generate more tokens. The more obvious reason is it seems like people rate responses better the longer they are. Lmsys demonstrated that GPT tops the leaderboard because it tends to give much longer and more detailed answers, and it seems like OpenAI is optimizing or trying to maximize lmsys.
    - maeil 2y ago
      Agree with this take, though in an even broader way; they're optimizing for the leaderboards and benchmarks in general. Longer outputs lead to better scores on those. Even in this thread I see a lot of comments bring them up, so it works for marketing. My take is that the leaderboards and benchmarks are still very flawed if you're using LLMs for any non-chat purpose. In the product I'm building, I have to use all of the big 4 models (GPT, Claude, Llama, Gemini), because for each of them there is at least one tasks that it performs much better than the other 3.
surfingdino 2y ago
Is it still NTSAT (Never The Same Answer Twice)?
- H8crilA 2y ago
  Yes, this happens by design.
  - oblio 2y ago
    Interesting, why? Is there no theoretical way to have stable models? Or some kind of executive decision?
    - H8crilA 2y ago
      It's just how this sort of a model works in inference. You can set the temperature to zero, then you'll get mostly constant outputs (but not quite, since as far as I can tell nobody is working on determinization of such systems, and it is not really deterministic).
      - oblio 2y ago
        I'm going to have to dust off some ancient math... I think LLMs are based on neuronal networks which are based on regression models which are based on systems of differential equations for which last time I studied this stuff, we only had probabilistic solutions? It's been a bunch of decades since I did anything even remotely related to these areas (and I was never good at math).
        H8crilA 2y ago
        Check out 3blue1brown, they have a series on transformers. Very approacheable. The randomness is very explicit in inference.
H8crilA 2y ago
How is this different from function calling?
- binarymax 2y ago
  Function calling uses JSON mode. While it has been mostly correct, I do get an incorrectly formatted response sometimes (maybe 1 in 10k requests?). So it sounds like this fixes that bug.
- tedsanders 2y ago
  Under the hood, it's quite similar to function calling. A few differences: - Structured Outputs is a bit more straightforward. e.g., you don't have to pretend you're writing a function where the second arg could be a two-page report to the user, and then pretend the "function" was called successfully by returning {"success": true} - Having two interfaces lets us teach the model different default behaviors and styles, depending on which you use - Another difference is that our current implementation of function calling can return both a text reply plus a function call (e.g., "Let me look up that flight for you"), whereas Structured Outputs will only return the JSON (I worked on this feature at OpenAI.)
  - technics256 2y ago
    How can we enable the text reply with a function call? Usually the message returned is a tool call only when it calls a tool?
    - tedsanders 2y ago
      There's no special interface, but you can write an instruction in a system message in the first position. E.g., "Before each function call, explain to the user what you're about to do." It's not super reliable, but the model can do it. Few-shot prompting might help as well.
- nsonha 2y ago
  I guess practically this will be the equivalent to function calling with a single tool. The AI splits out some payload that you presumably feed into whatever you do next. There are only 2 possible outcomes: can/cannot produce payload, which is equivalent to call/not call the next functionality. With function/tool calling, you give it a list of tools with payload and the AI will select between them, or refuse, so there are more than 2 possible outcomes. (I've never used any of these APIs, get that from just reading the docs)
paradite 2y ago
Really important update that was not mentioned: gpt-4o-2024-08-06 has 16,384 tokens output limit instead of 4,096 tokens. https://platform.openai.com/docs/models/gpt-4o https://platform.openai.com/docs/models/gpt-4o We don't need the GPT-4o Long Output anymore.
- OutOfHere 2y ago
  But is this also the default or just the max? Is the default 4k or 16k? Also, the question of the default value applies both at the server level and at the SDK level.
  - paradite 2y ago
    Unlike Anthropic, OpenAI models don't have a `max_tokens` setting for API calls, so I assume the max token output limit is automatically applied to API calls. Otherwise the max token output limit stated on the models page would be meaningless.
    - kevlened 2y ago
      OpenAI has a `max_tokens` setting. For the /chat/completions api, it defaults to the maximum for a desired model, but for the /completions api, it defaults to 16. https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_tokens https://platform.openai.com/docs/api-reference/chat/create#c...
      - paradite 2y ago
        Ops. Not sure how I missed that.
- floam 2y ago
  Long Output is 64K though.
gdiamos 2y ago
We've had this for over 1 year in Lamini - https://lamini-ai.github.io/inference/json_output/ https://lamini-ai.github.io/inference/json_output/ Works with any open LLM, including Llama 3.1
- radarsat1 2y ago
  Looks useful!
- AStrangeMorrow 2y ago
  Also the outlines library: https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines
  - gdiamos 2y ago
    Yeah! - outlines, guidance, jsonformer were inspiring for this line of work
  - HanClinto 2y ago
    Also note llama.cpp with grammar support: https://github.com/ggerganov/llama.cpp/tree/master/grammars https://github.com/ggerganov/llama.cpp/tree/master/grammars Supports an EBNF-like syntax, as well as JSON-Schema.
- gdiamos 2y ago
  [flagged]
- msoad 2y ago
  Why not JSON Schema?
  - gdiamos 2y ago
    We did some user studies and found that people found it less intuitive.
zoogeny 2y ago
Totally tangential, totally not related to the post (unless you squint your eyes and really blur things) ... I was thinking about the old canard of the sufficiently smart compiler. It made me think about LLM output and how in some way the output of a LLM could be bytecode as much as it could be the English language. You have a tokenized input and the translated output. You have a massive and easily generatable training set. I wonder if, one day, our compilers will be LLMs?
- jcims 2y ago
  You definitely could, not far removed from text to image or text to audio generators.
- pjc50 2y ago
  Why would you tolerate a nonreliable compiler with no assured relationship between its inputs and its outputs? Have people just got too comfortable with the C++ model of "UB means I can insert a security bug for you"?
  - bigyikes 2y ago
    In a hypothetical future where the reliability of LLMs improves, I can imagine the model being able to craft optimizations that a traditional compiler cannot. Like there are already cases where hand-rolling assembly can eke out performance gains, but few do that because it’s so arduous. If the LLM could do it reliably it’d be a huge win. It’s a big if, but not outside the realm of possibility.
    - zoogeny 2y ago
      I agree it is currently a pipe dream. But if I was looking for a doctoral research idea, it might be fun to work on something like that. Lots of potential avenues to explore, e.g. going from a high-level language to some IR, from some IR to bytecode, or straight from high-level to machine code. I mean, -O3 is already so much of a black box that I can't understand it. And the tedium of hand optimizing massive chunks of code is why we automate it at all. Boredom is something we don't expect LLMs to suffer, so having one pore over some kind of representation and apply optimizations seems totally reasonable. And if it had some kinds of "emergent behaviors" based on intelligence that allow it to beat the suite of algorithmic optimization we program into compilers, it could actually be a benefit.
adagradschool 2y ago
While text and image generation are getting cheaper at a significant rate, audio still seems to be just as expensive with ElevenLabs. I wonder why it is so.
say_it_as_it_is 2y ago
This puts like a dozen popular python libraries out of business
- AStrangeMorrow 2y ago
  At least depends on the approach and use: stuff like outlines (https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines) that actually changes the sampling to adhere to a grammar and can be used with local/custom models shouldn't be too impacted. Those are not really used on top of openAI models
- fastball 2y ago
  Once it actually works, sure. I just tried to use structured outputs with the latest release (openai-python 1.40) and it doesn't think Structured Outputs is a thing. EDIT: turns out my JSON schema is too large (800 lines + recursive) and seems to be breaking OpenAI's "convert to CFG" step. Whoops.
sansseriff 2y ago
Preprocessing new schema takes 'under 10 seconds'. That's... a huge range? Unless the preprocessing time is a small fraction of the inference time, I don't see the point. I'm working on an app that dynamically generates schema based on user input (a union of arbitrary types pulled from a library). The resulting schema is often in the 800 token range. Curious how long that would take to preprocess
pton_xd 2y ago
Isn't "we hardcoded JSON into the latest model" kind of the opposite direction, strategically, from "we're on the way to AGI and I need 7 trillion to get there?"
- isoprophlex 2y ago
  You are witnessing the final stages in the evolution of OpenAI from a messianic hype machine to Yet Another Product Company. Hence all the people leaving, too.
  - gardenhedge 2y ago
    I am ootl, employees are leaving openai?
    - dangrossman 2y ago
      > John Schulman, one of the co-founders of artificial intelligence company OpenAI, has left the ChatGPT maker for rival Anthropic, he said in a post on social media platform X late Monday. > OpenAI's President and co-founder Greg Brockman is also taking a sabbatical through the end of the year, he said in a X post late Monday. > Peter Deng, a vice-president of product, also left in recent months, a spokesperson said. And earlier this year, several members of the company’s safety teams exited. That's after co-founder and Chief Scientist Ilya Sutskever left in May.
      - oblio 2y ago
        Are there any co-founders left?
        sashank_1509 2y ago
        sam Altman for one.
      - g15jv2dp 2y ago
        Some employees quitting in a 1500 people company? Impossible. Any departure must be interpreted as the doom of openai, there's no other possibility.
irgolic 2y ago
Wasn’t this already fully supported in the tool calling API?
deleted 2y ago
[deleted]
ramoz 2y ago
Can someone explain how this is different/better than the current state of function calling (which I’ve been using to get a consistent json schema response without issue)?
- mrshu 2y ago
  This post (from an OpenAI researcher) contains a bit more background: https://news.ycombinator.com/item?id=41174213 https://news.ycombinator.com/item?id=41174213
- zbyforgotp 2y ago
  This is guaranteed, function calling without it is not. The old way can work for you, but my experience is different, especially with complex schemas.
- jacobsimon 2y ago
  For starters, the naming is much less confusing. But the behavior also appears to be enforced/validated at some layer (hopefully?), which function calling did not seem to be. I was experimenting with it a couple weeks ago and it would work like 75% of the time but would often give me invalid results for schemas with relatively simple nested objects.
deleted 2y ago
[deleted]
Der_Einzige 2y ago
Now the question is when they will support soft constraints like this: https://huggingface.co/blog/constrained-beam-search https://huggingface.co/blog/constrained-beam-search
agtech_andy 2y ago
I have had a lot of success using BoundaryML (https://www.boundaryml.com/ https://www.boundaryml.com/) for this. They have also been super responsive for any of my questions.
- aaronvg 2y ago
  thanks for the shoutout, we benchmarked our approach against other function-calling techniques and we've been able to beat all other approaches every time (even by 8%!) just by getting better at parsing the data and representing schemas with less tokens using type definitions instead of json schema. You can take a look at our BFCL results on that site or the github: https://github.com/BoundaryML/baml https://github.com/BoundaryML/baml We'll be publishing our comparison against OpenAI structured outputs in the next 2 days, and a deeper dive into our results, but we aim to include this kind of constrained generation as a capability in the BAML DSL anyway longterm!
tarofchaos 2y ago
Two years too late. I think we are going through a bozo period at OpenAI where small things are being highlighted as achievements.
damsta 2y ago
Can we get something like that in Gemini 1.5 Flash?
mugivarra69 2y ago
cohere had this like a while ago
MattDaEskimo 2y ago
"We have ripped code from a bunch of open-source variations and slapped it behind our brutally abstracted API. Interoperable with other external models like the open source versions? What, are you mad?"
jumploops 2y ago
By using JSON mode, GPT-4{o} has been able to do this reliably for months (100k+ calls). We use GPT-4o to build dynamic UI+code[0], and almost all of our calls are using JSON mode. Previously it mostly worked, but we had to do some massaging on our end (backtick removal, etc.). With that said, this will be great for GPT-4o-mini, as it often struggles/forgets to format things as we ask. Note: we haven't had the same success rate with function calling compared to pure JSON mode, as the function calling seems to add a level of indirection that can reduce the quality of the LLMs output YMMV. Anyhow, excited for this! [0]https://magicloops.dev https://magicloops.dev
- deleted 2y ago
  [deleted]
- qwertox 2y ago
  What a cool product! I was about to recommend you to submit it as a "Show HN", but it turns out that it already got submitted one year ago. Would you mind sharing a bit on how things have evolved?
  - jumploops 2y ago
    Thanks and great question :) When we first launched, the tool was very manual; you had to generate each step via the UI. We then added a "Loop Creator agent" that now builds Loops for you without intervention. Over the past few months we've mostly been fixing feature gaps and improving the Loop Creator. Based on recent user feedback, we've put a few things in motion: - Form generator (for manual loops) - Chrome extension (for local automations) - In-house Google Sheets integration - Custom outputs (charts, tables, etc.) - Custom Blocks (shareable with other users) With these improvements, you'll be able to create "single page apps" like this one I made for my wife's annual mango tasting party[0]. In addition to those features, we're also launching a new section for Loop templates + educational content/how-tos, in an effort to help people get started. To be super candid, the Loop Creator has been a pain. We started at an 8% success rate and we're only just now at 25%. Theoretically we should be able to hit 80%+ based on existing loop requests, but we're running into limits with the current state of LLMs. [0]https://mangota.ngo https://mangota.ngo
msp26 2y ago
Is the JSON actually being fed into the LLM's context or is it still being converted into typescript? The previous setup didn't allow for custom types, only objects/string/num/bool. Are the enums put into context or purely used for constrained sampling?
OutOfHere 2y ago
Using this feature will obviously "lock you in" to OpenAI, specifically to this model too, at least until other companies catch on. While text prompts can more easily be moved to other LLMs, this feature cannot currently be ported as such. I would use it only if a text prompt is insufficient despite retries.
- BoorishBears 2y ago
  It's already supported by multiple other providers. Fireworks, Together, probably more.
- dtquad 2y ago
  OpenAI-style JSON mode and function calling rapidly became the industry standard way of doing it. It will probably also happen for this feature.
  - toomuchtodo 2y ago
    “S3 compatible”
- PufPufPuf 2y ago
  This feature has existed for quite some time in several inference libraries, like Outlines, under the names "constrained decoding" or "guided decoding". Some even include it in their OpenAI-compatible API in a very similar form (allowing to pass in a JSON Schema). All this required doing your own inference, though -- so the announcement really just brings this popular feature "to the masses".
- moralestapia 2y ago
  OTOH, not using it could "lock you out" of building a cool product for your users, so ...
  - OutOfHere 2y ago
    It depends. Quite often the intended schema is simple enough that parsing of a plaintext lines using a regex is sufficient. I do however have to use a verbose prompt to get it to comply with the expected format. Combine this with retries in case of rare failures, and the job is done. I think this new feature is more relevant for intricate schemas and dynamic schemas when a text prompt cannot do the job.
Brechreiz 2y ago
[dead]
jappgar 2y ago
It's nice that they're not making me pay for broken json anymore but touting this as a "feature" is laughable. It's a bug fix. They should never have been charging for malformed responses in the first place!
LAC-Tech 2y ago
Good to see JSON Schema being more widely adopted. I remember doing a project a few years ago in XML just because XML schemas were everywhere and JSON ones were still not really used.
Brosper 2y ago
I think they would like to have something like artifacts in Claude
Havoc 2y ago
Surprised to see OpenAI staff in comments. I don’t recall that being the case on previous threads. (Hello!)
sub7 2y ago
To "look up all my orders in May that were fulfilled and not delivered" is a 1 line query in every ORM I know of. Why in the world would I literally pass around a schema file with each request that is (according to the example) 91 lines of code and pay someone for the privilege of maybe doing it correctly?
roseway4 2y ago
We extensively use vLLM's support for Outlines Structured Output with small language models (llama3 8B, for example) in Zep[0][1]. OpenAI's Structured Output is a great improvement on JSON mode, but it is rather primitive compared to vLLM and Outlines. # Very Limited Field Typing OpenAI offers a very limited set of types[2] (String, Number, Boolean, Object, Array, Enum, anyOf) without the ability to define patterns and max/min lengths. Outlines supports defining arbitrary RegEx patterns, making extracting currencies, phone numbers, zip codes, comma-separated lists, and more a trivial exercise. # High Schema Setup Cost / Latency vLLM and Outlines offer near-zero cost schema setup: RegEx finite state machine construction is extremely cheap on the first inference call. While OpenAI's context-free grammar generation has a significant latency penalty of "under ten seconds to a minute". This may not impact "warmed-up" inference but could present issues if schemas are more dynamic in nature. Right now, this feels like a good first step, focusing on ensuring the right fields are present in schema-ed output. However, it doesn't yet offer the functionality to ensure the format of field contents beyond a primitive set of types. It will be interesting to watch where OpenAI takes this. [0] https://help.getzep.com/structured-data-extraction https://help.getzep.com/structured-data-extraction [1] https://help.getzep.com/dialog-classification https://help.getzep.com/dialog-classification [2] https://platform.openai.com/docs/guides/structured-outputs/supported-schemas https://platform.openai.com/docs/guides/structured-outputs/s...
bamboozled 2y ago
This is a game changer
tuan3w 2y ago
Just curious if structured outputs/constrained generation improve model accuracy, e.g., for information extraction problems. Does anyone have experience with this and why?
thrance 2y ago
And you still can't provide a custom grammar to the API... The company I work for desperately need the LLM to consistently generate results in a subset of HTML. I was able to craft a small grammar file that does just that, in under 5 minutes, and use it successfully with llama.cpp. Yet, there are still no API offering this basic feature that could really benefit everyone. Instead we have a thousand garbage medium articles with "tips & tricks" on how to prompt the AI to get better results. It's as if people don't care anymore about consistency and reliability.
blackcat201 2y ago
Do beware on some reasoning task, our recent work[0] actually found it may cause some performance degradation as well as possible reasoning weakening in JSON. I really hope they fix this in the latest GPT-4o version. [0] https://arxiv.org/abs/2408.02442 https://arxiv.org/abs/2408.02442
- kiratp 2y ago
  Thank you! This confirms my intuition! Structured generation seems counter to every other signal we have that chain of thought etc improves performance.
myprotegeai 2y ago
I send dynamic jsonschemas to LLMs in manifest[1] to simulate function calls based on function signatures. These structured outputs will be enormously useful. Other posts claim that you can generate jsonschema-conformant output reliably without this, and while I mostly agree, there is an edge case where gpt4o struggles, and that is simple data types. For example, a string, in jsonschema, has a schema of simply {"type": "string"}, and an example value of "hello world." However, gpt4o would produce something like {"value": "hello world"} at a very high probability. I had to include specific few shot examples of what not to do in order to make this simple case reliable. I suspect there are other non-obvious cases. 1. https://github.com/amoffat/manifest https://github.com/amoffat/manifest
franze 2y ago
finally at gpt.franzai.com we use a 3 times check until now if chatgpt does not return valid json then trim anything before and after last {} if this is not valid json feedback him full response and ask for valid json if this is not valid json put full response into self made json to get al least something we can work with in an "something did not work out" response
tomduncalf 2y ago
Would constraining the tokens that the model can choose from impact its “intelligence”/“creativity” in other (possibly negativel) ways?
armcat 2y ago
This is awesome, and simplifies lot of my workflows when using their APIs directly. I also want to give a shout out to Outlines team, https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines, they've been doing structured outputs for the last 12 months, and their open source lib can be applied across all open weight and closed/API-based models. Most likely, Outlines heavily inspired the OpenAI team, maybe even using some of their codebase.
brap 2y ago
Structured outputs seem like an important step forward for LLMs to make them more useful (this includes intermediate "outputs" that are internal to the model, e.g. producing code and then running it without AI, producing mathematical/logical statements and then proving them without AI, etc). Basically, once you have structure, you can use non-AI tools that are far more efficient and don't hallucinate. These tools can either be external, or embedded in the model itself (think RAG, reasoning, etc.). I wonder if JSON is just a first step, and eventually this will be generalized to any formal grammar...
mariarmestre 2y ago
Hmmm... this is not eligible for their zero data retention policy anymore. Not sure how this will go down.
- kevlened 2y ago
  They added a clause that the schemas themselves aren't covered by ZDR. Their policy for the prompts appears unchanged. https://platform.openai.com/docs/models/default-usage-policies-by-endpoint https://platform.openai.com/docs/models/default-usage-polici...
blixt 2y ago
I was impressed by Microsoft’s AICI where the idea is a WASM program can choose the next tokens. And relatedly their Guidance[1] framework which can use CFGs and programs for local inference to even speed it up with context aware token filling. I hope this implies API-based LLMs may be moving in a similar direction. [1] https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance
munro 2y ago
Wow, this is cool-- first time I've seen generating well formed output has been made easy, the use of Pydantic is clever/easy... Which has lead me to the prior art: using same Pydantic API to generate structured output from local LLMs https://github.com/outlines-dev/outlines?tab=readme-ov-file#efficient-json-generation-following-a-pydantic-model https://github.com/outlines-dev/outlines?tab=readme-ov-file#...
- Terretta 2y ago
  Commercial use of LLMs is one of those spaces a startup probably needs more than skating ahead of the incumbents on the incumbents' “missing” features as startup value prop. Even if the startup has a year or two head start, it may find its beta “OBE” (overcome by events): https://dottxt.co https://dottxt.co That said, Outline has been making this concept portable for a long time, it's the langchain the community deserves: https://outlines-dev.github.io/outlines/cookbook/ https://outlines-dev.github.io/outlines/cookbook/
tipsytoad 2y ago
How is this different from instructor? github.com/jxnl/instructor namely, why did they take so long for something that just seems like a wrapper around function calling?
hallqv 2y ago
They promised us super intelligence - all we got was valid json
MeetBagelPPML 2y ago
This is super cool! If you're looking to get in early on a free tool that allows you to finetune your model using encrypted data sets you can sign up here: https://waitlist.bagel.net/ https://waitlist.bagel.net/

764 points by davidbarker 2y ago

lmshn 2y ago

[flagged]

deleted 2y ago
[deleted]

gamegoblin 2y ago

I'm glad they gave up on their "fine-tuning is all you need" approach to structured output. It's possible fine-tuning will work in the long term, but in the short-term, people are trying to build things, and fine-tuning wasn't cutting it. Surprised it took them so long — llama.cpp got this feature 1.5 years ago (actually an even more general version of it that allows the user to provide any context free grammar, not just JSON schema)

chhabraamit 2y ago
How does llama.cpp’s grammar adherence work? Does it keep validating the predicted tokens and backtrack when it’s not valid?
- gamegoblin 2y ago
  It's essentially an Earley Parser[0]. It maintains a set of all possible currently valid parses, and zeroes out the probability of any token that isn't valid in at least 1 of the current potential parse trees. There are contrived grammars you can give it that will make it use exponential memory, but in practice most real-world grammars aren't like this. [0] https://en.wikipedia.org/wiki/Earley_parser https://en.wikipedia.org/wiki/Earley_parser
  - jules 2y ago
    Earley parsers should not need more than O(n^2) memory.
  - orlp 2y ago
    You don't even need that for JSON. JSON can be expressed using a LR(1) grammar, so you can do it in linear time and space.
    - gamegoblin 2y ago
      Yes, the llama.cpp work supports arbitrary CFGs, not just JSON
tcdent 2y ago
GPT is still a language model, so at some point it's still just tokens. Is this just a schema validation layer on their end to avoid the round trip (and cost) of repeating the call?
- gamegoblin 2y ago
  Language models like GPT output a large vector of probabilities for the next token. Then a sampler decides which of those tokens to pick. The simplest algorithm for getting good quality output is to just always pick the highest probability token. If you want more creativity, maybe you pick randomly among the top 5 highest probability tokens or something. There are a lot of methods. All that grammar-constrained decoding does is zero out the probability of any token that would violate the grammar.
  - nickreese 2y ago
    Thank you for this explanation. A few things just clicked for me.
BoorishBears 2y ago
I was surprised it took so long until I reached this line: > The model can fail to follow the schema if the model chooses to refuse an unsafe request. If it chooses to refuse, the return message will have the refusal boolean set to true to indicate this. I'm not sure how they implemented that, maybe they've figured out a way to give the grammar a token or set of tokens that are always valid mid generation and indicate the model would rather not continue generating. Right now JSON generation is one of the most reliable ways to get around refusals, and they managed not to introduce that weakness into their model
Der_Einzige 2y ago
For many things, fine-tuning as we know of it will NEVER fully solve it, there's no hope. Even fine-tuning a model to not use the letter "e" to an overwhelming degree doesn't entirely prevent it, only reduces its chances to increasingly small amounts. Shamesless self plug, and from before the ChatGPT era too! https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai-1 https://paperswithcode.com/paper/most-language-models-can-be...

leetharris 2y ago

At the bottom: >Acknowledgements Structured Outputs takes inspiration from excellent work from the open source community: namely, the outlines, jsonformer, instructor, guidance, and lark libraries. It is cool to see them acknowledge this, but it's also lame for a company named "OpenAI" to acknowledge getting their ideas from open source, then contributing absolutely NOTHING back to open source with their own implementation.

warkdarrior 2y ago
> it's also lame for a company named "OpenAI" to acknowledge getting their ideas from open source, then contributing absolutely NOTHING back to open source with their own implementation Maybe those projects were used as-is by OpenAI, so there was nothing new to contribute.
- reustle 2y ago
  I think they may be alluding to sponsorships as well as code contributions. i.e. https://github.com/sponsors/jxnl https://github.com/sponsors/jxnl
spencerchubb 2y ago
Is offering gpt4o for free through chatgpt not enough of a contribution? They didn't release source code, but they made a product free to use
- mplewis 2y ago
  No. If it were free you'd be able to use it as a programming API. It's not free and it's not unlimited - it's a time-limited marketing tool.
  - spencerchubb 2y ago
    How are you defining the word free?
- talldayo 2y ago

deleted 2y ago

[deleted]

jjcm 2y ago

Interesting tidbit at the very end that's worth noting for anyone using the API today: > By switching to the new gpt-4o-2024-08-06, developers save 50% on inputs ($2.50/1M input tokens) and 33% on outputs ($10.00/1M output tokens) compared to gpt-4o-2024-05-13.

scrollop 2y ago
From what I've learned from OpenAI, the "latest" "cheaper" model will perform worse than the previous model on various tasks (esp reasoning).
- ralusek 2y ago
  I don't think it's been well enough acknowledged that all of the shortcuts LLMs have been taking with ways of attempting to compress/refine/index the attention mechanism seem to result in dumber models. GPT 4 Turbo was more like GPT 3.9, and GPT 4o is more like GPT 3.7.
  - scrollop 2y ago
    Some commenters acknowledge it - and quantify it: https://www.youtube.com/watch?v=Tf1nooXtUHE&t=689s https://www.youtube.com/watch?v=Tf1nooXtUHE&t=689s
  - Der_Einzige 2y ago
    They try to gaslight us and tell us this isn't true because of benchmarks, as though anyone has done anything but do the latent space exploration equivalent of throwing darts at the ocean from space. It's taken years to get even preliminary reliable decision boundary examples from LLMs because doing so is expensive.
  - alach11 2y ago
    Do you have benchmarks demonstrating this? In my own personal/team benchmarks, I've seen 4o consistently outperform the original gpt-4.

deleted 2y ago

[deleted]

nerdjon 2y ago

I have a bad feeling that this is just going to introduce more shovelware apps that try to shove AI use in without really understanding what they are going to get back. Yay I can now ensure the json object will look how I want, but lets completely disregard any concern of wether or not the data returned is valuable. I don't understand why we are already treating these systems as general purpose AI when they are not. (Ok I do understand it, but it is frustrating). The example given of "look up all my orders in may of last year that were fulfilled but not delivered on time". First I have found these models incredibly dumb when it comes to handling time. But even beyond that, if you really are going to do this. I really hope you double check the data before presenting the data you get back as true. Worse that is just double checking what it gives back to you is accurate, not checking that it isn't telling you about something. Every time I try to experiment with supplying data and asking for data back, they fall flat on their face before we even get to the json being formatted properly. That was not the issue that needed to be solved yet when it still fundamentally messes up the data. Often just returning wrong information. Sometimes it will be right though, but that is the problem. It may luck out and be right enough times that you gain confidence in it and stop double checking what it is giving back to you. I guarantee you someone is going to have a discussion about using this, feeding it data, and then storing the response in a database.

titzer 2y ago

It's so wild that the bar for AI performance is both absurdly high and absurdly low at the same time. To specify an output format (language or grammar) for solving a computational problem is one of the oldest exercises around. On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification. It's weird reading the kind of self-congratulating blogpost about this, like OpenAI has just discovered flint knives. On the other hand, a computer system can process natural language with extremely ambiguous, open-ended problems, compute solutions to said problems, even correct its own mistakes--and then it can format the output correctly. And then on yet another hand, it only took about 10^25 floating point operations (yeah, just ten million trillion trillion, right!?) to get this outcome.

codingwagie 2y ago
I think it will take a long time for the world at large to realize and then operationalize the potential of this "mundane" technology. It is revolutionary, and also sitting in plain sight. Such a huge technological shift that was considered decades out only a few years ago
- ben_w 2y ago
  Although I am an optimist* about what this can do, I am very much aware — from personal experience — how easy it is to see more than is really there. The realisation of the tech might be fantastic new things… or it might be that people like me are Clever Hans-ing the models. * that may be the wrong word; "strong capabilities" is what I think is present, those can be used for ill effects which is pessimistic.
srcreigh 2y ago
If I wanted to be a silly pedant, I’d say that Turing machines are language specifications and thus it’s theoretically impossible for an LLM or any program to validate output formats in general.
- deleted 2y ago
  [deleted]

wewtyflakes 2y ago

Why would someone want `strict` to be anything other than `true`?

davidkunz 2y ago
Maybe if you can't precisely model your structure with (OpenAI's subset of) JSON schema.
ComputerGuru 2y ago
There are many reasons, though I am not sure which they had in mind. One thing is that LLMs in general tend to do better when they can be more verbose in their output and sort of “think aloud” to reach an answer. Insisting on strict output format would rob it of the benefits (because it doesn’t just not emit but completely skips those stages, or else you’d be paying for those elided output tokens).
- wewtyflakes 2y ago
  But then why would someone specify that the response has to be in a given JSON schema (by presence of the schema itself), but then also not care if it is actually using that schema (by specifying `strict` as `false`)? That is the use-case I can't wrap my head around.
  - ComputerGuru 2y ago
    Ah, good point. It honestly feels/looks like a hack to flip a switch on their end and they’re just bubbling it back up to the user.
tedsanders 2y ago
We didn't cover this in the announcement post, but there are a few reasons: - The first request with each JSON schema will be slow, as we need to preprocess the JSON schema into a context-free grammar. If you don't want that latency hit (e.g., you're prototyping, or have a use case that uses variable one-off schemas), then you might prefer "strict": false - You might have a schema that isn't covered by our subset of JSON schema. (To keep performance fast, we don't support some more complex/long-tail features.) - In JSON mode and Structured Outputs, failures are rarer but more catastrophic. If the model gets too confused, it can get stuck in loops where it just prints technically valid output forever without ever closing the object. In these cases, you can end up waiting a minute for the request to hit the max_token limit, and you also have to pay for all those useless tokens. So if you have a really tricky schema, and you'd rather get frequent failures back quickly instead of infrequent failures back slowly, you might also want "strict": false But in 99% of cases, you'll want "strict": true.

_vaporwave_ 2y ago

Anyone else catch this reference in one of the examples? > 9.11 and 9.9 -- which is bigger https://community.openai.com/t/why-9-11-is-larger-than-9-9-incredible/869824 https://community.openai.com/t/why-9-11-is-larger-than-9-9-i...

jodacola 2y ago
Amusingly, I immediately thought 9.11 - but in the context of a newer version of software. Ever have those moments where you're so deep in context of some ecosystem that you skip right past the basics, like 9.9 being a larger number than 9.11?

elpocko 2y ago

Doesn't the BNF grammar approach in llama.cpp solve this issue in a generic way that should work with any model? Why wouldn't they use that?

ejones 2y ago
Similar approach to llama.cpp under the hood - they convert the schema to a grammar. Llama.cpp's implementation was specific to the ggml stack, but what they've built sounds similar to Outlines, which they acknowledged.
HanClinto 2y ago
llama.cpp's GBNF grammar is generic, and indeed works with any model. I can't speak for other approaches, but -- while llama.cpp's implementation is nice in that it always generates valid grammars token-by-token (and doesn't require any backtracking), it is tough in that -- in the case of ambiguous grammars (where we're not always sure where we're at in the grammar until it finishes generating), then it keeps all valid parsing option stacks in memory at the same time. This is good for the no-backtracking case, but it adds a (sometimes significant) cost in terms of being rather "explosive" in the memory usage (especially if one uses a particularly large or poorly-formed grammar). Creating a grammar that is openly hostile and crashes the inference server is not difficult. People have done a lot of work to try and address some of the more egregious cases, but the memory load can be significant. One example of memory optimization: https://github.com/ggerganov/llama.cpp/pull/6616 https://github.com/ggerganov/llama.cpp/pull/6616 I'm not entirely sure what other options there are for approaches to take, but I'd be curious to learn how other libraries (Outlines, jsonformer) handle syntax validation.

behnamoh 2y ago

Well, there goes one of the big advantages of open-source models... For a long time, I was relying on such guaranteed structured outputs as a "secret sauce" that only works using llama.cpp's GBNF grammars. Now OpenAI literally introduced the same concept but a bit more accessible (since you create a JSON and they convert it to a grammar). Those of you who have used GBNF, do you think it still has any advantage over what OpenAI just announced?

ejones 2y ago
FWIW, llama.cpp has always had a JSON schema -> GBNF converter, although it launched as a companion script. Now I think it's more integrated in the CLI and server. But yeah I mean, GBNF or other structured output solutions would of course allow you to supply formats other than JSON schema. It sounds conceivable though that OpenAI could expose the grammars directly in the future, though.
- behnamoh 2y ago
  I think for certain tasks it's still easier to write the grammar directly. Does converting from JSON to a CFG limit the capabilities of the grammar? i.e., are there things JSON can't represent that a context free grammar can?
  - ejones 2y ago
    You might be right that they're similarly powerful. In some cases, an arbitrary output format might in and of itself be desirable. Like it might result in token savings or be more natural for the LLM. For instance, generating code snippets to an API or plain text with constraints. And this is more esoteric, but technically in the case of JSON I suppose you could embed a grammar inside a JSON string, which I'm not sure JSON schema can express.
J_Shelby_J 2y ago
JSON is a sub-set of what GBNF can do, so there are still advantages to that approach. But even GBNF doesn’t go far enough. Ever try to restrict a model to a single sentence? root ::= \" \" item{{{min_count},{max_count}}} item ::= [A-Z] [^\\r\\n\\x0b\\x0c\\x85\\u2028\\u2029.?!]+ [a-z] (\". \" | \"? \" | \"! \") This kinda works if you don't mind no abbreviations, but you can't do something like this with JSON grammars afaik.

enobrev 2y ago

In a startup I was working on last year, I had a surprisingly good experience with using a json-schema in my prompt. I had to tweak the json response a bit because it was always invalid, but the issue was generally a missing colon or misplaced bracket. Data-wise it stuck to the schema very well, and cleaning up the json was simple enough that we got to zero parsing errors. I believe this was with 3.5. Sadly, that project was a final (relatively successful) attempt at getting traction before the startup was sold and is no longer live. Edit: Ouch, are the down-votes disbelief? Annoyance? Not sure what the problem is.

nichochar 2y ago

I'm a little confused why you have to specify "strict: true" to get this behavior. It is obviously always desired, I would be surprised for people to ever specify "strict: false". That API design leaves to be desired. I also learned about constrainted decoding[1], that they give a brief explanation about. This is a really clever technique! It will increase reliability as well as reduce latency (less tokens to pick from) once the initial artifacts are loaded. [1] https://www.aidancooper.co.uk/constrained-decoding/ https://www.aidancooper.co.uk/constrained-decoding/

dgellow 2y ago
Could you develop a bit re: the API? What do you dislike other than the “strict: true”?
athyuttamre 2y ago
Hi, I work on the OpenAI API — structured outputs schemas have limitations (e.g. all fields must be required, no additional properties allowed): https://platform.openai.com/docs/guides/structured-outputs/supported-schemas https://platform.openai.com/docs/guides/structured-outputs/s.... If your schema is not supported, but you still want to use the model to generate output, you would use `strict: false`. Unfortunately we cannot make `strict: true` the default because it would break existing users. We hope to make it the default in a future API version.
- Der_Einzige 2y ago
  You should also mention that before you had done custom alignment accounting for this feature, that it was an excellent alignment breaker (therefor a big no-no to release too early) For example, if I ask an LLM to generate social security numbers, it will give the whole "I'm sorry Hal, I can't do that". If I ban all tokens except numbers and hyphens, prior to your "refusal = True" approach, it was guaranteed that even "aligned" models would generate what appeared to be social security numbers.
  - ethical_source 2y ago

srcreigh 2y ago

> The tokens that are valid at the beginning of the output include things like {, {“, {\n, etc. However, once the model has already sampled {“val, then { is no longer a valid token Oops, this is incorrect. {“val{“:2} is valid json. (modulo iOS quotes lol)

jhgg 2y ago
Valid JSON, sure, but that key does not conform to the schema provided in the example. The LLM must generate valid JSON that also conforms to the provided schema.
jeffhuys 2y ago
FYI you can just hold down the “ button and get a ".

simonw 2y ago

The price decrease is particularly notable because it represents a 50% cut in the price to handle image inputs, across any OpenAI model. Previously image inputs on GPT-4o-mini were priced the SAME as GPT-4o, so using mini wouldn't actually save you any money on image analysis. This new gpt-4o-2024-08-06 model is 50% cheaper than both GPT-4o AND GPT-4o-mini for image inputs, as far as I can tell. UPDATE: I may be wrong about this. The pricing calculator for image inputs on https://openai.com/api/pricing/ https://openai.com/api/pricing/ doesn't indicate any change in price for the new model.

minimaxir 2y ago
The calculator doesn't account for the fact that there are now two different prices in a given price matrix.
jeffharris 2y ago
yep image input on the new model is also 50% cheaper and apologies for the outdated pricing calculator ... we'll be updating it later today
- brianjking 2y ago
  So we can send an image + text to the new structured output model and use the chain of thought schema? I'm getting an error. openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid content type. image_url is only supported by certain models.', 'type': 'invalid_request_error', 'param': 'messages.[1].content.[1].type', 'code': None}}

cvhc 2y ago

I wonder why the top level has to be an object instead of an array... I have some pretty normal use cases where I expect the model to extract a list of objects from the text. ``` openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid schema for response_format \'PolicyStatements\': schema must be a JSON Schema of \'type: "object"\', got \'type: "array"\'.', 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}} ``` I know I can always put the array into a single-key object but it's just so annoying I also have to modify the prompts accordingly to accomodate this.

manquer 2y ago
I can't say for OpenAI, but in general I have seen and used this design pattern to keep consistency of root object output and remove a lot of unnecessary validations and branching flows Otherwise you will to handle the scenarios in code everywhere if you don't know if the root is object or array. If the root has a key that confirms to a known schema then validation becomes easier to write for that scenario, Similar reasons to why so many APIs wrap with a key like 'data', 'value' or 'error' all responses or in RESTful HTTP endpoints collection say GET /v1/my-object endpoints do no mix with resource URIs GET /v1/my-object/1 the former is always an array the latter is always an object.
tomComb 2y ago
Well, this wouldn’t be a very satisfying explanation, but these JSON objects are often represented as Python dictionaries and those can’t have top level arrays.
- Too 2y ago
  Try json.loads("[1,2,3]"), you'll get a list back. The reasons others already posted about extensibility are more correct.
heliophobicdude 2y ago
Back in the old days, top level arrays were a security risk because the array constructor in JS could be redefined and do bad-guy stuff. I cannot think of any json parsing clients that are vulnerable to this.

__jl__ 2y ago

There is another big change in gpt-4o-2024-08-06: It supports 16k output tokens compared to 4k before. I think it was only available in beta before. So gpt-4o-2024-08-06 actually brings three changes. Pretty significant for API users 1. Reliable structured outputs 2. Reduced costs by 50% for input, 33% for output 3. Up to 16k output tokens compared to 4k https://platform.openai.com/docs/models/gpt-4o https://platform.openai.com/docs/models/gpt-4o

Culonavirus 2y ago
That's actually pretty impressive... if they didn't dumb it down that is, which only time will tell.
santiagobasulto 2y ago
I’ve noticed that lately GPT has gotten more and more verbose. I’m wondering if it’s a subtle way to “raise prices”, as the average response is going to incur I more tokens, which makes any API conversation to keep growing in tokens of course (each IN message concatenates the previous OUT messages).
- sashank_1509 2y ago
  they also spend more to generate more tokens. The more obvious reason is it seems like people rate responses better the longer they are. Lmsys demonstrated that GPT tops the leaderboard because it tends to give much longer and more detailed answers, and it seems like OpenAI is optimizing or trying to maximize lmsys.
  - maeil 2y ago
    Agree with this take, though in an even broader way; they're optimizing for the leaderboards and benchmarks in general. Longer outputs lead to better scores on those. Even in this thread I see a lot of comments bring them up, so it works for marketing. My take is that the leaderboards and benchmarks are still very flawed if you're using LLMs for any non-chat purpose. In the product I'm building, I have to use all of the big 4 models (GPT, Claude, Llama, Gemini), because for each of them there is at least one tasks that it performs much better than the other 3.

surfingdino 2y ago

Is it still NTSAT (Never The Same Answer Twice)?

H8crilA 2y ago
Yes, this happens by design.
- oblio 2y ago
  Interesting, why? Is there no theoretical way to have stable models? Or some kind of executive decision?
  - H8crilA 2y ago
    It's just how this sort of a model works in inference. You can set the temperature to zero, then you'll get mostly constant outputs (but not quite, since as far as I can tell nobody is working on determinization of such systems, and it is not really deterministic).
    - oblio 2y ago
      I'm going to have to dust off some ancient math... I think LLMs are based on neuronal networks which are based on regression models which are based on systems of differential equations for which last time I studied this stuff, we only had probabilistic solutions? It's been a bunch of decades since I did anything even remotely related to these areas (and I was never good at math).
      - H8crilA 2y ago
        Check out 3blue1brown, they have a series on transformers. Very approacheable. The randomness is very explicit in inference.

H8crilA 2y ago

How is this different from function calling?

binarymax 2y ago
Function calling uses JSON mode. While it has been mostly correct, I do get an incorrectly formatted response sometimes (maybe 1 in 10k requests?). So it sounds like this fixes that bug.
tedsanders 2y ago
Under the hood, it's quite similar to function calling. A few differences: - Structured Outputs is a bit more straightforward. e.g., you don't have to pretend you're writing a function where the second arg could be a two-page report to the user, and then pretend the "function" was called successfully by returning {"success": true} - Having two interfaces lets us teach the model different default behaviors and styles, depending on which you use - Another difference is that our current implementation of function calling can return both a text reply plus a function call (e.g., "Let me look up that flight for you"), whereas Structured Outputs will only return the JSON (I worked on this feature at OpenAI.)
- technics256 2y ago
  How can we enable the text reply with a function call? Usually the message returned is a tool call only when it calls a tool?
  - tedsanders 2y ago
    There's no special interface, but you can write an instruction in a system message in the first position. E.g., "Before each function call, explain to the user what you're about to do." It's not super reliable, but the model can do it. Few-shot prompting might help as well.
nsonha 2y ago
I guess practically this will be the equivalent to function calling with a single tool. The AI splits out some payload that you presumably feed into whatever you do next. There are only 2 possible outcomes: can/cannot produce payload, which is equivalent to call/not call the next functionality. With function/tool calling, you give it a list of tools with payload and the AI will select between them, or refuse, so there are more than 2 possible outcomes. (I've never used any of these APIs, get that from just reading the docs)

paradite 2y ago

Really important update that was not mentioned: gpt-4o-2024-08-06 has 16,384 tokens output limit instead of 4,096 tokens. https://platform.openai.com/docs/models/gpt-4o https://platform.openai.com/docs/models/gpt-4o We don't need the GPT-4o Long Output anymore.

OutOfHere 2y ago
But is this also the default or just the max? Is the default 4k or 16k? Also, the question of the default value applies both at the server level and at the SDK level.
- paradite 2y ago
  Unlike Anthropic, OpenAI models don't have a `max_tokens` setting for API calls, so I assume the max token output limit is automatically applied to API calls. Otherwise the max token output limit stated on the models page would be meaningless.
  - kevlened 2y ago
    OpenAI has a `max_tokens` setting. For the /chat/completions api, it defaults to the maximum for a desired model, but for the /completions api, it defaults to 16. https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_tokens https://platform.openai.com/docs/api-reference/chat/create#c...
    - paradite 2y ago
      Ops. Not sure how I missed that.
floam 2y ago
Long Output is 64K though.

gdiamos 2y ago

We've had this for over 1 year in Lamini - https://lamini-ai.github.io/inference/json_output/ https://lamini-ai.github.io/inference/json_output/ Works with any open LLM, including Llama 3.1

radarsat1 2y ago
Looks useful!
AStrangeMorrow 2y ago
Also the outlines library: https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines
- gdiamos 2y ago
  Yeah! - outlines, guidance, jsonformer were inspiring for this line of work
- HanClinto 2y ago
  Also note llama.cpp with grammar support: https://github.com/ggerganov/llama.cpp/tree/master/grammars https://github.com/ggerganov/llama.cpp/tree/master/grammars Supports an EBNF-like syntax, as well as JSON-Schema.
gdiamos 2y ago
[flagged]
msoad 2y ago
Why not JSON Schema?
- gdiamos 2y ago
  We did some user studies and found that people found it less intuitive.

zoogeny 2y ago

Totally tangential, totally not related to the post (unless you squint your eyes and really blur things) ... I was thinking about the old canard of the sufficiently smart compiler. It made me think about LLM output and how in some way the output of a LLM could be bytecode as much as it could be the English language. You have a tokenized input and the translated output. You have a massive and easily generatable training set. I wonder if, one day, our compilers will be LLMs?

jcims 2y ago
You definitely could, not far removed from text to image or text to audio generators.
pjc50 2y ago
Why would you tolerate a nonreliable compiler with no assured relationship between its inputs and its outputs? Have people just got too comfortable with the C++ model of "UB means I can insert a security bug for you"?
- bigyikes 2y ago
  In a hypothetical future where the reliability of LLMs improves, I can imagine the model being able to craft optimizations that a traditional compiler cannot. Like there are already cases where hand-rolling assembly can eke out performance gains, but few do that because it’s so arduous. If the LLM could do it reliably it’d be a huge win. It’s a big if, but not outside the realm of possibility.
  - zoogeny 2y ago
    I agree it is currently a pipe dream. But if I was looking for a doctoral research idea, it might be fun to work on something like that. Lots of potential avenues to explore, e.g. going from a high-level language to some IR, from some IR to bytecode, or straight from high-level to machine code. I mean, -O3 is already so much of a black box that I can't understand it. And the tedium of hand optimizing massive chunks of code is why we automate it at all. Boredom is something we don't expect LLMs to suffer, so having one pore over some kind of representation and apply optimizations seems totally reasonable. And if it had some kinds of "emergent behaviors" based on intelligence that allow it to beat the suite of algorithmic optimization we program into compilers, it could actually be a benefit.

adagradschool 2y ago

While text and image generation are getting cheaper at a significant rate, audio still seems to be just as expensive with ElevenLabs. I wonder why it is so.

say_it_as_it_is 2y ago

This puts like a dozen popular python libraries out of business

AStrangeMorrow 2y ago
At least depends on the approach and use: stuff like outlines (https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines) that actually changes the sampling to adhere to a grammar and can be used with local/custom models shouldn't be too impacted. Those are not really used on top of openAI models
fastball 2y ago
Once it actually works, sure. I just tried to use structured outputs with the latest release (openai-python 1.40) and it doesn't think Structured Outputs is a thing. EDIT: turns out my JSON schema is too large (800 lines + recursive) and seems to be breaking OpenAI's "convert to CFG" step. Whoops.

sansseriff 2y ago

Preprocessing new schema takes 'under 10 seconds'. That's... a huge range? Unless the preprocessing time is a small fraction of the inference time, I don't see the point. I'm working on an app that dynamically generates schema based on user input (a union of arbitrary types pulled from a library). The resulting schema is often in the 800 token range. Curious how long that would take to preprocess

pton_xd 2y ago

Isn't "we hardcoded JSON into the latest model" kind of the opposite direction, strategically, from "we're on the way to AGI and I need 7 trillion to get there?"

isoprophlex 2y ago
You are witnessing the final stages in the evolution of OpenAI from a messianic hype machine to Yet Another Product Company. Hence all the people leaving, too.
- gardenhedge 2y ago
  I am ootl, employees are leaving openai?
  - dangrossman 2y ago
    > John Schulman, one of the co-founders of artificial intelligence company OpenAI, has left the ChatGPT maker for rival Anthropic, he said in a post on social media platform X late Monday. > OpenAI's President and co-founder Greg Brockman is also taking a sabbatical through the end of the year, he said in a X post late Monday. > Peter Deng, a vice-president of product, also left in recent months, a spokesperson said. And earlier this year, several members of the company’s safety teams exited. That's after co-founder and Chief Scientist Ilya Sutskever left in May.
    - oblio 2y ago
      Are there any co-founders left?
      - sashank_1509 2y ago
        sam Altman for one.
    - g15jv2dp 2y ago
      Some employees quitting in a 1500 people company? Impossible. Any departure must be interpreted as the doom of openai, there's no other possibility.

irgolic 2y ago

Wasn’t this already fully supported in the tool calling API?

deleted 2y ago

[deleted]

ramoz 2y ago

Can someone explain how this is different/better than the current state of function calling (which I’ve been using to get a consistent json schema response without issue)?

mrshu 2y ago
This post (from an OpenAI researcher) contains a bit more background: https://news.ycombinator.com/item?id=41174213 https://news.ycombinator.com/item?id=41174213
zbyforgotp 2y ago
This is guaranteed, function calling without it is not. The old way can work for you, but my experience is different, especially with complex schemas.
jacobsimon 2y ago
For starters, the naming is much less confusing. But the behavior also appears to be enforced/validated at some layer (hopefully?), which function calling did not seem to be. I was experimenting with it a couple weeks ago and it would work like 75% of the time but would often give me invalid results for schemas with relatively simple nested objects.

deleted 2y ago

[deleted]

Der_Einzige 2y ago

Now the question is when they will support soft constraints like this: https://huggingface.co/blog/constrained-beam-search https://huggingface.co/blog/constrained-beam-search

agtech_andy 2y ago

I have had a lot of success using BoundaryML (https://www.boundaryml.com/ https://www.boundaryml.com/) for this. They have also been super responsive for any of my questions.

aaronvg 2y ago
thanks for the shoutout, we benchmarked our approach against other function-calling techniques and we've been able to beat all other approaches every time (even by 8%!) just by getting better at parsing the data and representing schemas with less tokens using type definitions instead of json schema. You can take a look at our BFCL results on that site or the github: https://github.com/BoundaryML/baml https://github.com/BoundaryML/baml We'll be publishing our comparison against OpenAI structured outputs in the next 2 days, and a deeper dive into our results, but we aim to include this kind of constrained generation as a capability in the BAML DSL anyway longterm!

tarofchaos 2y ago

Two years too late. I think we are going through a bozo period at OpenAI where small things are being highlighted as achievements.

damsta 2y ago

Can we get something like that in Gemini 1.5 Flash?

mugivarra69 2y ago

cohere had this like a while ago

MattDaEskimo 2y ago

"We have ripped code from a bunch of open-source variations and slapped it behind our brutally abstracted API. Interoperable with other external models like the open source versions? What, are you mad?"

jumploops 2y ago

By using JSON mode, GPT-4{o} has been able to do this reliably for months (100k+ calls). We use GPT-4o to build dynamic UI+code[0], and almost all of our calls are using JSON mode. Previously it mostly worked, but we had to do some massaging on our end (backtick removal, etc.). With that said, this will be great for GPT-4o-mini, as it often struggles/forgets to format things as we ask. Note: we haven't had the same success rate with function calling compared to pure JSON mode, as the function calling seems to add a level of indirection that can reduce the quality of the LLMs output YMMV. Anyhow, excited for this! [0]https://magicloops.dev https://magicloops.dev

deleted 2y ago
[deleted]
qwertox 2y ago
What a cool product! I was about to recommend you to submit it as a "Show HN", but it turns out that it already got submitted one year ago. Would you mind sharing a bit on how things have evolved?
- jumploops 2y ago
  Thanks and great question :) When we first launched, the tool was very manual; you had to generate each step via the UI. We then added a "Loop Creator agent" that now builds Loops for you without intervention. Over the past few months we've mostly been fixing feature gaps and improving the Loop Creator. Based on recent user feedback, we've put a few things in motion: - Form generator (for manual loops) - Chrome extension (for local automations) - In-house Google Sheets integration - Custom outputs (charts, tables, etc.) - Custom Blocks (shareable with other users) With these improvements, you'll be able to create "single page apps" like this one I made for my wife's annual mango tasting party[0]. In addition to those features, we're also launching a new section for Loop templates + educational content/how-tos, in an effort to help people get started. To be super candid, the Loop Creator has been a pain. We started at an 8% success rate and we're only just now at 25%. Theoretically we should be able to hit 80%+ based on existing loop requests, but we're running into limits with the current state of LLMs. [0]https://mangota.ngo https://mangota.ngo

msp26 2y ago

Is the JSON actually being fed into the LLM's context or is it still being converted into typescript? The previous setup didn't allow for custom types, only objects/string/num/bool. Are the enums put into context or purely used for constrained sampling?

OutOfHere 2y ago

Using this feature will obviously "lock you in" to OpenAI, specifically to this model too, at least until other companies catch on. While text prompts can more easily be moved to other LLMs, this feature cannot currently be ported as such. I would use it only if a text prompt is insufficient despite retries.

BoorishBears 2y ago
It's already supported by multiple other providers. Fireworks, Together, probably more.
dtquad 2y ago
OpenAI-style JSON mode and function calling rapidly became the industry standard way of doing it. It will probably also happen for this feature.
- toomuchtodo 2y ago
  “S3 compatible”
PufPufPuf 2y ago
This feature has existed for quite some time in several inference libraries, like Outlines, under the names "constrained decoding" or "guided decoding". Some even include it in their OpenAI-compatible API in a very similar form (allowing to pass in a JSON Schema). All this required doing your own inference, though -- so the announcement really just brings this popular feature "to the masses".
moralestapia 2y ago
OTOH, not using it could "lock you out" of building a cool product for your users, so ...
- OutOfHere 2y ago
  It depends. Quite often the intended schema is simple enough that parsing of a plaintext lines using a regex is sufficient. I do however have to use a verbose prompt to get it to comply with the expected format. Combine this with retries in case of rare failures, and the job is done. I think this new feature is more relevant for intricate schemas and dynamic schemas when a text prompt cannot do the job.

Brechreiz 2y ago

[dead]

jappgar 2y ago

It's nice that they're not making me pay for broken json anymore but touting this as a "feature" is laughable. It's a bug fix. They should never have been charging for malformed responses in the first place!

LAC-Tech 2y ago

Good to see JSON Schema being more widely adopted. I remember doing a project a few years ago in XML just because XML schemas were everywhere and JSON ones were still not really used.

Brosper 2y ago

I think they would like to have something like artifacts in Claude

Havoc 2y ago

Surprised to see OpenAI staff in comments. I don’t recall that being the case on previous threads. (Hello!)

sub7 2y ago

To "look up all my orders in May that were fulfilled and not delivered" is a 1 line query in every ORM I know of. Why in the world would I literally pass around a schema file with each request that is (according to the example) 91 lines of code and pay someone for the privilege of maybe doing it correctly?

roseway4 2y ago

We extensively use vLLM's support for Outlines Structured Output with small language models (llama3 8B, for example) in Zep[0][1]. OpenAI's Structured Output is a great improvement on JSON mode, but it is rather primitive compared to vLLM and Outlines. # Very Limited Field Typing OpenAI offers a very limited set of types[2] (String, Number, Boolean, Object, Array, Enum, anyOf) without the ability to define patterns and max/min lengths. Outlines supports defining arbitrary RegEx patterns, making extracting currencies, phone numbers, zip codes, comma-separated lists, and more a trivial exercise. # High Schema Setup Cost / Latency vLLM and Outlines offer near-zero cost schema setup: RegEx finite state machine construction is extremely cheap on the first inference call. While OpenAI's context-free grammar generation has a significant latency penalty of "under ten seconds to a minute". This may not impact "warmed-up" inference but could present issues if schemas are more dynamic in nature. Right now, this feels like a good first step, focusing on ensuring the right fields are present in schema-ed output. However, it doesn't yet offer the functionality to ensure the format of field contents beyond a primitive set of types. It will be interesting to watch where OpenAI takes this. [0] https://help.getzep.com/structured-data-extraction https://help.getzep.com/structured-data-extraction [1] https://help.getzep.com/dialog-classification https://help.getzep.com/dialog-classification [2] https://platform.openai.com/docs/guides/structured-outputs/supported-schemas https://platform.openai.com/docs/guides/structured-outputs/s...

bamboozled 2y ago

This is a game changer

tuan3w 2y ago

Just curious if structured outputs/constrained generation improve model accuracy, e.g., for information extraction problems. Does anyone have experience with this and why?

thrance 2y ago

And you still can't provide a custom grammar to the API... The company I work for desperately need the LLM to consistently generate results in a subset of HTML. I was able to craft a small grammar file that does just that, in under 5 minutes, and use it successfully with llama.cpp. Yet, there are still no API offering this basic feature that could really benefit everyone. Instead we have a thousand garbage medium articles with "tips & tricks" on how to prompt the AI to get better results. It's as if people don't care anymore about consistency and reliability.

blackcat201 2y ago

Do beware on some reasoning task, our recent work[0] actually found it may cause some performance degradation as well as possible reasoning weakening in JSON. I really hope they fix this in the latest GPT-4o version. [0] https://arxiv.org/abs/2408.02442 https://arxiv.org/abs/2408.02442

kiratp 2y ago
Thank you! This confirms my intuition! Structured generation seems counter to every other signal we have that chain of thought etc improves performance.

myprotegeai 2y ago

I send dynamic jsonschemas to LLMs in manifest[1] to simulate function calls based on function signatures. These structured outputs will be enormously useful. Other posts claim that you can generate jsonschema-conformant output reliably without this, and while I mostly agree, there is an edge case where gpt4o struggles, and that is simple data types. For example, a string, in jsonschema, has a schema of simply {"type": "string"}, and an example value of "hello world." However, gpt4o would produce something like {"value": "hello world"} at a very high probability. I had to include specific few shot examples of what not to do in order to make this simple case reliable. I suspect there are other non-obvious cases. 1. https://github.com/amoffat/manifest https://github.com/amoffat/manifest

franze 2y ago

finally at gpt.franzai.com we use a 3 times check until now if chatgpt does not return valid json then trim anything before and after last {} if this is not valid json feedback him full response and ask for valid json if this is not valid json put full response into self made json to get al least something we can work with in an "something did not work out" response

tomduncalf 2y ago

Would constraining the tokens that the model can choose from impact its “intelligence”/“creativity” in other (possibly negativel) ways?

armcat 2y ago

This is awesome, and simplifies lot of my workflows when using their APIs directly. I also want to give a shout out to Outlines team, https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines, they've been doing structured outputs for the last 12 months, and their open source lib can be applied across all open weight and closed/API-based models. Most likely, Outlines heavily inspired the OpenAI team, maybe even using some of their codebase.

brap 2y ago

Structured outputs seem like an important step forward for LLMs to make them more useful (this includes intermediate "outputs" that are internal to the model, e.g. producing code and then running it without AI, producing mathematical/logical statements and then proving them without AI, etc). Basically, once you have structure, you can use non-AI tools that are far more efficient and don't hallucinate. These tools can either be external, or embedded in the model itself (think RAG, reasoning, etc.). I wonder if JSON is just a first step, and eventually this will be generalized to any formal grammar...

mariarmestre 2y ago

Hmmm... this is not eligible for their zero data retention policy anymore. Not sure how this will go down.

kevlened 2y ago
They added a clause that the schemas themselves aren't covered by ZDR. Their policy for the prompts appears unchanged. https://platform.openai.com/docs/models/default-usage-policies-by-endpoint https://platform.openai.com/docs/models/default-usage-polici...

blixt 2y ago

I was impressed by Microsoft’s AICI where the idea is a WASM program can choose the next tokens. And relatedly their Guidance[1] framework which can use CFGs and programs for local inference to even speed it up with context aware token filling. I hope this implies API-based LLMs may be moving in a similar direction. [1] https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance

munro 2y ago

Wow, this is cool-- first time I've seen generating well formed output has been made easy, the use of Pydantic is clever/easy... Which has lead me to the prior art: using same Pydantic API to generate structured output from local LLMs https://github.com/outlines-dev/outlines?tab=readme-ov-file#efficient-json-generation-following-a-pydantic-model https://github.com/outlines-dev/outlines?tab=readme-ov-file#...

Terretta 2y ago
Commercial use of LLMs is one of those spaces a startup probably needs more than skating ahead of the incumbents on the incumbents' “missing” features as startup value prop. Even if the startup has a year or two head start, it may find its beta “OBE” (overcome by events): https://dottxt.co https://dottxt.co That said, Outline has been making this concept portable for a long time, it's the langchain the community deserves: https://outlines-dev.github.io/outlines/cookbook/ https://outlines-dev.github.io/outlines/cookbook/

tipsytoad 2y ago

How is this different from instructor? github.com/jxnl/instructor namely, why did they take so long for something that just seems like a wrapper around function calling?

hallqv 2y ago

They promised us super intelligence - all we got was valid json

MeetBagelPPML 2y ago

This is super cool! If you're looking to get in early on a free tool that allows you to finetune your model using encrypted data sets you can sign up here: https://waitlist.bagel.net/ https://waitlist.bagel.net/