8 ms·

It's so wild that the bar for AI performance is both absurdly high and absurdly low at the same time. To specify an output format (language or grammar) for solv

by titzer 2y ago

It's so wild that the bar for AI performance is both absurdly high and absurdly low at the same time. To specify an output format (language or grammar) for solving a computational problem is one of the oldest exercises around. On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification. It's weird reading the kind of self-congratulating blogpost about this, like OpenAI has just discovered flint knives. On the other hand, a computer system can process natural language with extremely ambiguous, open-ended problems, compute solutions to said problems, even correct its own mistakes--and then it can format the output correctly. And then on yet another hand, it only took about 10^25 floating point operations (yeah, just ten million trillion trillion, right!?) to get this outcome.

codingwagie 2y ago
I think it will take a long time for the world at large to realize and then operationalize the potential of this "mundane" technology. It is revolutionary, and also sitting in plain sight. Such a huge technological shift that was considered decades out only a few years ago
- ben_w 2y ago
  Although I am an optimist* about what this can do, I am very much aware — from personal experience — how easy it is to see more than is really there. The realisation of the tech might be fantastic new things… or it might be that people like me are Clever Hans-ing the models. * that may be the wrong word; "strong capabilities" is what I think is present, those can be used for ill effects which is pessimistic.
srcreigh 2y ago
If I wanted to be a silly pedant, I’d say that Turing machines are language specifications and thus it’s theoretically impossible for an LLM or any program to validate output formats in general.
- deleted 2y ago
  [deleted]
- jes5199 2y ago
  in _general_ sure, but if you restricted each token to conform to a Kleene-star grammar you should be able to guarantee that you get something that parses according to a context-free grammar
thruway516 2y ago
I dont understand your complaint at all. If you develop a new revolutionary technology called an automobile, developing steering, brakes, starter, mufflers for it is a pretty big deal even if reins, clamps, mufflers and keys are mundane and have existed for decades. Structured outputs are a pretty big step in making this magic actually usable by developers as opposed to generating impressive cat pictures or whatever has captured the public imagination.
- Bjartr 2y ago
  I don't think it was an complaint, just a observation.
  - thruway516 2y ago
    Yes probably. But considering non-deterministic outputs is the nature of the beast with Llms and we're (mostly) engineers here, calling any part of this mundane sounds almost more like fighting words than just observation
    - the8thbit 2y ago
      Extremely pedantic, but is "non-deterministic" really the right language? The same input will always produce the same output, provided you haven't intentionally configured the system to use the model non-deterministically. It seems like the right way to describe it is as a chaotic deterministic system. The same input will always produce the same output, but small shifts in the input or weights can result in dramatic and difficult to predict changes in outputs.
      - davedx 2y ago
        Llms are indeed non deterministic
      - visarga 2y ago
        > The same input will always produce the same output Not guaranteed even with the same seed. If you don't perform all operations in exactly the same order, even a simple float32 sum, if batched differently, will result in different final value. This depends on the load factor and how resources are allocated.
tommica 2y ago
For some reason it reminds me of my civilization runs - rush to certain high level tech and then after that discovery writing :D
ramraj07 2y ago
This is like saying “we shouldn’t be celebrating a computer that can talk, my parrot can do that!”
- deleted 2y ago
  [deleted]
throwawaymaths 2y ago
> On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification. I highly doubt it's the model that does this... It's very likely code injected into the token picker. You could put this into any model all the way down to gpt-2.
- crowcroft 2y ago
  I wonder if you get 90% of the way with prompt engineering, and then the last 10% is just brute force, validate output, if it fails, rerun the prompt. My assumption is if that's all this is they would have done it a long time ago though.
  - jeeceebees 2y ago
    You can just mask the output probabilities for each token based on which options are valid according to a grammar. There are quite a few open source implementations of this e.g. https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines
    - contravariant 2y ago
      You could simply censor invalid tokens, but that does rely on 2 assumptions. 1. There is always a valid next token. 2. This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm. The latter isn't too obvious, and may in fact be (very) false. Look up maze generation algorithms if you want some feeling for the effects this could have. If you just want a quick argument, consider what happens if picking the most likely token would increase the chance of an invalid token further down the line to nearly 100%. By the time your token-picking algorithm has any effect it would be too late to fix it.
      - throwawaymaths 2y ago
        Sorry, how could there not be a valid next token? Presumably your interface would generate a state machine with appropriate masking arrays, and iirc generally speaking all 256 byte choices are in the token list. There's no way to get stuck in a place where the JSON is invalid? Can you give an example? If you want to be really clever about your picker, a deterministic result would blat out the all the known possible strings. For example, if you had an object with defined a defined set of properties, you could just go ahead and not bother generating tokens for all the properties and just tokenize, E.G. `{"foo":"` (6-ish tokens) without even passing through the LLM. As soon as an unescaped `"` arrives, you know the continuation must be `,"bar":"`, for example > This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm. It absolutely will. But so will adding an extra newline in your prompt, for example. That sort of thing is part and parcel of how llms work
scarmig 2y ago
I have struggled writing valid YAML before (my tokenizer doesn't handle whitespace very well). And it probably takes me a quadrillion operations on the reals to get a minimal YAML file (I think your 10^25 fp ops is an overestimate--I think it's more like 10^18-10^19). It's kind of like an inverse Moravec's paradox.
- theturtle32 2y ago
  Relatable!!
m3kw9 2y ago
It’s doing more, it is allowing user to input using natural language and the output is the json format the API that is defined
raincole 2y ago
I don't know, it doesn't sound wild at all to me. Human languages are very imprecise, vague and error-tolerant, which is the opposite of an output format like JSON. So the a model can't do these two things well at the same time is quite an intuitive conclusion. The wild part is that a model trained with so much human language text can still outputs mostly compilable code.
berkes 2y ago
I am so often surprised by "The AI Communities" software. Often unpleasantly surprised, often just eye-rolling. When we first started using the OpenAI API's, the first thing I reached for was some way "to be certain that the response is properly formatted". There wasn't one. A common solution was (is?) "just run the query again, untill you can parse the JSON". Really? After decades of software engineering, we still go for the "have you tried turning it off and on again" on all levels. Then I reached for common, popular tools: everyone's using them, they ought to be good, right? But many of these tools, from langchain's to dify to promptflow are a mess (Edit: to alter the tone: I'm honestly impressed by the breadth and depth of these tools. I'm just suprised about the stability - lack thereof, of them). Nearly all of them suffer from always-outdated-documentation. Several will break right after installing it, due to today's ecosystem updates that haven't been incorporated entirely yet. Understandably: they operate in an ecosystem that changes by the hour. But after decades of software engineering, I want stuff that's stable, predictable, documented. If that means I'm running LLM models from a year ago: fine. At least it'll work. Sure, this constant-state-of-brokeness is fine for a PoC, a demo, or some early stage startup. But terrible for something that I want to ensure to still work in 12 months, or 4 years even without a weekly afternoon of upgrade-all-dependencies-and-hope-it-works-the-update-my-business-logic-code-to-match-the-now-deprecated-apis.
- KoolKat23 2y ago
  Well the simple answer is don't use it then. In the same way people revert to older stable releases. You're welcome to revert to writing boilerplate code yourself. The reason people are excited and use it, is because they show promise, it already offers significant benefits even if it isn't "stable".
  - berkes 2y ago
    "Just write your own if you aren't happy" doesn't make my critique invalid. My problem is that this community, and thus the ecosystem, is repeating many mistakes, reinventing wheels, employing known bad software-engineering practices, or lacking any software-engineering practices at all. It is, in short, not learning from decades of work. It's not standing on shoulders of giants, it's cludging together wobbly ladders to race to the top. Even if "getting to the top first" is the primary goal, it's not the fastest way. But certainly not the most durable way. (To be clear: I have seen the JS (node) community doing the exact same, racing fast towards cliffs and walls, realizing this, throwing it all out, racing to another cliff and so on. And I see many areas in the Python community doing this too. Problems have been solved! For decades! Learn from these instead of repeating the entire train of failures to maybe finally solve the problems in your niche/language/platform/ecosystem)

by titzer 2y ago

codingwagie 2y ago

I think it will take a long time for the world at large to realize and then operationalize the potential of this "mundane" technology. It is revolutionary, and also sitting in plain sight. Such a huge technological shift that was considered decades out only a few years ago

ben_w 2y ago
Although I am an optimist* about what this can do, I am very much aware — from personal experience — how easy it is to see more than is really there. The realisation of the tech might be fantastic new things… or it might be that people like me are Clever Hans-ing the models. * that may be the wrong word; "strong capabilities" is what I think is present, those can be used for ill effects which is pessimistic.

srcreigh 2y ago

If I wanted to be a silly pedant, I’d say that Turing machines are language specifications and thus it’s theoretically impossible for an LLM or any program to validate output formats in general.

deleted 2y ago
[deleted]
jes5199 2y ago
in _general_ sure, but if you restricted each token to conform to a Kleene-star grammar you should be able to guarantee that you get something that parses according to a context-free grammar

thruway516 2y ago

I dont understand your complaint at all. If you develop a new revolutionary technology called an automobile, developing steering, brakes, starter, mufflers for it is a pretty big deal even if reins, clamps, mufflers and keys are mundane and have existed for decades. Structured outputs are a pretty big step in making this magic actually usable by developers as opposed to generating impressive cat pictures or whatever has captured the public imagination.

Bjartr 2y ago
I don't think it was an complaint, just a observation.
- thruway516 2y ago
  Yes probably. But considering non-deterministic outputs is the nature of the beast with Llms and we're (mostly) engineers here, calling any part of this mundane sounds almost more like fighting words than just observation
  - the8thbit 2y ago
    Extremely pedantic, but is "non-deterministic" really the right language? The same input will always produce the same output, provided you haven't intentionally configured the system to use the model non-deterministically. It seems like the right way to describe it is as a chaotic deterministic system. The same input will always produce the same output, but small shifts in the input or weights can result in dramatic and difficult to predict changes in outputs.
    - davedx 2y ago
      Llms are indeed non deterministic
    - visarga 2y ago
      > The same input will always produce the same output Not guaranteed even with the same seed. If you don't perform all operations in exactly the same order, even a simple float32 sum, if batched differently, will result in different final value. This depends on the load factor and how resources are allocated.

tommica 2y ago

For some reason it reminds me of my civilization runs - rush to certain high level tech and then after that discovery writing :D

ramraj07 2y ago

This is like saying “we shouldn’t be celebrating a computer that can talk, my parrot can do that!”

deleted 2y ago
[deleted]

throwawaymaths 2y ago

> On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification. I highly doubt it's the model that does this... It's very likely code injected into the token picker. You could put this into any model all the way down to gpt-2.

crowcroft 2y ago
I wonder if you get 90% of the way with prompt engineering, and then the last 10% is just brute force, validate output, if it fails, rerun the prompt. My assumption is if that's all this is they would have done it a long time ago though.
- jeeceebees 2y ago
  You can just mask the output probabilities for each token based on which options are valid according to a grammar. There are quite a few open source implementations of this e.g. https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines
  - contravariant 2y ago
    You could simply censor invalid tokens, but that does rely on 2 assumptions. 1. There is always a valid next token. 2. This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm. The latter isn't too obvious, and may in fact be (very) false. Look up maze generation algorithms if you want some feeling for the effects this could have. If you just want a quick argument, consider what happens if picking the most likely token would increase the chance of an invalid token further down the line to nearly 100%. By the time your token-picking algorithm has any effect it would be too late to fix it.
    - throwawaymaths 2y ago
      Sorry, how could there not be a valid next token? Presumably your interface would generate a state machine with appropriate masking arrays, and iirc generally speaking all 256 byte choices are in the token list. There's no way to get stuck in a place where the JSON is invalid? Can you give an example? If you want to be really clever about your picker, a deterministic result would blat out the all the known possible strings. For example, if you had an object with defined a defined set of properties, you could just go ahead and not bother generating tokens for all the properties and just tokenize, E.G. `{"foo":"` (6-ish tokens) without even passing through the LLM. As soon as an unescaped `"` arrives, you know the continuation must be `,"bar":"`, for example > This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm. It absolutely will. But so will adding an extra newline in your prompt, for example. That sort of thing is part and parcel of how llms work

scarmig 2y ago

I have struggled writing valid YAML before (my tokenizer doesn't handle whitespace very well). And it probably takes me a quadrillion operations on the reals to get a minimal YAML file (I think your 10^25 fp ops is an overestimate--I think it's more like 10^18-10^19). It's kind of like an inverse Moravec's paradox.

theturtle32 2y ago
Relatable!!

m3kw9 2y ago

It’s doing more, it is allowing user to input using natural language and the output is the json format the API that is defined

raincole 2y ago

I don't know, it doesn't sound wild at all to me. Human languages are very imprecise, vague and error-tolerant, which is the opposite of an output format like JSON. So the a model can't do these two things well at the same time is quite an intuitive conclusion. The wild part is that a model trained with so much human language text can still outputs mostly compilable code.

berkes 2y ago

I am so often surprised by "The AI Communities" software. Often unpleasantly surprised, often just eye-rolling. When we first started using the OpenAI API's, the first thing I reached for was some way "to be certain that the response is properly formatted". There wasn't one. A common solution was (is?) "just run the query again, untill you can parse the JSON". Really? After decades of software engineering, we still go for the "have you tried turning it off and on again" on all levels. Then I reached for common, popular tools: everyone's using them, they ought to be good, right? But many of these tools, from langchain's to dify to promptflow are a mess (Edit: to alter the tone: I'm honestly impressed by the breadth and depth of these tools. I'm just suprised about the stability - lack thereof, of them). Nearly all of them suffer from always-outdated-documentation. Several will break right after installing it, due to today's ecosystem updates that haven't been incorporated entirely yet. Understandably: they operate in an ecosystem that changes by the hour. But after decades of software engineering, I want stuff that's stable, predictable, documented. If that means I'm running LLM models from a year ago: fine. At least it'll work. Sure, this constant-state-of-brokeness is fine for a PoC, a demo, or some early stage startup. But terrible for something that I want to ensure to still work in 12 months, or 4 years even without a weekly afternoon of upgrade-all-dependencies-and-hope-it-works-the-update-my-business-logic-code-to-match-the-now-deprecated-apis.

KoolKat23 2y ago
Well the simple answer is don't use it then. In the same way people revert to older stable releases. You're welcome to revert to writing boilerplate code yourself. The reason people are excited and use it, is because they show promise, it already offers significant benefits even if it isn't "stable".
- berkes 2y ago
  "Just write your own if you aren't happy" doesn't make my critique invalid. My problem is that this community, and thus the ecosystem, is repeating many mistakes, reinventing wheels, employing known bad software-engineering practices, or lacking any software-engineering practices at all. It is, in short, not learning from decades of work. It's not standing on shoulders of giants, it's cludging together wobbly ladders to race to the top. Even if "getting to the top first" is the primary goal, it's not the fastest way. But certainly not the most durable way. (To be clear: I have seen the JS (node) community doing the exact same, racing fast towards cliffs and walls, realizing this, throwing it all out, racing to another cliff and so on. And I see many areas in the Python community doing this too. Problems have been solved! For decades! Learn from these instead of repeating the entire train of failures to maybe finally solve the problems in your niche/language/platform/ecosystem)