4 ms·

the more it goes, the more I realize that the true power of LLMs is not in unstructured text that they can generate, but in structured output. but there are two

the more it goes, the more I realize that the true power of LLMs is not in unstructured text that they can generate, but in structured output. but there are two approaches to achieve this: 1. LMQL/guidance/JSONformer/OP's post 2. finetuning the model to understand function calls and their (potentially) JSON schemas. there was a comment here about OpenAI's approach (finetuning a model to understand function call) which raised a good point: since finetuning is often forgetful (previous knowledge learnt by the model gets forgotten a little bit), it's not clear if OpenAI's approach has made GPT-4 less capable than it was before. Not to mention that you're still dealing with a statistical process (LLM), not a locked-in algorithm that generates the desired schema 100% the time. Which brings me to the other approach: steering the LLM's output __as it is generating tokens__, which is what LMQL does. This results in less token usage (you don't send function schema as part of your prompt/message to OpenAI) and 100% accuracy because token probabilities are modified (e.g., 0% chance of any character except ":" after a double quotation mark).

brucethemoose2 3y ago
> Which brings me to the other approach: steering the LLM's output __as it is generating tokens__ A relevant PR: https://github.com/ggerganov/llama.cpp/pull/1773 https://github.com/ggerganov/llama.cpp/pull/1773 The plan is to support arbitrary grammar files to constrain token generation, similar to the grammar files here: https://github.com/antlr/grammars-v4 https://github.com/antlr/grammars-v4
- kelseyfrog 3y ago
  I mean even Jsonformer used LogitsWarper when generating numbers, but yes arbitrary grammars are infinitely more powerful.
darkteflon 3y ago
I'd never heard of LMQL before today, but it looks very nice. Do you have any experience building with it and, if so, would you be willing to comment on what it's like?
- theblazehen 3y ago
  I did a POC project with it recently. The guidance on gpt-3.5-turbo and gpt-4 models isn't as functional as plain gpt-3. I found I had better results using https://github.com/piercefreeman/gpt-json https://github.com/piercefreeman/gpt-json and it doesn't require multiple calls to the API. Not as feature filled, but it may meet your needs
  - darkteflon 3y ago
    Thanks for the recommendation - gpt-json looks quite nice, actually - will check it out.
sebmellen 3y ago
Yes! I don’t have much to say that wouldn’t be restating what I’ve already written, so I’ll link this for reference: https://www.sebastianmellen.com/post/2023/the-killer-use-case-for-llms-is-summarization/ https://www.sebastianmellen.com/post/2023/the-killer-use-cas....
lbeurerkellner 3y ago
Thank you for bringing up LMQL. We have active branches with regex, parsers and types (structured and other types) which will also soon be upstreamed, improving typed LLM use beyond the current template-based approach we support.
sandGorgon 3y ago
has there been any work done to finetune OSS models to behave the same as openai functions (to constrain json output) ? this is tablestakes now, but it doesnt seem ANY opensource model has this capability
- sandkoan 3y ago
  You wouldn't actually want to, because you'd be losing generalizability, and it's a lot of unnecessary work. I think approach #1 outlined above is the better (more cost- and time-efficient) technique—where a pretrained model already understands JSON (among myriad other formats), and you merely constrain it at text-gen time to valid JSON (or other format).
  - sandGorgon 3y ago
    so im not sure what is the difference between what you wrote and what i wrote ? are you distinguishing between "pretrained models" (as in base models) and finetuned models ? here's my question then - was the GPT 0613 update (which introduced functions) a completely new base model or simply a finetuned model ? it seems to be the latter.
    - sandkoan 3y ago
      Yeah, then it seems we agree. I was just pointing out that it's not necessary to finetune OSS models to behave like OpenAI functions if you're able to do something similar to what we did (no tuning involved!).

brucethemoose2 3y ago

> Which brings me to the other approach: steering the LLM's output __as it is generating tokens__ A relevant PR: https://github.com/ggerganov/llama.cpp/pull/1773 https://github.com/ggerganov/llama.cpp/pull/1773 The plan is to support arbitrary grammar files to constrain token generation, similar to the grammar files here: https://github.com/antlr/grammars-v4 https://github.com/antlr/grammars-v4

kelseyfrog 3y ago
I mean even Jsonformer used LogitsWarper when generating numbers, but yes arbitrary grammars are infinitely more powerful.

darkteflon 3y ago

I'd never heard of LMQL before today, but it looks very nice. Do you have any experience building with it and, if so, would you be willing to comment on what it's like?

theblazehen 3y ago
I did a POC project with it recently. The guidance on gpt-3.5-turbo and gpt-4 models isn't as functional as plain gpt-3. I found I had better results using https://github.com/piercefreeman/gpt-json https://github.com/piercefreeman/gpt-json and it doesn't require multiple calls to the API. Not as feature filled, but it may meet your needs
- darkteflon 3y ago
  Thanks for the recommendation - gpt-json looks quite nice, actually - will check it out.

sebmellen 3y ago

Yes! I don’t have much to say that wouldn’t be restating what I’ve already written, so I’ll link this for reference: https://www.sebastianmellen.com/post/2023/the-killer-use-case-for-llms-is-summarization/ https://www.sebastianmellen.com/post/2023/the-killer-use-cas....

lbeurerkellner 3y ago

Thank you for bringing up LMQL. We have active branches with regex, parsers and types (structured and other types) which will also soon be upstreamed, improving typed LLM use beyond the current template-based approach we support.

sandGorgon 3y ago

has there been any work done to finetune OSS models to behave the same as openai functions (to constrain json output) ? this is tablestakes now, but it doesnt seem ANY opensource model has this capability

sandkoan 3y ago
You wouldn't actually want to, because you'd be losing generalizability, and it's a lot of unnecessary work. I think approach #1 outlined above is the better (more cost- and time-efficient) technique—where a pretrained model already understands JSON (among myriad other formats), and you merely constrain it at text-gen time to valid JSON (or other format).
- sandGorgon 3y ago
  so im not sure what is the difference between what you wrote and what i wrote ? are you distinguishing between "pretrained models" (as in base models) and finetuned models ? here's my question then - was the GPT 0613 update (which introduced functions) a completely new base model or simply a finetuned model ? it seems to be the latter.
  - sandkoan 3y ago
    Yeah, then it seems we agree. I was just pointing out that it's not necessary to finetune OSS models to behave like OpenAI functions if you're able to do something similar to what we did (no tuning involved!).