5 ms·

Oh nice! I built a similar system a few weeks ago: https://github.com/newhouseb/clownfish https://github.com/newhouseb/clownfish I think the main differentiati

Oh nice! I built a similar system a few weeks ago: https://github.com/newhouseb/clownfish https://github.com/newhouseb/clownfish I think the main differentiating factor here is that this is better if you have a simpler JSON schema without enums or oneOf constraints. If you do have these constraints, i.e. let's say you wanted an array of different types that represented a items on a menu { kind: pizza, toppings: [pepperoni] } or { kind: ice_cream, flavor: vanilla | strawberry } then you would need something more sophisticated like clownfish that can ask the LLM to pick specific properties (and an ability to do some backtracking so you can do proper beam search). For completeness, another common approach can be found here: https://github.com/ShreyaR/guardrails https://github.com/ShreyaR/guardrails which essentially boils down to "provide the schema in the prompt and ask the LLM to correct things if it fails to get the schema right the first time."

gamegoblin 3y ago
I hate that gpt-3.5-turbo is so cheap that using systems like guardrails is a sane thing to do. I can almost always prompt davinci-003 without guardrails in a way to get my exact schema 1-shot, whereas guardrails + 3.5-turbo will often consume 2-4x more tokens, but that still makes it significantly cheaper.
- brigadier132 3y ago
  The problem people are having is hitting the rate limits for chat gpt.
  - selcuka 3y ago
    Is there a rate limit for the paid API [1] too? [1] https://platform.openai.com/docs/models/gpt-3-5 https://platform.openai.com/docs/models/gpt-3-5
    - gamegoblin 3y ago
      You will get rate limit errors on all the models if you push hard enough. I always wrap the API with some exponential backoff retries to work around this.
sudb 3y ago
Another very similar approach to guardrails which manages to avoid XML that I've been using with some success is langchain's OutputFixingParser: https://python.langchain.com/en/latest/modules/prompts/output_parsers/examples/output_fixing_parser.html https://python.langchain.com/en/latest/modules/prompts/outpu...
killthebuddha 3y ago
One thing that I really like about the approach you took with clownfish is that it doesn't constrain or modify the structure of the prompt. One of the primary difficulties with writing LLM applications is that prompts are basically not composable, and any LLM library that modifies your prompt is going to be a nightmare to work with.
- killthebuddha 3y ago
  Follow-up thought I just had: It seems that prompt structure standards are going to have to emerge if any of these tools have a shot at interoperability. I don't have hard data, but IME if a prompt is structured MEMORY EXAMPLE INSTRUCTION [COMPLETION] it will basically not work to wrap it in a prompt that's structured INSTRUCTION MEMORY EXAMPLE [COMPLETION]
  - ianbutler 3y ago
    Interoperability can also be achieved with small adapters written for the prompting style of the particular model being interfaced with, I'd be surprised if like LangChain or AutoGPT don't already do something like this in their systems. I'm currently building something that leverages an ensemble of different LLMs depending on the difficulty of a task and ran into this issue. Dolly V2 takes "###Instruction: <your stuff> ###Response" as the structure fed to the model where as GPT3.5 Turbo wasn't trained to treat that particular structure as important. The nice thing is that GPT3.5 Turbo will just roll with the prompt structure Dolly uses but that only works in very large LLMs, I'd imagine I wouldn't get away with it in other 12BN parameter models. But realistically this could look like taking the "INSTRUCTION MEMORY EXAMPLE [COMPLETION]" schema represented in a library and each adapter would transform it into "MEMORY EXAMPLE INSTRUCTION [COMPLETION]" schema or whatever is needed by the different model.
    - killthebuddha 3y ago
      I think I agree, and am doing something similar as well. Even the adapters approach does require some amount of consistency across prompts. For example, if you have one prompt that says “you are a very helpful assistant” and one prompt that says “you are a very lazy assistant”, then even if those prompts are otherwise written to be as orthogonal as possible you will still probably see degradation in completion quality.
joshuanapoli 3y ago
Thank you for the really clear and complete description of "ControLogit"s and your approach in clownfish!
gordian-not 3y ago
I have stumbled upon your repository a week ago and I have to say, great work and great ideas! Another thing I thought about is integrating formatting for fields using a similar system. ISO-8601 dates comes immediately to mind but also number and currency formatting are other examples. Probabilistic enums is another thing that I can think of that might be useful for fallback values, I am pretty sure there's a lot of work that can be done in this area, also for other parser kinds related and highly recommended resource is https://github.com/mkuchnik/relm https://github.com/mkuchnik/relm and https://arxiv.org/abs/2211.15458 https://arxiv.org/abs/2211.15458. It is a similar system used to validate LLMs using regexes, however built for completely different use cases. I imagine integrating regex checks to the output fields can also have a lot of use cases.
- newhouseb 3y ago
  Thank you! ReLM is a great find! I like that it drives the generation itself so that it can explore different beams more intentionally. And to do the JSON Parsing well against enums/unions/oneOf, you really have to support backtracking in a way that works basically the same as it does for regex so I'm looking forward to digging into their implementation.
wjessup 3y ago
Thank you for your README, I'm sharing it with my team.

gamegoblin 3y ago

I hate that gpt-3.5-turbo is so cheap that using systems like guardrails is a sane thing to do. I can almost always prompt davinci-003 without guardrails in a way to get my exact schema 1-shot, whereas guardrails + 3.5-turbo will often consume 2-4x more tokens, but that still makes it significantly cheaper.

brigadier132 3y ago
The problem people are having is hitting the rate limits for chat gpt.
- selcuka 3y ago
  Is there a rate limit for the paid API [1] too? [1] https://platform.openai.com/docs/models/gpt-3-5 https://platform.openai.com/docs/models/gpt-3-5
  - gamegoblin 3y ago
    You will get rate limit errors on all the models if you push hard enough. I always wrap the API with some exponential backoff retries to work around this.

sudb 3y ago

Another very similar approach to guardrails which manages to avoid XML that I've been using with some success is langchain's OutputFixingParser: https://python.langchain.com/en/latest/modules/prompts/output_parsers/examples/output_fixing_parser.html https://python.langchain.com/en/latest/modules/prompts/outpu...

killthebuddha 3y ago

One thing that I really like about the approach you took with clownfish is that it doesn't constrain or modify the structure of the prompt. One of the primary difficulties with writing LLM applications is that prompts are basically not composable, and any LLM library that modifies your prompt is going to be a nightmare to work with.

killthebuddha 3y ago
Follow-up thought I just had: It seems that prompt structure standards are going to have to emerge if any of these tools have a shot at interoperability. I don't have hard data, but IME if a prompt is structured MEMORY EXAMPLE INSTRUCTION [COMPLETION] it will basically not work to wrap it in a prompt that's structured INSTRUCTION MEMORY EXAMPLE [COMPLETION]
- ianbutler 3y ago
  Interoperability can also be achieved with small adapters written for the prompting style of the particular model being interfaced with, I'd be surprised if like LangChain or AutoGPT don't already do something like this in their systems. I'm currently building something that leverages an ensemble of different LLMs depending on the difficulty of a task and ran into this issue. Dolly V2 takes "###Instruction: <your stuff> ###Response" as the structure fed to the model where as GPT3.5 Turbo wasn't trained to treat that particular structure as important. The nice thing is that GPT3.5 Turbo will just roll with the prompt structure Dolly uses but that only works in very large LLMs, I'd imagine I wouldn't get away with it in other 12BN parameter models. But realistically this could look like taking the "INSTRUCTION MEMORY EXAMPLE [COMPLETION]" schema represented in a library and each adapter would transform it into "MEMORY EXAMPLE INSTRUCTION [COMPLETION]" schema or whatever is needed by the different model.
  - killthebuddha 3y ago
    I think I agree, and am doing something similar as well. Even the adapters approach does require some amount of consistency across prompts. For example, if you have one prompt that says “you are a very helpful assistant” and one prompt that says “you are a very lazy assistant”, then even if those prompts are otherwise written to be as orthogonal as possible you will still probably see degradation in completion quality.

joshuanapoli 3y ago

Thank you for the really clear and complete description of "ControLogit"s and your approach in clownfish!

gordian-not 3y ago

I have stumbled upon your repository a week ago and I have to say, great work and great ideas! Another thing I thought about is integrating formatting for fields using a similar system. ISO-8601 dates comes immediately to mind but also number and currency formatting are other examples. Probabilistic enums is another thing that I can think of that might be useful for fallback values, I am pretty sure there's a lot of work that can be done in this area, also for other parser kinds related and highly recommended resource is https://github.com/mkuchnik/relm https://github.com/mkuchnik/relm and https://arxiv.org/abs/2211.15458 https://arxiv.org/abs/2211.15458. It is a similar system used to validate LLMs using regexes, however built for completely different use cases. I imagine integrating regex checks to the output fields can also have a lot of use cases.

newhouseb 3y ago
Thank you! ReLM is a great find! I like that it drives the generation itself so that it can explore different beams more intentionally. And to do the JSON Parsing well against enums/unions/oneOf, you really have to support backtracking in a way that works basically the same as it does for regex so I'm looking forward to digging into their implementation.

wjessup 3y ago

Thank you for your README, I'm sharing it with my team.