4 ms·

This seems just a clone of Microsoft Guidance.

simonw 3y ago
This IS Microsoft Guidance, they seem to have spun off a separate GitHub organization for it. https://github.com/microsoft/guidance https://github.com/microsoft/guidance redirects to https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance now.
- verdverm 3y ago
  Simon in the house to clear the air! Do you have similar capabilities in your LLM project?
  - simonw 3y ago
    Not yet. I've been investigating llama-cpp grammars recently with an eye to getting those working in LLM - maybe even using them to get an equivalent to OpenAI Functions working (which I'd like to include in LLM too). Notes on grammars here: https://til.simonwillison.net/llms/llama-cpp-python-grammars https://til.simonwillison.net/llms/llama-cpp-python-grammars
    - verdverm 3y ago
      It's still unclear to me if that is the right direction or a scalable solution. Seeing how far prompt engineering can get you on this from (specifying the grammar as a one/few shot) does pretty well, it seems like something that could be better handled at training/refining time? My general feeling is that it is inserting a specific and complicated cog, and probably has to be tweaked for each model (as they all have their little quirks) Curious what you think having been working directly on these things? From my experience, you can get an LLM to follow a "grammar" for pretty much anything, without it being an actual grammar spec in one of the many formats. You can pretty much make it up. Here's an example of us getting CUE out of a model by giving "tricking" the LLM to generate JSON with less syntax (a subset of both CUE and JSON). Bonus, fewer quotes and commas meant fewer tokens. We turn this into JSON afterwards, works surprisingly well https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/prompts/dm.cue#L27 https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/pro...
      - simonw 3y ago
        I don't trust most LLMs to reliably follow instructions to "only output JSON with no extra text". Llama 2 for example really isn't very good at following those kinds of instructions in my experience. I really like how grammars offer a realistic path to getting completely dependable formatted output from these models.
        verdverm 3y ago
        interesting, I hadn't thought about the problem where they don't follow the instruction "only output the JSON, do not add explanations or other text" I haven't pushed on codellama2 much yet, but my initial experiments, it did not really output anything extra, and my prompt became a one-liner compared to the really long instructions I had to give chatgpt for controlling output. Shows how far you can get with a purpose trained model fine-tuning is important to getting more consistent output, none of the smaller models (open-source sized) are going to get there with just few-shot. Sounds like the grammar logit influencer is a low-cost/effort way to constrain output without the fine-tuning cycle. I can imagine they might be better together, but my hunch is that fine-tuning will still dominate the improvements and consistency. If you don't have the training data, that is a very good reason to use this technique too
        simonw 3y ago
        My favourite joke about LLMs not following formatting instructions is this from Riley Goodside: https://twitter.com/goodside/status/1657396491676164096 https://twitter.com/goodside/status/1657396491676164096

This IS Microsoft Guidance, they seem to have spun off a separate GitHub organization for it. https://github.com/microsoft/guidance https://github.com/microsoft/guidance redirects to https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance now.

verdverm 3y ago
Simon in the house to clear the air! Do you have similar capabilities in your LLM project?
- simonw 3y ago
  Not yet. I've been investigating llama-cpp grammars recently with an eye to getting those working in LLM - maybe even using them to get an equivalent to OpenAI Functions working (which I'd like to include in LLM too). Notes on grammars here: https://til.simonwillison.net/llms/llama-cpp-python-grammars https://til.simonwillison.net/llms/llama-cpp-python-grammars
  - verdverm 3y ago
    It's still unclear to me if that is the right direction or a scalable solution. Seeing how far prompt engineering can get you on this from (specifying the grammar as a one/few shot) does pretty well, it seems like something that could be better handled at training/refining time? My general feeling is that it is inserting a specific and complicated cog, and probably has to be tweaked for each model (as they all have their little quirks) Curious what you think having been working directly on these things? From my experience, you can get an LLM to follow a "grammar" for pretty much anything, without it being an actual grammar spec in one of the many formats. You can pretty much make it up. Here's an example of us getting CUE out of a model by giving "tricking" the LLM to generate JSON with less syntax (a subset of both CUE and JSON). Bonus, fewer quotes and commas meant fewer tokens. We turn this into JSON afterwards, works surprisingly well https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/prompts/dm.cue#L27 https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/pro...
    - simonw 3y ago
      I don't trust most LLMs to reliably follow instructions to "only output JSON with no extra text". Llama 2 for example really isn't very good at following those kinds of instructions in my experience. I really like how grammars offer a realistic path to getting completely dependable formatted output from these models.
      - verdverm 3y ago
        interesting, I hadn't thought about the problem where they don't follow the instruction "only output the JSON, do not add explanations or other text" I haven't pushed on codellama2 much yet, but my initial experiments, it did not really output anything extra, and my prompt became a one-liner compared to the really long instructions I had to give chatgpt for controlling output. Shows how far you can get with a purpose trained model fine-tuning is important to getting more consistent output, none of the smaller models (open-source sized) are going to get there with just few-shot. Sounds like the grammar logit influencer is a low-cost/effort way to constrain output without the fine-tuning cycle. I can imagine they might be better together, but my hunch is that fine-tuning will still dominate the improvements and consistency. If you don't have the training data, that is a very good reason to use this technique too
        simonw 3y ago
        My favourite joke about LLMs not following formatting instructions is this from Riley Goodside: https://twitter.com/goodside/status/1657396491676164096 https://twitter.com/goodside/status/1657396491676164096