5 ms·

A lot of models have also been overly chat trained. Responding with stuff like “sure I can help you with that” That’s just unwanted noise if you’re trying to u

by Havoc 2y ago

A lot of models have also been overly chat trained. Responding with stuff like “sure I can help you with that” That’s just unwanted noise if you’re trying to use them as a code building block in an application. So you need to force json or similar…which I suspect harms accuracy over free form

phillipcarter 2y ago
I've not had that experience when I include in the prompt for a coding LLM "only respond with the code". Though it's worth noting that I often do want an explanation, and currently my workflow is to just use a different LLM.
- michaelt 2y ago
  There were some models in the past [1] that were extremely keen to produce chatty noise, even when you explicitly asked them not to. Of course this was back in May 2023, so things might have improved since then. [1] https://news.ycombinator.com/item?id=35964018 https://news.ycombinator.com/item?id=35964018
msp26 2y ago
> which I suspect harms accuracy over free form Untrue in my testing. If you want to use chain of thought, you can always throw in a `thoughts` field (json field/xml tags) before the rest of your output.
- n2d4 2y ago
  If you want to be really sure, you can also first ask it to respond in chat format, and then ask it again to respond in JSON format, if you can afford the cost.
  - msp26 2y ago
    It really isn't necessary when using constrained decoding (aka structured outputs) which guarantees that you'll get JSON output in the correct structure.
    - qeternity 2y ago
      This is not true at all. Just because you can force the logits to give syntactically valid outputs, doesn't mean you're going to get a useful result. Constrained generation, without a proper understanding of the model's natural response tendencies, can give horrible results.
      - msp26 2y ago
        I agree with you completely. I was talking about the parsing being easy with this, not referring to the outputs being correct in reality. You can get awful results with poorly defined constraints.
      - imtringued 2y ago
        Depends on the way you do constrained generation. If all you do is reject tokens using a grammar, then yeah it is bad. If your software inserts things like field names and braces instead of forcing the model to produce them token by token and then afterwards rejecting the wrong tokens, then you should be good to go.
        potatoman22 2y ago
        Why is it bad to reject tokens using a grammar?
Kuinox 2y ago
Are you not using instruct tuned models ?
- TeMPOraL 2y ago
  Obviously they are, that's why they have this problem. Or did the terms "instruction tuning" and "instruct models" change their meaning when I wasn't looking?
  - knicholes 2y ago
    Shoot, maybe someone edited something, but I don't see anyone else in this conversation using the terms "instruction tuning" and "instruct models"?
petesergeant 2y ago
This isn’t a problem in practice. Most of my prompts ask the LLM to do a bunch of chain of thought before asking them to spit out JSON. I extract the JSON, which works 97.5% of the time, and have a retry step being real specific about “here’s the conversation so far but I need JSON now” that handles the rest. Adding examples really helps.
- imtringued 2y ago
  https://lmsys.org/blog/2024-02-05-compressed-fsm/ https://lmsys.org/blog/2024-02-05-compressed-fsm/ I'm not trying to shill sglang specifically, just pointing out that there's a better way, btw.
  - hansvm 2y ago
    ...with the obvious caveat that the distribution of responses isn't the same Elaborating slightly, retrying till the schema is adhered to has a different distribution from greedily selecting tokens adhering to the schema. The simplest toy example I can come up with for that property is a universe of answers "aa", "ab", "bc", all of which the model is equally likely to output for a given prompt with normal auto-regressive invocations. The schema, in regex, is ".[bc]". Retry-till-success produces "ab" 1/2 of the time and "bc" the other half. Greedily adhering to the schema produces "ab" 2/3 of the time and "bc" the remaining third. Last I checked large-scale LLMs, it was a problem in the wild for large string fields. They tend to want to finish the string with ellipses (this creating an incorrect response), but when they made that mistake they'd tend to truncate the entire json record and generate something that doesn't adhere to the schema. Retry-till-success has a high successful parse rate. Greedily adhering to the schema converts those ellipses errors into syntactically correct garbage. Other such bugs can be much harder to quantify (model explainability is hard), but I'd be cautious employing the technique without a lot of case studies for your particular problem domain.
TeMPOraL 2y ago
Unfortunately, that "unwanted noise" is a space for the models to compute; trying to eliminate it gives suboptimal responses. What you can do instead is try to corral it - let the model "think" like it wants, but guide it to add markers wrapping the thinking and/or result, then filter out the thinking in UI (for interactive applications) or as an intermediate/post-processing step (for hidden "building blocks"). If you're using Anthropic models, you may actually get improvements from prompting the model to maintain a tagging discipline; see https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags https://docs.anthropic.com/en/docs/build-with-claude/prompt-....
- iknownthing 2y ago
  interesting
- hedgehog 2y ago
  As other people pointed out here you can also add "verbosity sinks" as text fields in structured output, recently I've also been experimenting with tool calls to support guided self-talk in a way that doesn't necessarily all accumulate in the context (e.g. if not all the tool parameters get echoed back).
  - glaugh 2y ago
    Thank you (and teMPOral) for these comments, this sounds potentially useful to me. I hate to ask this, but I'm struggling to find any thorough posts or articles or papers about this, do you have any links you could point me toward?
    - hedgehog 2y ago
      Speaking only for myself these ideas are a combination of things I've seen scanning new papers and informal discussions with other people working in the area. Feel free to shoot me an e-mail though, maybe I can point you somewhere more specific. Edit: The "verbosity sink" name is inspired by the idea from the paper below although they're not actually at all the same thing. https://arxiv.org/abs/2309.17453 https://arxiv.org/abs/2309.17453
ActionHank 2y ago
I also firmly believe that number of tokens served is a metric that is tracked and encouraged to go up, because more tokens mean more charges. o1 "does more" by using a whole lot more tokens for a very slight bump in usefulness.
- baq 2y ago
  The problem with such takes is 1) the facts are also consistent with performance plateauing for fundamental reasons and 2) there are good, perhaps better than them, competitors with similar issues.
  - ActionHank 2y ago
    I think we’re on the same page here. In lieu of growing the product because of fundamental constraints they’ve opted to grow business metrics to net more revenue which in turn looks good on paper to investors.

by Havoc 2y ago

phillipcarter 2y ago

I've not had that experience when I include in the prompt for a coding LLM "only respond with the code". Though it's worth noting that I often do want an explanation, and currently my workflow is to just use a different LLM.

michaelt 2y ago
There were some models in the past [1] that were extremely keen to produce chatty noise, even when you explicitly asked them not to. Of course this was back in May 2023, so things might have improved since then. [1] https://news.ycombinator.com/item?id=35964018 https://news.ycombinator.com/item?id=35964018

msp26 2y ago

> which I suspect harms accuracy over free form Untrue in my testing. If you want to use chain of thought, you can always throw in a `thoughts` field (json field/xml tags) before the rest of your output.

n2d4 2y ago
If you want to be really sure, you can also first ask it to respond in chat format, and then ask it again to respond in JSON format, if you can afford the cost.
- msp26 2y ago
  It really isn't necessary when using constrained decoding (aka structured outputs) which guarantees that you'll get JSON output in the correct structure.
  - qeternity 2y ago
    This is not true at all. Just because you can force the logits to give syntactically valid outputs, doesn't mean you're going to get a useful result. Constrained generation, without a proper understanding of the model's natural response tendencies, can give horrible results.
    - msp26 2y ago
      I agree with you completely. I was talking about the parsing being easy with this, not referring to the outputs being correct in reality. You can get awful results with poorly defined constraints.
    - imtringued 2y ago
      Depends on the way you do constrained generation. If all you do is reject tokens using a grammar, then yeah it is bad. If your software inserts things like field names and braces instead of forcing the model to produce them token by token and then afterwards rejecting the wrong tokens, then you should be good to go.
      - potatoman22 2y ago
        Why is it bad to reject tokens using a grammar?

Kuinox 2y ago

Are you not using instruct tuned models ?

TeMPOraL 2y ago
Obviously they are, that's why they have this problem. Or did the terms "instruction tuning" and "instruct models" change their meaning when I wasn't looking?
- knicholes 2y ago
  Shoot, maybe someone edited something, but I don't see anyone else in this conversation using the terms "instruction tuning" and "instruct models"?

petesergeant 2y ago

This isn’t a problem in practice. Most of my prompts ask the LLM to do a bunch of chain of thought before asking them to spit out JSON. I extract the JSON, which works 97.5% of the time, and have a retry step being real specific about “here’s the conversation so far but I need JSON now” that handles the rest. Adding examples really helps.

imtringued 2y ago
https://lmsys.org/blog/2024-02-05-compressed-fsm/ https://lmsys.org/blog/2024-02-05-compressed-fsm/ I'm not trying to shill sglang specifically, just pointing out that there's a better way, btw.
- hansvm 2y ago
  ...with the obvious caveat that the distribution of responses isn't the same Elaborating slightly, retrying till the schema is adhered to has a different distribution from greedily selecting tokens adhering to the schema. The simplest toy example I can come up with for that property is a universe of answers "aa", "ab", "bc", all of which the model is equally likely to output for a given prompt with normal auto-regressive invocations. The schema, in regex, is ".[bc]". Retry-till-success produces "ab" 1/2 of the time and "bc" the other half. Greedily adhering to the schema produces "ab" 2/3 of the time and "bc" the remaining third. Last I checked large-scale LLMs, it was a problem in the wild for large string fields. They tend to want to finish the string with ellipses (this creating an incorrect response), but when they made that mistake they'd tend to truncate the entire json record and generate something that doesn't adhere to the schema. Retry-till-success has a high successful parse rate. Greedily adhering to the schema converts those ellipses errors into syntactically correct garbage. Other such bugs can be much harder to quantify (model explainability is hard), but I'd be cautious employing the technique without a lot of case studies for your particular problem domain.

TeMPOraL 2y ago

Unfortunately, that "unwanted noise" is a space for the models to compute; trying to eliminate it gives suboptimal responses. What you can do instead is try to corral it - let the model "think" like it wants, but guide it to add markers wrapping the thinking and/or result, then filter out the thinking in UI (for interactive applications) or as an intermediate/post-processing step (for hidden "building blocks"). If you're using Anthropic models, you may actually get improvements from prompting the model to maintain a tagging discipline; see https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags https://docs.anthropic.com/en/docs/build-with-claude/prompt-....

iknownthing 2y ago
interesting
hedgehog 2y ago
As other people pointed out here you can also add "verbosity sinks" as text fields in structured output, recently I've also been experimenting with tool calls to support guided self-talk in a way that doesn't necessarily all accumulate in the context (e.g. if not all the tool parameters get echoed back).
- glaugh 2y ago
  Thank you (and teMPOral) for these comments, this sounds potentially useful to me. I hate to ask this, but I'm struggling to find any thorough posts or articles or papers about this, do you have any links you could point me toward?
  - hedgehog 2y ago
    Speaking only for myself these ideas are a combination of things I've seen scanning new papers and informal discussions with other people working in the area. Feel free to shoot me an e-mail though, maybe I can point you somewhere more specific. Edit: The "verbosity sink" name is inspired by the idea from the paper below although they're not actually at all the same thing. https://arxiv.org/abs/2309.17453 https://arxiv.org/abs/2309.17453

ActionHank 2y ago

I also firmly believe that number of tokens served is a metric that is tracked and encouraged to go up, because more tokens mean more charges. o1 "does more" by using a whole lot more tokens for a very slight bump in usefulness.

baq 2y ago
The problem with such takes is 1) the facts are also consistent with performance plateauing for fundamental reasons and 2) there are good, perhaps better than them, competitors with similar issues.
- ActionHank 2y ago
  I think we’re on the same page here. In lieu of growing the product because of fundamental constraints they’ve opted to grow business metrics to net more revenue which in turn looks good on paper to investors.