3 ms·
> which I suspect harms accuracy over free form Untrue in my testing. If you want to use chain of thought, you can always throw in a `thoughts` field (json fie
by msp26 2y ago
> which I suspect harms accuracy over free form
Untrue in my testing. If you want to use chain of thought, you can always throw in a `thoughts` field (json field/xml tags) before the rest of your output.
- n2d4 2y agoIf you want to be really sure, you can also first ask it to respond in chat format, and then ask it again to respond in JSON format, if you can afford the cost.
- msp26 2y agoIt really isn't necessary when using constrained decoding (aka structured outputs) which guarantees that you'll get JSON output in the correct structure.
- qeternity 2y agoThis is not true at all. Just because you can force the logits to give syntactically valid outputs, doesn't mean you're going to get a useful result. Constrained generation, without a proper understanding of the model's natural response tendencies, can give horrible results.
- msp26 2y agoI agree with you completely. I was talking about the parsing being easy with this, not referring to the outputs being correct in reality. You can get awful results with poorly defined constraints.
- imtringued 2y agoDepends on the way you do constrained generation. If all you do is reject tokens using a grammar, then yeah it is bad. If your software inserts things like field names and braces instead of forcing the model to produce them token by token and then afterwards rejecting the wrong tokens, then you should be good to go.
- potatoman22 2y agoWhy is it bad to reject tokens using a grammar?