4 ms·
Justification before decision still works out better in practice, though, because of chain of thought [1]. You'll tend to get more accurate and better-justified
by bunderbunder 2y ago
Justification before decision still works out better in practice, though, because of chain of thought [1]. You'll tend to get more accurate and better-justified decisions.
With decision before justification, you tend to have a greater risk of the output being a wrong decision followed by convincing BS justifying it.
(edit: Another way you could think of it is, LLMs still can't violate causality. Attention heads' ability to look in both directions with respect to a particular token's position in the sequence does not enable them to see into the future and observe tokens that don't exist yet.)
1: https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903
- wtarreau 2y agoI totally agree, that's what I had to do with my patchbot that evaluates haproxy patches to be backported ( https://github.com/haproxy/haproxy/tree/master/dev/patchbot/ https://github.com/haproxy/haproxy/tree/master/dev/patchbot/ ). Originally it would just provide a verdict and justify it and it worked extremely poorly, often with a justification that directly contradicted the verdict. I swapped that by asking the analysis and the final verdict and now the success rate is totally amazing (particularly with mistral that remains unbeatable at this task by obeying extremely well to instructions).
- BeefySwain 2y agoYou find Mistral to be the best "open"/local model? Or you find it to be the best model period?
- deleted 2y ago[deleted]