3 ms·
Interesting! What results did you get with that? Also, do you think this is what O3 is doing?
by EMIRELADERO 2y ago
Interesting! What results did you get with that?
Also, do you think this is what O3 is doing?
- noosphr 2y agoAbove sota on logical reasoning for grounding text. LLMs at the time were so bad at it that even fornier models when given A, not B would derive A and B in the output about half the time.
- bloomingkales 2y agoWhat do you use to benchmark?
- noosphr 2y agoIt was a SAT solver with the symbolic expression fed into the LLM. Using the SAT solver I'd rate how well the LLM was able to solve a given boolean formula, if it got it wrong it would get a second prompt to ask it to split the boolean expression at the main connective into two sub expressions. If it got that right it would be asked to solve the sub expressions etc.. Then reward/punish based on how well it did overall. There was meant to be a lot more to the system than that, but I never had the training budget to do anything but the first draft of the system.
- bloomingkales 2y agoI was hoping for something more reproducible. Free work, I know.
- noosphr 2y agoThis was done as part of a contract and like I said I never got permission to write a paper about it, let alone share code.
- bloomingkales 2y agoNot even a hint?